## 4 Data, Analysis, and Observations  
### Data Collection and Pre-Processing  
We collected data from Reddit over two months in 2022 (60 days) using an automated crawler. At the time of this writing, Reddit has limited the possibility of collecting data from the platform using any automated system. However, our dataset was collected in 2022 when no such restrictions were in place. The data were collected between July and August 2022; they consisted of 125,000 subreddits and approximately 55 million daily active users. We obtained posts and comments for every user by subreddits. The crawler for obtaining the data was run on a system running a Ubuntu 20.04 server and a 10G Ethernet connection. The total size of the collected data on disk was 1 TB. There were several possibilities for collecting data from Reddit automatically before the restrictions on automated data collection were applied. (1) The Reddit Application Programming Interface (API), which was provided by Reddit. (2) Pushshift API, which is an open-source project that offers a flexible and efficient way of accessing large amounts of data from Reddit. Note that since Reddit made changes to their data policy in mid-2023, this method has been rendered obsolete. (3) Python PRAW Package (version 7.7.1) (using Reddit's API): PRAW stands for Python (version 3.8+) Reddit API wrapper. This method allows one to access Reddit's API and collect posts and comments from any subreddit or thread (discussion chain). Pushshift offered several advantages over the other data collection methods at the time we designed and implemented our crawlers, and hence, we decided to use this for implementing our system. One of the most notable advantages was its ability to retrieve historical data, which were not available through the other available means. Data can be retrieved as far back as 2005, which is useful for analyzing trends and patterns over  
Figure 4: Effect of interaction and neighbor pruning on subreddit Castaneda. (**a**) Interaction pruning a = 0.0, b = 2, and c = 0. (**b**) Neighbor pruning a = 0.0, b = 0, and c = 2. Figure 5: (**a**): SIG for subreddit Castaneda with simultaneous weight and neighbor pruning a = 2.0, b = 0, c = 2. (**b**): Out-Degree distribution for subreddit Books. time. Pushshift offers volume-based queries so that one can retrieve large amounts of data quickly and efficiently. It also provides endpoints for retrieving data from various Reddit features, including submissions, comments, and subreddits. This flexibility allows users to tailor their queries to specific needs and interests. Due to all these advantages, we decided to use Pushshift for implementing our crawler. We developed a crawler that leveraged Pushshift's bulk data capabilities. The components of the crawler are shown in Figure 6. By downloading all posts and comments over a two-month period, we obtained approximately one terabyte of data, which required extensive processing. The data were large and unordered, and hence, in order to optimize the downstream data processing, we organized it into buckets. A bucket conceptually corresponds to a subreddit. Within each bucket, data for a subreddit were organized by the days of the month on which the data were generated. Since, in our case, the data were collected for two months, each bucket had entries for 60 days. Each entry in a bucket corresponds to a post and the associated comments chain. For each post, we searched five days into the future to obtain all the comments. This was conducted because we observed that most activity on posts on Reddit occurs within a short time window after a post is made. Additionally, we limited our search to at most five days in the future based on our observation that posts typically garner attention for five days on average on Reddit. Another important challenge involved reconciling submissions (posts) that were made on the last day of a month with comments that were made on the first day of the next month.