Package: rcounting
A collection of tools for working with data on the counting threads in /r/reddit.com/r/counting.DescriptionThere's acommunity on redditthat likes to count collaboratively. Any kind of count - if you can think of a sequence of numbers (or letters, or words) what could be fun to count in order, we've probably tried counting it.As well as counting, we also make stats on who's made how many counts, how fast they were, and who managed to count numbers ending in lots of nines or zeros.This repository has tools for interacting with the reddit api through thePython Reddit API Wrapper, to help gathering these statistics.Installation and usageThe package is available onpypias a package, so installation is as easy aspip3 install rcounting. If you want the very latest commit, you can install by typingpip3 install git+https://github.com/cutonbuminband/rcounting.git.The command line interface for the package is all under the commandrcounting. Typercounting -hto see what options there are -- the main ones are described below.The first time you run the program you will be asked to authorize it to interact with reddit on your behalf. Specifically, it needs to be able toRead posts on redditRead wiki pages on redditEdit wiki pages (for updating the thread directory)Thread LoggingThe package has functionality for logging threads which can be invoked by typingrcounting log. The default behaviour is to log the latest complete thread (as found in thedirectory, saving the output to csv files. You can specify that you want to log a different threads, want to log a while chain of threads, or want to store the output in a database instead. Try typingrcounting log_thread -hto see a more detailed usage explanation.ValidationThe package can also validate threads according to specific rules. This is done by typingrcounting validate, and the program takes an additional--ruleparameter specifying which rule should be checked. The following options are available:default: No counter can reply to themselveswait2: Counters can only count once two others have countedwait3: Counters can only count once three others have countedonce_per_thread: Counters can only count once on a given reddit submissionslow: One minute must elapse between countsslower: Counters must wait one hour between each of their countsslowestest: One hour must elapse between each count, and counters must wait 24h between each of their countsonly_double_counting: Counters must reply to themselves exactly once before it's somebody else's turn.If no rule is supplied, the program will only check that nobody double counted.After you run it, it'll print out whether all the counts in the chain were valid, and if there was an invalid count, which one it was.Updating the thread directoryFinally, there's a program to update thedirectory of side threads. It's invoked by callingrcounting update-directory, and roughly follows the following stepsIt gets all submissions to r/counting made within the last six monthsIt tries to find a link to the parent submission of each submissionFirst it looks for a reddit url in the body of each submission (trying to find the "continued from here" lineIf that fails, it goes through all the top level comments of the submission looking for a reddit urlIt constructs a chain for each thread from parent submission to child submissionFor each row in each table in the directory, it extractsWhich side thread it is, based on the link to the first threadWhat the previous submission, comment and total count were.It then uses the chain of submissions to find the latest submission of each thread typeAnd walks down the comments on each submission to the latest one. At each level of comments it goes to the first valid reply based onA per-thread rule describing when a count is validA per-thread rule describing what looks like a count (to skip over mid-thread conversations)If the latest submission is not the same as the previous one, it tries to update the total count fieldSome threads hang around for a long time, so there's also anarchiveof older threads. If a submission is more than six months old and still been completed, the thread is moved to the archive.Some of the threads from the last six months might not be in the directory (yet). These are potentially new or revived threads. If a submission contains no links to previous submissions, it's considered a new thread, and once it has more than 50 counts by 5 different users, it's automatically added to the directory. Submissions which link to archived threads are considered to be revivals of the archived thread, and once the submission has 20 counts, it's moved from the archive to the new threads table.If you run the script with no parameters it takes around 15 minutes to run, depending on how out of date the directory pages are. That's an unavoidable consequence of the rate-limiting that reddit does.Data analysisUsing the scripts here (and an archive supplied by members of r/counting), I've scraped every comment in the main counting chain, including the comment bodies. There are a number of interesting plots and tables that can be made using this data; here's a list ofexamplesof working with the data.DependenciesThe program has been tested on python 3.7, 3.8 and 3.10, on Windows and on Linux.The program makes use of thePython Reddit API Wrapperto interact with reddit andpandasto work with tabular data. Updating some of the side threads requiresscipy-- yes, really!The command line interface is built usingclick.LicenseThis project is licensed under the terms of the GNU General Public License v3.0, or, at your discretion, any later version. You can read the license termshere.Contributing and Future IdeasThis is a loosely organised list of things which could be done in the future. If you have any suggestions, don't hesitate to write, or to send a pull request. As a heads up, when you submit a pull request, you are also agreeing to license your code under the GPL (see github'sterms of service).Recovering gracefully if a linked comment is inaccessible because it's been deleted or removedMaking the comment and url extraction less brittleGet in touchIf you have any questions, suggestions or comments about this project, you can contact the maintainer atcutonbuminband@gmail.com, or visit thecounting subredditand post in the weekly Free Talk Friday thread.