rubygems/rubygems.org — Issue #384: Making download statistics available, in bulk
Author: RichMorin
Created: 2012-01-18 17:44:10+00:00
State: open

sferik's "gems" gem allows an external site to request certain statistics (eg, total_downloads). This is very cool, but I'd like to track this sort of thing on a daily basis for all 170K gems. Making that many requests (eg, on a daily basis) seems Wrong, so I'm looking for a better option.

One possibility would be to create text files, updated nightly, that contain the statistics. Ideally, these would be both human- and machine-readable. I'd prefer a simple YAML format, to allow extensibility and provide self-documentation. If the files were stored on GitHub, folks could examine the files online, go back and pick up any that they might have missed, etc.

It might be useful to break up the information in some manner. For example, I've been putting mirrored gems into buckets by the first two (downcased) characters of the name (eg, "12/12_hour_time"). For details, see
http://cfcl.com/twiki/bin/view/Projects/SPARQLy/Gems/File_Layout#Bucket_distribution