alphagov/rummager — pull request #116 by jamiecobbett (2013-06-27 09:55:12+00:00)
Title: Add a script for some stats on GOV.UK content

I thought it would be interesting to get some stats on the contents of the search index. Then I thought it might be useful/interesting to keep the script, and get feedback to improve the accuracy.

The script takes about forty seconds to run on my dev VM.

Known issues:
* it only looks at the ``indexable_content`` field, so ignores titles in many or
all cases. Lots of government content has no ``indexable_content``.
* the dictionary size is likely inflated because punctuation isn't stripped
* If there's a timeout, it falls over and dies horribly.