duckduckgo/zeroclickinfo-fathead — pull request #95 by yanirs (2014-08-15 05:22:44+00:00)
Title: Converted pypi to use the simple HTML and JSON APIs

Following the discussion in PR #93, this change includes:
* Fetching all the PyPI package information using the simple HTML and JSON APIs.
* Reimplementation of the parser in Python, as the old Ruby parser was partly broken and relied on a deprecated HTML parsing library (hpricot). The new parser is completely different, as it doesn't need to do any HTML parsing, so it only relies on the Python standard library.

@jdorweiler, I haven't included the removal of special characters from PR #94 because I think it'd break some descriptions. Examples: https://pypi.python.org/pypi/02exercicio/1.0.0 and https://pypi.python.org/pypi/genius/3.1.4 -- doesn't encoding the output file in UTF8 address the issue fixed by PR #94? Is there a way for me to test it locally?

Also, as noted in README.md and fetch.sh, is there a better way of speeding up the download than the solution I used? Downloading the JSONs sequentially is painfully slow.

Finally, I wasn't sure whether it's worth including the long package descriptions in the abstract, because according to https://duck.co/duckduckhack/fathead_overview it should be a single sentence. It's now easy to add the description and other fields if people think it'd be useful. Personally, I think that things like the number of downloads and last release date would also be useful (potentially more than the long description).