trivio/common_crawl_index — pull request #16 by wiseman (2013-03-25 19:02:10+00:00)
Title: Cleanup, re-structuring and installation

This isn't really a pull request, but more of a heads up in case you are interested in merging.

My fork has the following changes:

* Style & de-linting changes to be compliant with PEP8, PEP257, and the Google Python style guidelines.
* Uses setuptools so it can be installed with `python setup.py install`. I've also added it to PyPi so you can do `pip install commoncrawlindex` or `easy_install commoncrawlindex`.
* Combines the `read` and `remote_read` scripts into a single `cci_lookup` script that can be given a `--index_path` argument to an s3:// URI or a local file path.
* Adds a `cci_fetch` script that can download the contents of URLs using the index.
* Moves your README to index-structure.md and has a new user-oriented README.

I don't have a local copy of the index so I wasn't able to test `cci_lookup` in local mode.

I used the `cci_` prefix for the executables to prevent name collisions if they're installed globally with `pip` etc.