trivio/common_crawl_index — pull request #11 by packerjs (2013-03-01 21:06:20+00:00)
Title: remote_copy script to copy pages looked-up with the index to your s3 bucket

Hi, this pull request adds a script "remote_copy", which given a domain or list of domains, looks them up in the index, downloads the relevant webpage bytes from the Crawl on S3, concatenates them back into larger files, and reuploads them to an S3 location of your choosing. It uses python multiprocessing and a cache of boto s3 key objects to increase performance.

A note with example usage is added to the bottom of README.md