mortardata/common_crawl_index — pull request #1 by packerjs (2013-02-11 16:58:43+00:00)
Title: jpacker_remote_copy

Script to lookup webpages from a domain or list of domains from the index and copy them to S3

Example usage:

chmod +x bin/remote_copy
export AWS_ACCESS_KEY=
export AWS_SECRET_KEY=
bin/remote_copy check "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus"
bin/remote_copy copy "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus" --bucket mortar-example-output-data --key common_crawl/blogs_crawl --parallel 4