jpacker_remote_copy

Script to lookup webpages from a domain or list of domains from the index and copy them to S3

Example usage:

    chmod +x bin/remote_copy
    export AWS_ACCESS_KEY=<your aws access key>
    export AWS_SECRET_KEY=<your aws secret key>
    bin/remote_copy check "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus"
    bin/remote_copy copy "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus" --bucket mortar-example-output-data --key common_crawl/blogs_crawl --parallel 4