Shopify/brigade — pull request #1 by aybabtme (2014-06-11 19:05:13+00:00)
Title: Add a command to dump the listing of an S3 bucket.

## Problem:

Syncing two large buckets takes forever. It's hard to know just how long it will take because we don't know how many keys are in a bucket to sync, and there's no [API call](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) on AWS that will get us this information quickly (the recommended way is to recursively list the bucket).

For instance, the standard `aws` tool crashes during sync of a large bucket. There is no way to know how far in the sync process it failed.

Because `aws s3` fails and is slow, we want to write our own sync tool. We need to be able to tell if a bucket sync succeeded and know which keys failed. For that we need to know the keys of the source bucket. There the standard `aws s3` tool also offers a way to do this:

```
aws s3 ls s3://shopify/s --recursive
```

However, it is too slow to be usable for buckets containing many millions of keys.

## Solution:

This command will list an S3 bucket in parallel. For a large bucket (such as our bucket), this will take about 40min on a `c3.8xlarge` (32 cores, 10 Gbps), maxing out the network capacity. The result of the listing will be a gzip'd file containing each [s3.Key](http://godoc.org/github.com/crowdmob/goamz/s3#Key) serialized to JSON. All the keys are newline separated.

This file can then be consumed by a second command that syncs all the keys it contains (this command is in a separate PR). With this file as a ground truth of the sync process, we gain visibility on the quality of our backup and can take informed corrective actions.

r: @boourns, @bouk , @camilo
cc: @benbjohnson , @snormore , @mkobetic , @fbogsany