Shopify/brigade — pull request #6 by aybabtme (2014-06-19 22:42:01+00:00)
Title: Add sync command

## Problem

Syncing S3 buckets is long, it's hard to know how long it will take or what fails, what works.

## Solution

Reading a list of S3 keys from a file, we can track how far in the process we are and when it should be done. The work of sync'ing can also be distributed over many workers, since we can slice an S3 keys listing over many subfiles.

Reading and syncing at most N keys at a time means that the long process of syncing all the keys never use more than O(n) memory, which prevents OOMs.

## Implementation

Command `sync` reads the keys from a file and sync them from a __source__ bucket to a __destination__ bucket. Because S3 often errors, we retry keys using exponential backoff, if they gave _retryable_ errors:

* Network errors
* S3 errors such as _SlowDown_, _Timeout_, _ServiceUnavailable_ and such.

The sync command provides pluggable method for how the sync is to be done:

- PutCopy
- FetchPut intra AWS
- FetchPut, AWS -> GCS

Right now the only implementation is for PutCopy. The buckets can possibly be created with different AWS credentials, in which case PutCopy is impossible.

Test coverage is:

```bash
$ go test ./... -cover
ok	github.com/Shopify/brigade/cmd/list	2.973s	coverage: 93.8% of statements
ok	github.com/Shopify/brigade/cmd/sync	1.543s	coverage: 78.3% of statements
ok	github.com/Shopify/brigade/s3mock	0.171s	coverage: 74.5% of statements
?	github.com/Shopify/brigade/s3util	[no test files]
```

Missing from the coverage are abort conditions and many errors, thus the relatively low (~78%) coverage.

It looks like this:

```insert pretty gif here```

r: waiting for 3 other PRs to be LGTM'd.