Shopify/brigade — pull request #5 by aybabtme (2014-06-19 22:21:38+00:00)
Title: Add slice command, breaks a big S3 key listing into sublistings.

## Problem

When working with brigade, files containing S3 key listings are often huge and thus take forever to read fully. Also, being one monolithic file means the work on the listing can only be done on one machine at a time.

## Solution

A `slice` command to break a bucket key listing into subfiles.

## Implementation

The key list files are JSON objects separated by `\n` characters. So the slicing operation:

* Reads the source file line by line.
* Write each line to a subfile in a round-robbin fashion.

The implementation is trivial/naive, it could be more performant using sequential reads and writes to one file at a time. However, testing with _1 reader 0 writer_, or _1 reader, then 1 writer_ shows no noticeable performance gain; the bulk of the time spent seems to be in gzip processing.

There's no way around the gzip, since bucket list files take ~1.5GB compressed, and last time I tried to decompress the file, I stopped the decompression at ~50GB.

Test coverage is:

```bash
$ go test ./... -cover
ok	github.com/Shopify/brigade/cmd/slice	0.040s	coverage: 83.9% of statements
```

It looks like this:

![slice-demo](https://cloud.githubusercontent.com/assets/1189716/3334441/f0af909c-f7ff-11e3-9e93-0e08f826a207.gif)

r: waiting for 2 other PRs to be LGTM'd.