Shopify/brigade — pull request #23 by aybabtme (2014-08-22 17:41:00+00:00)
Title: Chunked get, multipart put

Brigade artifacts are large (~2GB) and get/put operations on them consistently timeout or fail in progress. This PR introduce two things:

* A chunk getter, which does range requests with retries per chunk. Each retry starts from where it stopped last. The getter works pretty well, achieving 800Mbps on a 1Gbps link and consistently fetching large objects.

* A multipart putter, which uses the multipart API of S3. The implementation in `goamz` already does retries per part, but the operation still often fail. Thus I use small parts (10mb) and retry the multipart put as a whole many times.

I deployed this SHA in prod and it solved the issues of brigade failing to fetch and upload its artifacts.

r: @camilo @fbogsany @shuhaowu
cc: @Shopify/webscale