Shopify/brigade — pull request #21 by aybabtme (2014-08-18 17:47:20+00:00)
Title: fetch last source to a file, then use that file for diff

Before doing a sync on all the keys, we try to fetch from S3 the last key listing that was sync'd.

The current code uses the `io.Reader` straight from S3 to do the comparison with the current listing. In practice, this stream is large (>1GB) and errors out in the process, leading to failure of the diff and resulting in a full list sync, which is orders of magnitude slower than an incremental sync.

This PR will instead fetch the whole object to a temporary file, then use that file to do the diff.

r: @camilo @fbogsany @shuhaowu
cc: @Shopify/webscale