laurilehmijoki/s3_website — pull request #88 by pjanik (2014-05-02 14:16:40+00:00)
Title: Improve performance of the push operation

These changes improve performance of push operation (in particular the diff calculation). See https://github.com/laurilehmijoki/s3_website/issues/44.

Main changes:

1. Use `ignore_on_server` to optimise diff.
2. Use [GET Bucket](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) instead of separate HEAD requests.
3. Use optimised version of filey-diff gem, which:
* doesn't send HEAD and GET requests during data sources comparison.
* has optimised some slow methods (e.g. select_in_outer_array).
4. MD5 of gzipped S3 objects is now calculated from gzipped content (instead of raw, non-gzipped like previously), so e.g. each time when you update your `gzip` config option and push, all the files affected by this setting will be updated. We can consider it as a breaking change or kind of bug fix.

3 is [a separate pull request to filey-diff](https://github.com/laurilehmijoki/filey-diff/pull/10).

I also recorded almost all VCR tests again and did minor changes in their configuration. New tests are also added:
- spec for gzip_helper
- large site update (2100 files, as [GET Bucket](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) is limited to 1000 objects max)
- update of site after `gzip` config change (what's related to 4.)

I didn't update gem version, as I'm 100% not sure how. These changes are mostly about performance, but it also depends on how we treat change in behaviour described in 4.

We are already testing this branch to deploy a site that consists of about 15,000 files and it seems to work fine - previously the diff operation couldn't be completed due to performance issues.