aws/aws-cli — pull request #386 by jamesls (2013-10-01 00:19:51+00:00)
Title: Improve and verify memory utilization for s3 transfers

This PR changes the uploads and downloads to always stream directly from a file rather than reading the entire contents in memory, essentially this: https://github.com/jamesls/aws-cli/compare/memory-utilization#diff-27d1c90fc02eac56b64acbd0745c53d6L269, along with all the supporting code necessary to make this work. This included:

* changing etag verification to be based on a fileobj rather than a string.
* adding a `ReadFileChunk` abstraction that allows us to only read a section of a file (used for `UploadPart` in multipart uploads)

This also adds integration tests that verify the memory usage during an upload. This can be seen here: https://github.com/jamesls/aws-cli/compare/memory-utilization#diff-ee8189eeaaa621c78b2559200247ea23R383

The calculated max memory allowed for a multipart upload/ranged download is that number of threads times the chunk size times some runtime margin to account for the python VM + botocore + supporting library overhead (set to 1.5 for now). The idea here is to just detect whether we're using a fixed amount of memory of whether memory is growing with respect to the file size.