boto/boto — pull request #977 by jamesls (2012-09-10 23:02:27+00:00)
Title: Allow upload_archive to accept file like objects

The entire file is never read into memory, only single
chunks are read at a time.

This greatly reduces the memory usage, but it also appears to have sped
up performance for single operation uploads. In testing various
archive sizes, I saw roughly 20% performance improvements.

A few notes about the changes:

* I changed `x-amz-content-length` to `Content-Length`. From the
docs: http://docs.amazonwebservices.com/amazonglacier/latest/dev/api-common-request-headers.html
I couldn't find any info on this header, and if a Content-Length
is not provided, boto.connection.HTTPRequest would try to
`str(len(body))` to calculate this.
* Sigv4 assumes the body is a string. I modified the payload method
to support being passed file like objects and computing the hash
by reading in configurable chunk sizes at a time. This does
mean that the sha256 of the body is being computed twice, once for the
x-amz-content-sha256 header needed by glacier, and once for the
signature. We *could* avoid doing this twice by checking if that header
exists when we're signing the request, and to just use
x-amz-content-sha256, but that seems like we're crossing abstraction
boundaries. However, if you guys want this, I can add it.