fog/fog — pull request #776 by ktheory (2012-03-04 18:25:01+00:00)
Title: Simple multipart S3 uploads

To upload files larger than 5GB to S3, you must use multipart uploads.

This is a complicated process of:
1. initializing the upload (keeping track of the upload_id)
2. uploading each part (keeping track of each part's ETag)
3. completing the upload

This adds simple support for multipart uploads to the File model by passing the :multipart_chunk_size parameter.

E.g.
chunk_size = 5242880 # 5mb
directory.files.create(:key => 'large_file', :body => File.open('large_file'), :multipart_chunk_size => chunk_size)

Considerations: each chunk must be read into memory, so small chunk sizes are preferable. Multipart uploads cannot have more than 10k parts, so a 1TB file would need at least a 100MB chunk size.

(I couldn't see how to avoid reading the whole chunk into memory w/out changing Excon's internals to take a file handle w/ a byte offset to start reading.)

Currently each part is uploaded serially. Some applications would benefit from uploading parts in parallel.