boto/boto — pull request #2256 by mraposa (2014-05-07 11:12:36+00:00)
Title: Concurrent S3 Uploader

Although the S3 Boto interface supports a multipart upload, the multipart upload is not done concurrently.

This pull request adds a concurrent multipart upload to S3 Boto. Since the task of uploading a file to S3 is IO bound especially over high latency links, breaking the upload into multiple parts and uploading them concurrenlty provides a big performance boost. On my workstation which is 7ms RTT from US-East, transfer speeds for a 500MB file increased by 2-3x using 10 threads.

There are many methods available for providing similar functionality, e.g. http://www.elastician.com/2010/12/s3-multipart-upload-in-boto.html and http://bcbio.wordpress.com/2011/04/10/parallel-upload-to-amazon-s3-with-python-boto-and-multiprocessing/ However, in each case the source file is split into smaller parts and each part uploaded separately. This technique introduces multiple problems:

1. It only works on operating systems that natively have the split command
2. The splitting process generates additional local read and write IO
3. The additional split files consume additional local disk space
4. There is no easy way to clean up the split files after a failure, e.g. power failure

A better solution is to use the seek() method on a file pointer to upload parts of the file. The file is logically chunked into parts, the part pointers are added to a worker queue, and each worker uploads part of the file. When all parts are uploaded, the multipart upload is closed. No additional local files are generated.

Here is a Github Gist with sample code for using the ConcurrentUploader https://gist.github.com/mraposa/a4fb2d4a526ac5b2a65a

I tried to remain as consistent as possible with the Glacier.concurrent module. In fact most of the code is an exact derivation from that module.

If this pull request is accepted, it may be possible to write a concurrent downloader as well. S3 supports get range requests which is similar to what is used with the Glacier Concurrent Downloader.

FYI – I wrote the original Glacier Concurrent Downloader. Pull request #1123. There have been some very nice improvements made since that original pull request.