mortardata/luigi — pull request #8 by ddaniels888 (2014-10-30 17:35:41+00:00)
Title:  Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3

In order to upload files larger than 5GB to S3 you have to use [S3 Multipart Uploads](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html). It's also recommended for large files < 5GB for reliability.

This pull request adds support using [boto's multi-part upload API](http://boto.readthedocs.org/en/latest/s3_tut.html#storing-large-data), with the inclusion of https://github.com/boto/boto/pull/2704 to avoid external library dependencies.

I have modified `AtomicS3File` to call `put_multipart` instead of `put`, as `put_multipart` will automatically fallback to `put` for files that are smaller than the part size.