scottohara/tvmanager — Issue #30: Confirm if Content-MD5 check would fail for #multipart_upload
Author: scottohara
Created: 2011-12-12 06:08:26+00:00
State: open
Labels: Waiting

The awk-sdk tries to intelligently decide whether to upload an object to S3 as a single upload, or split it up and use the #multipart_upload method to upload chunks.

(My understanding is that the min. chunk size is 5Mb, so it would probably take close to 50k episodes to reach that point....so we're a fair way off this being an issue).

Currently we're passing the Content-MD5 header as part of the upload request, which AWS uses to compare against the hash of the stored file, to make sure that the file was received and stored OK. We're also double-checking the etag returned from the upload (belt & braces...).

Question is: when #multipart_upload is used, each part has their own MD5 hash, so will the checking (at either end) fail if the aws-sdk call suddenly decides to switch from single upload to multipart?

If so, we could force it to always use single upload (eg. :single_request => true); but it would be nice to know exactly whether this would be necessary.