boto/boto — pull request #2716 by NicolasTr (2014-10-29 15:17:00+00:00)
Title: Client side encryption 2: multipart upload

Overview
-----------

This pull request adds the support of multipart uploads with client-side encryption. It is based of the work done in #2671.

Multipart upload without client-side encryption provides [the following advantages](http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html):
> 1. **Improved throughput**—You can upload parts in parallel to improve throughput.
> 2. **Quick recovery from any network issues**—Smaller part size minimizes the impact of restarting a failed upload due to a network error.
> 3. **Pause and resume object uploads**—You can upload object parts over time. Once you initiate a multipart upload there is no expiry; you must explicitly complete or abort the multipart upload.
> 4. **Begin an upload before you know the final object size**—You can upload an object as you are creating it.

The advantages `2.`, `3.` and `4.` will still exist with client-side encryption but not `1.` because the current implementation uses [CBC](http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29). This could be improved in the future by using GCM (see [counter mode](http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_.28CTR.29)).

Usage
-------

If you upload all the parts in the same process, you only need to:

* instantiate your connection with an encryption key (see #2671)
* upload the parts in order

Advanced usage
-------------------

By default, you won't be able to upload different parts from different processes because to encrypt a part you need the last block of the previous part, the envelope key and some information about the padding. These values are kept on the connection in the `client_side_encryption_registry`.

You can get around the problem by implementing your own `client_side_encryption_registry`. Example from the tests:
```
class MemcachedDictionary(object):

def __init__(self, memcache_client):
self.memcache_client = memcache_client

def __getitem__(self, key):
return self.memcache_client.get(key)

def get(self, key, default):
if not key in self:
return default
return self[key]

def __setitem__(self, key, value):
self.memcache_client.set(key, value, time=60*60)

def __delitem__(self, key):
self.memcache_client.delete(key)

def keys(self):
# This will prevent the deletion of the data of completed uploads, but should be enough for the tests
return []

def __contains__(self, key):
return self.memcache_client.get(key) is not None

memcache_client = MemcachedDictionary(memcache.Client(['127.0.0.1:11211']))
connection = S3Connection(
client_side_encryption_key=...,
client_side_encryption_registry=memcache_client,
)

...
```

You only need to implement the methods listed above to have a functional `client_side_encryption_registry`.

Testing
---------

I added some tests to make sure that the content is encrypted properly and that the parts can be uploaded from different processes.