boto/boto — pull request #2671 by NicolasTr (2014-10-10 10:41:05+00:00)
Title: Client side encryption

Overview
-----------
This pull request add client-side encryption in boto using AES 256 in CBC mode.

It uses a process called [envelope encryption](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html) (Each file is encrypted using a different envelope key. The envelope key is encrypted with the master key and stored in the metadata. The master key can be rotated.). It is meant to be compatible with the [implementation provided in the AWS SDK for Java](http://aws.amazon.com/articles/2850096021478074) (this means that you can encrypt with java and decrypt with python or the other way around).

Some functionalities are voluntarily left out for further pull requests:

- Key pairs. At the moment, only AES symmetric keys of 128, 192 and 256 bits are supported. This wouldn't be a big change but it will require some extra compatibility testing with the java implementation.
- Multipart upload. Some extra code and tests have to be added to handle the padding properly. Please note that CBC is not suitable for parallel uploading.
- [Client-Side Authenticated Encryption](http://java.awsblog.com/post/Tx3305Q4J6AVNQK/Amazon-S3-Client-Side-Authenticated-Encryption). This uses AES-GCM. Besides from the extra integrity check, I think this could be used to for parallel uploading.

There's one related pull-request, #1453.

Usage
-------

To enable the feature, you just need to instantiate you connection with your key. That's it.

```python
# Zero-key, please use your own key
encryption_key = ''.join([chr(0)] * (256 / 8))

connection = S3Connection(client_side_encryption_key=encryption_key)
bucket = connection.get_bucket('your-bucket')
key = bucket.new_key('your-key')
key.set_contents_from_filename('your-file')
# Thi file is now in S3, encrypted

content = key.get_contents_as_string()

```

Your buckets can contain encrypted and plain text files. The unencrypted files will be accessible as usual. Any new or updated file will be transparently encrypted when sent to S3.

The encryption is transparent. For example, the attributes `key.md5` and `key.size` won't change (even if the encrypted file will be up to 16 bytes larger)

It is also efficient:

- The file will be process at most onet extra time to get it's encrypted md5.
- And it's processed on-the-fly, which will greatly limit the memory usage of the encryption.

Implementation
-------------------

The encryption is done using `pycrypto` which it already a requirement of boto (indirectly, via `paramiko`).

I added two private classes, `_AESEncryptor` and `_AESDecryptor` that implement partially the interface of [`RawIOBase`](https://docs.python.org/2/library/io.html#io.RawIOBase). They are used:

- to replace the file pointer when sending a file
- in conjunction with `_Delegator` to replace the response when receiving a file

As AES is operating in blocks of 16 bytes, some padding bytes have to be added at the end of the files. Each byte of the padding will contain it's size. The size of the padding will always be `16 - len(file) % 16` (so always at least one byte). This is the convention followed by the java implementation.

Testing
---------

I added two kinds of integration tests:

- Correctness tests (`S3ClientSideEncryptionTest`). They test what's described above with different key sizes, different content (to force the encryption in one or many `read()`, to use different sizes of padding), and different methods of the `Key` object.
- Cross-compatibility tests (`S3ClientSideEncryptionTestJavaImplementationCompatibility`). They test that files encrypted with java can be downloaded with python and vice versa. They use the same combinations of keys and contents as the correctness tests. Tu run these tests, you need to install `psutil` and `py4j`. I added some code to download the AWS SDK for java and it's dependencies. And a java file...

I tested on Linux Mint Debian using python 2.7.5. The code hasn't been tested with Google Cloud Storage.