How to Store Encrypted Files on AWS S3
By: Nasi Jofche
Tags: Encryption, AWS, Security, Programming, Python

How to Store Encrypted Files on AWS S3

Photo by Chris Barbalis on Unsplash

AWS S3 storage offers four ways of server-side data encryption:

SSE-S3, where the encryption keys are managed by AWS.

SSE-KMS, where the encryption keys are managed by AWS KMS, offering control over the key rotation policy.

SSE-C, where the user/uploader provides the encryption key.

Client-side encryption, which uses the S3 encryption client.

Suppose that we are building a web application that would store data on S3, and that data should be encrypted using a private key that would be only known to the application.

In this scenario, we would utilize SSE-C data encryption, where the application is responsible for creating and maintaining private keys for data encryption, and the objects stored on S3 are encrypted with the keys provided by the application.

Any other application would not be permitted to access an encrypted file without the original key used for encryption. In this way, only the owner of the encryption key can retrieve the file.

The SSE-C encryption workflow is quite simple and I will try to explain it in some steps:

The application creates and stores a private key — this could be a random 32-byte string. The application can store an encrypted object on S3 using an SDK of choice by providing the private key to the PUT request for storing the object. Note that this request must be performed over HTTPS, as your application is transferring sensitive data. The S3 bucket receives the object to store and the encryption key. It uses the key to encrypt the data and then deletes the key. Now the encrypted data is stored on S3. To retrieve the data, the application must provide the private key that was initially used for encryption. If the same key is not provided, the encrypted data cannot be retrieved.

Let’s set up a quick demo to demonstrate this workflow.

First of all, I am going to use OpenSSL to generate an AES256 encryption key. The following command generates an encryption key along with the initialization vector (IV) derived from the provided password.

openssl enc -nosalt -aes-256-cbc -k mypassword -P Output: key=89E01536AC207279409D4DE1E5253E01F4A1769E696DB0D6062CA9B8F56767C8iv =EE99333010B23C01E6364E035E97275C

To quickly demonstrate the encryption/decryption process, I will create a test file, called simple_file.txt with the following content in it:

This is the original data for the simple file

Then I will try to encrypt it using the generated key and IV by using the following command:

openssl enc -nosalt

-aes-256-cbc

-in simple_file.txt

-out simple_file.txt.enc

-base64

-K 89E01536AC207279409D4DE1E5253E01F4A1769E696DB0D6062CA9B8F56767C8

-iv EE99333010B23C01E6364E035E97275C

The encrypted contents should look like the following. When the file will be uploaded on S3, along with an encryption key, it will be stored in an encrypted form like this.

tqcd/5VEOzAVd0Cqi6FN7aGXronMdy2Zf7MPOUr5nNDGA3adPlP8reBDjf3iacja

To do this, I will only use the generated IV since a 32-byte string is required as a private key header when uploading an object to S3.

You could use an SDK of your choice — in this example, I will demonstrate the workflow using the Boto 3 Python library for interaction with AWS services.

First of all, you need to install theAWS CLI (you can use the latest version, v2 by following the instructions in this link).

After installing the CLI, you need to create an IAM user with programmatic access as shown in the following image, to obtain the Access Key ID and Secret Access Key.

The Access Key ID and the Secret Key would be used to configure the AWS CLI in your terminal using the aws configure command, which would ask you to enter the obtained keys.

After configuring the CLI, you are ready to use the Boto 3 library to interact with the services that your newly-created user has access to.

The following code snippet shows how to upload a local file, in my case, the simple_file.txt to an S3 bucket, by performing a PUT request along with the previously generated private key:

The required headers to be included in a request for SSE-C are the following:

x-amz-server-side​-encryption​-customer-algorithm , which is used to specify the encryption algorithm. In our case, it would be AES256.

, which is used to specify the encryption algorithm. In our case, it would be AES256. x-amz-server-side​-encryption​-customer-key , which is used to specify the private key that would be used for encryption.

In the Boto 3 library, we can specify these headers in the PUT request using the keys SSECustomerAlgorithm and SSECustomerKey respectively.

After the execution of the script, the file is uploaded to the S3 bucket, and if I try to open it directly from the AWS UI, I will not be able to read its contents, as I am not providing the original key that was used for encryption.

If I try to retrieve the object through the CLI, again without specifying the original encryption key, I should receive an error message as well.

aws s3api get-object --bucket secure-cc --key encrypted.txt encrypted_downloaded.txt Output: An error occurred (InvalidRequest) when calling the GetObject operation: The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.

The only way to obtain the encrypted file would be to provide the original encryption key to the request in the header --sse-customer-key , as shown in the following snippet:

aws s3api get-object --bucket secure-cc --sse-customer-algorithm AES256 --sse-customer-key EE99333010B23C01E6364E035E97275C --key encrypted.txt encrypted_downloaded.txt Output: {

"AcceptRanges": "bytes",

"LastModified": "2020-03-01T00:16:38+00:00",

"ContentLength": 46,

"ETag": "\"3120cb523320824e1f665f7698ea3e73\"",

"ContentType": "binary/octet-stream",

"Metadata": {},

"SSECustomerAlgorithm": "AES256",

"SSECustomerKeyMD5": "5UTC1gbVVUKFd6J/MClx9Q=="

}

Thank you for reading. I hope you found this post interesting and useful for any future project related to data encryption on AWS S3.

Take care.