Repository: aws/aws-cli

Issue:
Requiring a 32 byte ascii key is a bug, it cannot have 256 bits of entropy of a real key.
At this spot here, the AWS CLI requires a 32 byte string as a key.  A true, correctly-generated AES256 key has a one in 18 quintillion chance of being valid base64 (and a similar chance of being ascii).  This is because a true random AES key has each character take on a random value `0..255` uniformly, with 32 such bytes to make 256 bits.  If a true 32 bytes of randomness is converted to base64 to get printable ASCII characters, then truncating the (44) bytes of base64 of the real key to 32 bytes, then 64 bits of the key have effectively been chopped off.  In effect, you have a 192-bit AES key inefficiently represented in 256-bits; not an AES-256 key.

https://github.com/aws/aws-cli/blob/4ff0cbacbac69a21d4dd701921fe0759cf7852ed/awscli/customizations/s3/subcommands.py#L199

The current workaround is to supply a binary argument to `--sse-customer-key=fileb://theKey` to reference a key on disk.  Writing the key to disk may be worse than having a key that is effectively shortened to 192 bits.

In detail, I had started writing about it here:

https://github.com/aws/aws-sdk-go/issues/1726

The real, and backwards compatible, fix is to just look at the length of the key. 
- If it's length 32, then assume that it will be cast to a literal byte array.  In this case, even though an AES256 cipher is used, the key only has 192 bits of entropy.
- If it's 64 bytes and looks like hex, then it is hex (because it will decode to an arbitrary array of 32 bytes.  
- If it looks like base64, then decode it as base64 and verify that it's 32 bytes at that point.  It might be 44 bytes.  Some encoders include spurious `==` on the end.  What is important is that the base64 decode is 32 bytes.


The key is NOT a valid "string" in most languages.  It is a byte array.  In Golang, embedded null chars in strings are ok.  In other languages they are not.  But from Go, Java, etc, it is possible to generate keys with zeroes in them.  It should be possible to use all valid AES keys from the command-line; because real AES keys are completely random, with almost no chance that they will ever be printable ASCII.

Being printable ASCII, be warned that it's an approximately 192-bit key in reality.  ie: No high bit (low ASCII), no embedded nulls in string, no spaces, no tabs, no bell chars, etc.  Base64 is effectively `0..64` per character, rather than `0..255` of full bytes.

> 2^64 is 18quintillion.  Using an AES256 key that is known to be valid printable ASCII speeds up a brute-force search for the key by 18quintillion.  Also, if a user generated a real AES key, there is only a one in 18quintillion chance that it can be represented as valid base64.  So, keys properly generated cannot even be passed in unless they can be base64, or base16 (hexadecimal).  If the key is not length 32, we know unambiguously that we need to decode it.


Discussion:
Yeah, this should be modeled as a bytes type not a string. Thankfully this is only a problem for the high level s3 commands in the CLI.

Resolving patch:
diff --git a/awscli/customizations/s3/subcommands.py b/awscli/customizations/s3/subcommands.py
--- a/awscli/customizations/s3/subcommands.py
+++ b/awscli/customizations/s3/subcommands.py
@@ -197,7 +197,7 @@
 
 
 SSE_C_KEY = {
-    'name': 'sse-c-key',
+    'name': 'sse-c-key', 'cli_type_name': 'blob',
     'help_text': (
         'The customer-provided encryption key to use to server-side '
         'encrypt the object in S3. If you provide this value, '
@@ -234,7 +234,7 @@
 
 
 SSE_C_COPY_SOURCE_KEY = {
-    'name': 'sse-c-copy-source-key',
+    'name': 'sse-c-copy-source-key', 'cli_type_name': 'blob',
     'help_text': (
         'This parameter should only be specified when copying an S3 object '
         'that was encrypted server-side with a customer-provided '


Test patch:
diff --git a/tests/functional/s3/test_cp_command.py b/tests/functional/s3/test_cp_command.py
--- a/tests/functional/s3/test_cp_command.py
+++ b/tests/functional/s3/test_cp_command.py
@@ -12,6 +12,7 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 import mock
+import os
 
 from awscli.testutils import BaseAWSCommandParamsTest
 from awscli.testutils import capture_input, set_invalid_utime
@@ -363,6 +364,86 @@ def test_cp_with_sse_c_flag(self):
              'SSECustomerKeyMD5': 'rL0Y20zC+Fzt72VPzMSk2A=='}
         )
 
+    def test_cp_with_sse_c_fileb(self):
+        file_path = self.files.create_file('foo.txt', 'contents')
+        key_path = self.files.create_file('foo.key', '')
+        with open(key_path, 'wb') as f:
+            f.write(
+                b'K\xc9G\xe1\xf9&\xee\xd1\x03\xf3\xd4\x10\x18o9E\xc2\xaeD'
+                b'\x89(\x18\xea\xda\xf6\x81\xc3\xd2\x9d\\\xa8\xe6'
+            )
+        cmdline = (
+            '%s %s s3://bucket/key.txt --sse-c --sse-c-key fileb://%s' % (
+                self.prefix, file_path, key_path
+            )
+        )
+        self.run_cmd(cmdline, expected_rc=0)
+        self.assertEqual(len(self.operations_called), 1)
+        self.assertEqual(self.operations_called[0][0].name, 'PutObject')
+
+        expected_args = {
+            'Key': 'key.txt', 'Bucket': 'bucket',
+            'ContentType': 'text/plain',
+            'Body': mock.ANY,
+            'SSECustomerAlgorithm': 'AES256',
+            'SSECustomerKey': 'S8lH4fkm7tED89QQGG85RcKuRIkoGOra9oHD0p1cqOY=',
+            'SSECustomerKeyMD5': 'mL8/mshNgBObhAC1j5BOLw=='
+        }
+        self.assertDictEqual(self.operations_called[0][1], expected_args)
+
+    def test_cp_with_sse_c_copy_source_fileb(self):
+        self.parsed_responses = [
+            {
+                "AcceptRanges": "bytes",
+                "LastModified": "Tue, 12 Jul 2016 21:26:07 GMT",
+                "ContentLength": 4,
+                "ETag": '"d3b07384d113edec49eaa6238ad5ff00"',
+                "Metadata": {},
+                "ContentType": "binary/octet-stream"
+            },
+            {
+                "AcceptRanges": "bytes",
+                "Metadata": {},
+                "ContentType": "binary/octet-stream",
+                "ContentLength": 4,
+                "ETag": '"d3b07384d113edec49eaa6238ad5ff00"',
+                "LastModified": "Tue, 12 Jul 2016 21:26:07 GMT",
+                "Body": six.BytesIO(b'foo\n')
+            },
+            {}
+        ]
+
+        file_path = self.files.create_file('foo.txt', '')
+        key_path = self.files.create_file('foo.key', '')
+        with open(key_path, 'wb') as f:
+            f.write(
+                b'K\xc9G\xe1\xf9&\xee\xd1\x03\xf3\xd4\x10\x18o9E\xc2\xaeD'
+                b'\x89(\x18\xea\xda\xf6\x81\xc3\xd2\x9d\\\xa8\xe6'
+            )
+        cmdline = (
+            '%s s3://bucket-one/key.txt s3://bucket/key.txt '
+            '--sse-c-copy-source --sse-c-copy-source-key fileb://%s' % (
+                self.prefix, key_path
+            )
+        )
+        self.run_cmd(cmdline, expected_rc=0)
+        self.assertEqual(len(self.operations_called), 2)
+        self.assertEqual(self.operations_called[0][0].name, 'HeadObject')
+        self.assertEqual(self.operations_called[1][0].name, 'CopyObject')
+
+        expected_args = {
+            'Key': 'key.txt', 'Bucket': 'bucket',
+            'ContentType': 'text/plain',
+            'CopySource': 'bucket-one/key.txt',
+            'CopySourceSSECustomerAlgorithm': 'AES256',
+            'CopySourceSSECustomerKey': (
+                'S8lH4fkm7tED89QQGG85RcKuRIkoGOra9oHD0p1cqOY='
+            ),
+            'CopySourceSSECustomerKeyMD5': 'mL8/mshNgBObhAC1j5BOLw==',
+        }
+        self.assertDictEqual(self.operations_called[1][1], expected_args)
+
+
     # Note ideally the kms sse with a key id would be integration tests
     # However, you cannot delete kms keys so there would be no way to clean
     # up the tests