Repository: localstack/localstack

Issue:
S3 Put/Get Object is Collapsing XML, causes MD5 Digest validation errors
Loving this tool so far! I recently ran into an issue where it seems that performing a Put or a Get operation against my local S3 bucket is collapsing XML-like structures. This in turn causes MD5 digest validation errors.

I have an S3 bucket...
```
aws --endpoint-url=http://localhost:4572 s3api create-bucket --bucket my-bucket
```

I start with a file called **test.xml**:
```
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl"?>
<submission>
    <field1>fwefwefewfew</field1>
    <field2>fwefwefwe</field2>
    <field3>fefwefwe</field3>
</submission>
```

I then put this object into S3 using the following command:
```
 aws --endpoint-url=http://localhost:4572 s3api put-object --bucket my-bucket --key data/test1 --body test.xml
```

I'll grab the file back using the following command:
```
aws --endpoint-url=http://localhost:4572 s3api get-object --bucket my-bucket --key data/test1 test-back.xml
```

The file now looks like this:
```
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl"?>
<submission><field1>fwefwefewfew</field1><field2>fwefwefwe</field2><field3>fefwefwe</field3></submission>
```

I found this while troubleshooting the following exception stack:
```
com.amazonaws.SdkClientException: Unable to verify integrity of data download.  Client calculated content hash didn't match hash calculated by Amazon S3.  The data may be corrupt.
	at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79)
	at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61)
	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at com.amazonaws.util.IOUtils.toByteArray(IOUtils.java:44)
	at com.amazonaws.util.IOUtils.toString(IOUtils.java:58)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectAsString(AmazonS3Client.java:1485)
```

I was able to work around this bug when I collapsed the XML structure myself prior to adding the file to S3. This is not ideal though.

Any ideas on why this is happening? I am using localstack version 0.8.8, installed with pip.

Thanks for any help!



Discussion:
Got the same problem. When uploading XML localstack seems to strip whitespaces automatically.
I have the same problem, the XML is "normalized". An interesting thing is that the file is padded with binary zeroes to get the same size as the original file.
the issue seems still there in current latest version (0.10.5)

Resolving patch:
diff --git a/localstack/services/s3/s3_listener.py b/localstack/services/s3/s3_listener.py
--- a/localstack/services/s3/s3_listener.py
+++ b/localstack/services/s3/s3_listener.py
@@ -446,6 +446,20 @@ def expand_redirect_url(starting_url, key, bucket):
     return redirect_url
 
 
+def is_bucket_specified_in_domain_name(path, headers):
+    host = headers.get('host', '')
+    return re.match(r'.*s3(\-website)?\.([^\.]+\.)?amazonaws.com', host)
+
+
+def is_object_specific_request(path, headers):
+    """ Return whether the given request is specific to a certain S3 object.
+        Note: the bucket name is usually specified as a path parameter,
+        but may also be part of the domain name! """
+    bucket_in_domain = is_bucket_specified_in_domain_name(path, headers)
+    parts = len(path.split('/'))
+    return parts > (1 if bucket_in_domain else 2)
+
+
 def normalize_bucket_name(bucket_name):
     bucket_name = bucket_name or ''
     # AWS appears to automatically convert upper to lower case chars in bucket names
@@ -761,17 +775,19 @@ def return_response(self, method, path, data, headers, response):
                     if param_name in query_map:
                         response.headers[header_name] = query_map[param_name][0]
 
-            # We need to un-pretty-print the XML, otherwise we run into this issue with Spark:
-            # https://github.com/jserver/mock-s3/pull/9/files
-            # https://github.com/localstack/localstack/issues/183
-            # Note: yet, we need to make sure we have a newline after the first line: <?xml ...>\n
             if response_content_str and response_content_str.startswith('<'):
                 is_bytes = isinstance(response._content, six.binary_type)
+                response._content = response_content_str
 
                 append_last_modified_headers(response=response, content=response_content_str)
 
-                # un-pretty-print the XML
-                response._content = re.sub(r'([^\?])>\n\s*<', r'\1><', response_content_str, flags=re.MULTILINE)
+                # We need to un-pretty-print the XML, otherwise we run into this issue with Spark:
+                # https://github.com/jserver/mock-s3/pull/9/files
+                # https://github.com/localstack/localstack/issues/183
+                # Note: yet, we need to make sure we have a newline after the first line: <?xml ...>\n
+                # Note: make sure to return XML docs verbatim: https://github.com/localstack/localstack/issues/1037
+                if method != 'GET' or not is_object_specific_request(path, headers):
+                    response._content = re.sub(r'([^\?])>\n\s*<', r'\1><', response_content_str, flags=re.MULTILINE)
 
                 # update Location information in response payload
                 response._content = self._update_location(response._content, bucket_name)
diff --git a/localstack/services/s3/s3_starter.py b/localstack/services/s3/s3_starter.py
--- a/localstack/services/s3/s3_starter.py
+++ b/localstack/services/s3/s3_starter.py
@@ -18,6 +18,9 @@
 # max file size for S3 objects (in MB)
 S3_MAX_FILE_SIZE_MB = 2048
 
+# temporary state
+TMP_STATE = {}
+
 
 def check_s3(expect_shutdown=False, print_error=False):
     out = None
@@ -56,19 +59,41 @@ def init(self, name, value, storage='STANDARD', etag=None,
     original_init = s3_models.FakeKey.__init__
     s3_models.FakeKey.__init__ = init
 
+    def s3_update_acls(self, request, query, bucket_name, key_name):
+        # fix for - https://github.com/localstack/localstack/issues/1733
+        #         - https://github.com/localstack/localstack/issues/1170
+        acl_key = 'acl|%s|%s' % (bucket_name, key_name)
+        acl = self._acl_from_headers(request.headers)
+        if acl:
+            TMP_STATE[acl_key] = acl
+        if not query.get('uploadId'):
+            return
+        bucket = self.backend.get_bucket(bucket_name)
+        key = bucket and self.backend.get_key(bucket_name, key_name)
+        if not key:
+            return
+        acl = acl or TMP_STATE.pop(acl_key, None) or bucket.acl
+        if acl:
+            key.set_acl(acl)
+
     def s3_key_response_post(self, request, body, bucket_name, query, key_name, *args, **kwargs):
         result = s3_key_response_post_orig(request, body, bucket_name, query, key_name, *args, **kwargs)
-        if query.get('uploadId'):
-            # fix for https://github.com/localstack/localstack/issues/1733
-            key = self.backend.get_key(bucket_name, key_name)
-            acl = self._acl_from_headers(request.headers) or self.backend.get_bucket(bucket_name).acl
-            key.set_acl(acl)
+        s3_update_acls(self, request, query, bucket_name, key_name)
         return result
 
     s3_key_response_post_orig = s3_responses.S3ResponseInstance._key_response_post
     s3_responses.S3ResponseInstance._key_response_post = types.MethodType(
         s3_key_response_post, s3_responses.S3ResponseInstance)
 
+    def s3_key_response_put(self, request, body, bucket_name, query, key_name, headers, *args, **kwargs):
+        result = s3_key_response_put_orig(request, body, bucket_name, query, key_name, headers, *args, **kwargs)
+        s3_update_acls(self, request, query, bucket_name, key_name)
+        return result
+
+    s3_key_response_put_orig = s3_responses.S3ResponseInstance._key_response_put
+    s3_responses.S3ResponseInstance._key_response_put = types.MethodType(
+        s3_key_response_put, s3_responses.S3ResponseInstance)
+
 
 def main():
     setup_logging()


Test patch:
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -151,6 +151,26 @@ def test_s3_multipart_upload_with_small_single_part(self):
         self.sqs_client.delete_queue(QueueUrl=queue_url)
         self._delete_bucket(TEST_BUCKET_WITH_NOTIFICATION, [key_by_path])
 
+    def test_s3_multipart_upload_acls(self):
+        bucket_name = 'test-bucket-%s' % short_uid()
+        self.s3_client.create_bucket(Bucket=bucket_name, ACL='public-read')
+
+        def check_permissions(key, expected_perms):
+            grants = self.s3_client.get_object_acl(Bucket=bucket_name, Key=key)['Grants']
+            grants = [g for g in grants if 'AllUsers' in g.get('Grantee', {}).get('URI', '')]
+            self.assertEquals(len(grants), 1)
+            permissions = grants[0]['Permission']
+            permissions = permissions if isinstance(permissions, list) else [permissions]
+            self.assertEquals(len(permissions), expected_perms)
+
+        # perform uploads (multipart and regular) and check ACLs
+        self.s3_client.put_object(Bucket=bucket_name, Key='acl-key0', Body='something')
+        check_permissions('acl-key0', 1)
+        self._perform_multipart_upload(bucket=bucket_name, key='acl-key1')
+        check_permissions('acl-key1', 1)
+        self._perform_multipart_upload(bucket=bucket_name, key='acl-key2', acl='public-read-write')
+        check_permissions('acl-key2', 2)
+
     def test_s3_presigned_url_upload(self):
         key_by_path = 'key-by-hostname'
         queue_url, queue_attributes = self._create_test_queue()
@@ -583,8 +603,8 @@ def _delete_bucket(self, bucket_name, keys):
         self.s3_client.delete_bucket(Bucket=bucket_name)
 
     def _perform_multipart_upload(self, bucket, key, data=None, zip=False, acl=None):
-        acl = acl or 'private'
-        multipart_upload_dict = self.s3_client.create_multipart_upload(Bucket=bucket, Key=key, ACL=acl)
+        kwargs = {'ACL': acl} if acl else {}
+        multipart_upload_dict = self.s3_client.create_multipart_upload(Bucket=bucket, Key=key, **kwargs)
         uploadId = multipart_upload_dict['UploadId']
 
         # Write contents to memory rather than a file.