Repository: localstack/localstack

Issue:
Underscore converted to hyphen while put it as metadata using amazon-sdk
Hi,
When try to put object with metadata that include underscore(the metadata include underscore) we got the underscore converted to  hyphen.
The same code in Amazon will return the metadata  with underscore - not converted.

for example:
we put map of string as metadata - the "__key1" will convert to "--key1"

```
public class MyAwsS3Tester {

    public static final String IP = "10.0.0.24";
    public static final String BUCKET_NAME = "zanavi-test";

    public static void main(String[] args) {
        AmazonS3 s3 = AmazonS3ClientBuilder.standard()
                .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://" + IP + ":4572/", "us-east-1"))
                .disableChunkedEncoding()
                .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("zanavi", "1234")))
                .build();
        if (s3.doesBucketExistV2(BUCKET_NAME)) {
            System.out.println("bucket zanavi exists");
        }
        else {
            System.out.println("bucket " + BUCKET_NAME + " doesn't exists");
            s3.createBucket(BUCKET_NAME);
        }

        String dummyStr = "dummy-str";

        Map<String, String> myMap = new HashMap<String, String>();
        myMap.put("__key1", "val1");

        ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setUserMetadata(myMap);

        InputStream is = new ByteArrayInputStream(dummyStr.getBytes(StandardCharsets.UTF_8));
        s3.putObject(new PutObjectRequest(BUCKET_NAME, "my-key1", is, objectMetadata));

        S3Object getObj = s3.getObject(new GetObjectRequest(BUCKET_NAME, "my-key1"));
        ObjectMetadata objectMetadataResponse = getObj.getObjectMetadata();

        Map<String, String> myMap1 = objectMetadataResponse.getUserMetadata();

        System.out.println("done " + myMap1);
    }
```


Discussion:
Thanks for reporting @itaimarts . I was able to reproduce this bug. 

Note that S3 metadata values are transmitted via HTTP header values. In your example, the metadata is transmitted using the following header:
```
x-amz-meta-__key1: val1
```

The root cause for this issue is that underscores in HTTP header names have traditionally been a security issue for certain scenarios (you can find more details if you google for "underscore header spoofing in CGI"). Therefore, most web frameworks are stripping underscores from HTTP header names, or ignore these headers entirely.

In our case, LocalStack makes use of [moto](https://github.com/spulec/moto), which uses the [flask](https://github.com/pallets/flask) framework to start local API servers. Flask is based on [werkzeug](https://github.com/pallets/werkzeug) which provides an implementation of [WSGI](https://wsgi.readthedocs.io/en/latest/) (the Python equivalent of CGI). Werkzeug is the culprit in our case. As you can see in [line 139 of serving.py](https://github.com/pallets/werkzeug/blob/master/werkzeug/serving.py#L139), werkzeug creates an environment map by uppercasing all header names to variable names and replacing dashes with underscores:
```
        for key, value in self.headers.items():
            key = key.upper().replace('-', '_')
            ...
```

.. and then later when the environment variables are converted back to headers, underscores are converted back to dashes in [datastructures.py](
https://github.com/pallets/werkzeug/blob/master/werkzeug/datastructures.py#L1360-L1364):
```
        for key, value in iteritems(self.environ):
            ...
                yield (key[5:].replace('_', '-').title(), _unicodify_header_value(value))
```
Given the tricky nature of this bug, I'm going to mark this as "Won't fix". Please feel free to bring up this issue in the upstream repositories (moto/flask/werkzeug) to see if they can come up with a viable solution to pass HTTP headers with underscores to Flask apps. Note that there has been a similar bug report in the past which might be related: https://github.com/pallets/werkzeug/issues/940

At this point, I don't see an easy way we can fix this in LocalStack. I think the best workaround would be to not use underscores at all. If you find out more about this issue, please report back here. Thanks

Resolving patch:
diff --git a/localstack/services/s3/s3_listener.py b/localstack/services/s3/s3_listener.py
--- a/localstack/services/s3/s3_listener.py
+++ b/localstack/services/s3/s3_listener.py
@@ -391,6 +391,26 @@ def fix_delete_objects_response(bucket_name, method, parsed_path, data, headers,
     response._content = xmltodict.unparse({'DeleteResult': result})
 
 
+def fix_metadata_key_underscores(request_headers={}, response=None):
+    # fix for https://github.com/localstack/localstack/issues/1790
+    underscore_replacement = '---'
+    meta_header_prefix = 'x-amz-meta'
+    updated = False
+    for key in list(request_headers.keys()):
+        if key.lower().startswith(meta_header_prefix):
+            key_new = key.replace('_', underscore_replacement)
+            if key != key_new:
+                request_headers[key_new] = request_headers.pop(key)
+                updated = True
+    if response:
+        for key in list(response.headers.keys()):
+            if key.lower().startswith(meta_header_prefix):
+                key_new = key.replace(underscore_replacement, '_')
+                if key != key_new:
+                    response.headers[key_new] = response.headers.pop(key)
+    return updated
+
+
 def remove_xml_preamble(response):
     """ Removes <?xml ... ?> from a response content """
     response._content = re.sub(r'^<\?[^\?]+\?>', '', response._content)
@@ -767,6 +787,9 @@ def forward_request(self, method, path, data, headers):
         # remap metadata query params (not supported in moto) to request headers
         append_metadata_headers(method, query_map, headers)
 
+        # apply fixes
+        headers_changed = fix_metadata_key_underscores(request_headers=headers)
+
         if query == 'notification' or 'notification' in query_map:
             # handle and return response for ?notification request
             response = handle_notification_request(bucket, method, data)
@@ -804,7 +827,7 @@ def forward_request(self, method, path, data, headers):
             if method == 'PUT':
                 return set_object_lock(bucket, data)
 
-        if modified_data is not None:
+        if modified_data is not None or headers_changed:
             return Request(data=modified_data, headers=headers, method=method)
         return True
 
@@ -938,6 +961,7 @@ def return_response(self, method, path, data, headers, response):
             fix_location_constraint(response)
             fix_range_content_type(bucket_name, path, headers, response)
             fix_delete_objects_response(bucket_name, method, parsed, data, headers, response)
+            fix_metadata_key_underscores(response=response)
 
             # Remove body from PUT response on presigned URL
             # https://github.com/localstack/localstack/issues/1317


Test patch:
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -221,7 +221,6 @@ def test_s3_get_response_default_content_type(self):
     def test_s3_put_presigned_url_metadata(self):
         # Object metadata should be passed as query params via presigned URL
         # https://github.com/localstack/localstack/issues/544
-
         bucket_name = 'test-bucket-%s' % short_uid()
         self.s3_client.create_bucket(Bucket=bucket_name)
 
@@ -244,6 +243,22 @@ def test_s3_put_presigned_url_metadata(self):
         # clean up
         self._delete_bucket(bucket_name, [object_key])
 
+    def test_s3_put_metadata_underscores(self):
+        # Object metadata keys should accept keys with underscores
+        # https://github.com/localstack/localstack/issues/1790
+        bucket_name = 'test-%s' % short_uid()
+        self.s3_client.create_bucket(Bucket=bucket_name)
+
+        # put object
+        object_key = 'key-with-metadata'
+        metadata = {'test_meta_1': 'foo'}
+        self.s3_client.put_object(Bucket=bucket_name, Key=object_key, Metadata=metadata)
+        metadata_saved = self.s3_client.head_object(Bucket=bucket_name, Key=object_key)['Metadata']
+        self.assertEqual(metadata, metadata_saved)
+
+        # clean up
+        self._delete_bucket(bucket_name, [object_key])
+
     def test_s3_get_response_content_type_same_as_upload_and_range(self):
         bucket_name = 'test-bucket-%s' % short_uid()
         self.s3_client.create_bucket(Bucket=bucket_name)