Repository: localstack/localstack

Issue:
S3 ListAllMyBucketsResult response CreationDate Invalid format
can someone please advise  here-

I am using the S3 service on the latest localstack version,and using the Amazon's S3 Java SDK to perform all the operations with Java 8 (Additional configuration info : aws-sdk-java/1.11.542 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23 java/1.8.0_60 scala/2.10.5 groovy/2.4.10 vendor/Oracle_Corporation)

Although creating a new bucket seems to working as expected, retrieving the buckets via list buckets operation throws the following exception through the SDK:
```
 java.lang.IllegalArgumentException: Invalid format: "2019-12-23 05:12:26.039447" is malformed at " 05:12:26.039447"). Response Code: 200, Response Text: OK

response-

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01"><Owner><ID>bcaf1ffd86f41161ca5fb16fd081034f</ID><DisplayName>webfile</DisplayName></Owner><Buckets><Bucket><Name>tst-adapter-us-west-2</Name><CreationDate>2019-12-23 02:17:43.014814</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

trace-

Caused by:
        java.lang.IllegalArgumentException: Invalid format: "2019-12-23 05:12:26.039447" is malformed at " 05:12:26.039447"
            at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
            at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826)
            at com.amazonaws.util.DateUtils.doParseISO8601Date(DateUtils.java:98)
            at com.amazonaws.util.DateUtils.parseISO8601Date(DateUtils.java:77)
            at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListAllMyBucketsHandler.doEndElement(XmlResponsesSaxParser.java:996)
            at com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)
            at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2973)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
            at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
            at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
            at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
            at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
            at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:152)
            ... 31 more
```


Discussion:
I am encountering a similar issue while listing objects in S3

```
    Expected
        <*awserr.requestError | 0xc0003ea640>: {
            awsError: {
                code: "SerializationError",
                message: "failed to decode REST XML response",
                errs: [
                    {
                        Layout: "2006-01-02T15:04:05Z",
                        Value: "2019-12-24 13:08:13.979175",
                        LayoutElem: "T",
                        ValueElem: " 13:08:13.979175",
                        Message: "",
                    },
                ],
            },
            statusCode: 200,
            requestID: "",
            bytes: nil,
        }
    to be nil
```


```
SerializationError: failed to decode REST XML response
	status code: 200, request id:
caused by: parsing time "2019-12-24 13:08:13.979175" as "2006-01-02T15:04:05Z": cannot parse " 13:08:13.979175" as "T"
```
Same here; this is happening for me when trying out the new [0.10.6](https://github.com/localstack/localstack/releases/tag/v0.10.6) release. It does not happen with the 0.10.5 image, so perhaps there's a regression?

Based on [where the failure is occurring within Testcontainers' test suite](https://github.com/testcontainers/testcontainers-java/blob/master/modules/localstack/src/test/java/org/testcontainers/containers/localstack/LocalstackContainerTest.java#L65) it seems that this problem afflicts Java clients (tested with AWS Java SDK `1.11.683` and `1.11.699`) but _not_ AWS CLI (tested with `aws-cli/1.16.185 Python/2.7.15 Linux/4.9.184-linuxkit botocore/1.12.175`).


I think the problem is probably in a recent-ish change to `moto-ext`: https://github.com/spulec/moto/commit/ab0853cddc6bf9a0789999751a5d698186b8ebd0#diff-f28b3d402b1d050e247f518f3b94865bR1313

The `creation_date` value is, as far as I can tell, just string-formatted in the templated XML, resulting in a string that is not in the proper ISO8601 format:
```
>>> a = datetime.datetime.utcnow()
>>> str(a)
'2020-01-02 16:36:09.523540'
```

I'll raise an issue in the `moto-ext` repo...
I can relate the same issue with localstack 0.10.6 using the AWS Java SDK

Resolving patch:
diff --git a/localstack/services/s3/s3_listener.py b/localstack/services/s3/s3_listener.py
--- a/localstack/services/s3/s3_listener.py
+++ b/localstack/services/s3/s3_listener.py
@@ -394,26 +394,33 @@ def fix_delete_objects_response(bucket_name, method, parsed_path, data, headers,
 def fix_metadata_key_underscores(request_headers={}, response=None):
     # fix for https://github.com/localstack/localstack/issues/1790
     underscore_replacement = '---'
-    meta_header_prefix = 'x-amz-meta'
+    meta_header_prefix = 'x-amz-meta-'
+    prefix_len = len(meta_header_prefix)
     updated = False
     for key in list(request_headers.keys()):
         if key.lower().startswith(meta_header_prefix):
-            key_new = key.replace('_', underscore_replacement)
+            key_new = meta_header_prefix + key[prefix_len:].replace('_', underscore_replacement)
             if key != key_new:
                 request_headers[key_new] = request_headers.pop(key)
                 updated = True
     if response:
         for key in list(response.headers.keys()):
             if key.lower().startswith(meta_header_prefix):
-                key_new = key.replace(underscore_replacement, '_')
+                key_new = meta_header_prefix + key[prefix_len:].replace(underscore_replacement, '_')
                 if key != key_new:
                     response.headers[key_new] = response.headers.pop(key)
     return updated
 
 
+def fix_creation_date(method, path, response):
+    if method != 'GET' or path != '/':
+        return
+    response._content = re.sub(r'([0-9])</CreationDate>', r'\1Z</CreationDate>', to_str(response._content))
+
+
 def remove_xml_preamble(response):
     """ Removes <?xml ... ?> from a response content """
-    response._content = re.sub(r'^<\?[^\?]+\?>', '', response._content)
+    response._content = re.sub(r'^<\?[^\?]+\?>', '', to_str(response._content))
 
 
 # --------------
@@ -961,6 +968,7 @@ def return_response(self, method, path, data, headers, response):
             fix_range_content_type(bucket_name, path, headers, response)
             fix_delete_objects_response(bucket_name, method, parsed, data, headers, response)
             fix_metadata_key_underscores(response=response)
+            fix_creation_date(method, path, response=response)
 
             # Remove body from PUT response on presigned URL
             # https://github.com/localstack/localstack/issues/1317
diff --git a/localstack/utils/common.py b/localstack/utils/common.py
--- a/localstack/utils/common.py
+++ b/localstack/utils/common.py
@@ -916,8 +916,9 @@ def all_exist(*files):
     cert.gmtime_adj_notAfter(2 * 365 * 24 * 60 * 60)
     cert.set_issuer(cert.get_subject())
     cert.set_pubkey(k)
+    alt_names = b'DNS:localhost,DNS:test.localhost.atlassian.io,IP:127.0.0.1'
     cert.add_extensions([
-        crypto.X509Extension(b'subjectAltName', False, b'DNS:localhost,IP:127.0.0.1'),
+        crypto.X509Extension(b'subjectAltName', False, alt_names),
         crypto.X509Extension(b'basicConstraints', True, b'CA:false'),
         crypto.X509Extension(b'keyUsage', True, b'nonRepudiation,digitalSignature,keyEncipherment'),
         crypto.X509Extension(b'extendedKeyUsage', True, b'serverAuth')


Test patch:
diff --git a/localstack/ext/java/src/test/java/cloud/localstack/S3FeaturesTest.java b/localstack/ext/java/src/test/java/cloud/localstack/S3FeaturesTest.java
--- a/localstack/ext/java/src/test/java/cloud/localstack/S3FeaturesTest.java
+++ b/localstack/ext/java/src/test/java/cloud/localstack/S3FeaturesTest.java
@@ -15,6 +15,7 @@
 
 import org.apache.commons.io.IOUtils;
 import org.apache.http.*;
+import org.apache.http.conn.ssl.*;
 import org.apache.http.client.*;
 import org.apache.http.client.methods.*;
 import org.apache.http.entity.*;
@@ -29,7 +30,7 @@
 import cloud.localstack.docker.annotation.LocalstackDockerProperties;
 
 @RunWith(LocalstackTestRunner.class)
-@LocalstackDockerProperties(services = {"s3"})
+@LocalstackDockerProperties(services = {"s3"}, ignoreDockerRunErrors=true)
 public class S3FeaturesTest {
 
 	/**
@@ -102,11 +103,10 @@ public void testMetadata() {
 		String bucketName = UUID.randomUUID().toString();
 		s3.createBucket(bucketName);
 
-		String keyWithUnderscores = "__key1";
-		String keyWithDashes = keyWithUnderscores.replace("_", "-");
-
 		Map<String, String> originalMetadata = new HashMap<String, String>();
-		originalMetadata.put(keyWithUnderscores, "val1");
+		originalMetadata.put("key1", "val1");
+		originalMetadata.put("key_2", "val2");
+		originalMetadata.put("__key3", "val3");
 
 		ObjectMetadata objectMetadata = new ObjectMetadata();
 		objectMetadata.setUserMetadata(originalMetadata);
@@ -119,12 +119,7 @@ public void testMetadata() {
 
 		Map<String, String> receivedMetadata = objectMetadataResponse.getUserMetadata();
 
-		Map<String, String> actualResult = new HashMap<String, String>();
-		actualResult.put(keyWithDashes, "val1");
-
-		// TODO: We currently have a bug that converts underscores in metadata keys to dashes.
-		// See here for details: https://github.com/localstack/localstack/issues/459
-		Assert.assertTrue(receivedMetadata.equals(originalMetadata) || receivedMetadata.equals(actualResult) );
+		Assert.assertEquals(originalMetadata, receivedMetadata);
 	}
 
 	@Test
@@ -179,7 +174,11 @@ public void test() throws Exception {
 		String content = "test content";
 		HttpPut httpPut = new HttpPut(presignedUrl.toString());
     httpPut.setEntity(new StringEntity(content));
-		CloseableHttpClient httpclient = HttpClients.createDefault();
+		SSLContextBuilder builder = new SSLContextBuilder();
+		builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
+		SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build());
+		CloseableHttpClient httpclient = HttpClients.custom().setSSLSocketFactory(sslsf).build();
+
 		httpclient.execute(httpPut);
 		httpclient.close();
 
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -252,7 +252,7 @@ def test_s3_put_metadata_underscores(self):
 
         # put object
         object_key = 'key-with-metadata'
-        metadata = {'test_meta_1': 'foo'}
+        metadata = {'test_meta_1': 'foo', '__meta_2': 'bar'}
         self.s3_client.put_object(Bucket=bucket_name, Key=object_key, Metadata=metadata, Body='foo')
         metadata_saved = self.s3_client.head_object(Bucket=bucket_name, Key=object_key)['Metadata']
         self.assertEqual(metadata, metadata_saved)