Repository: localstack/localstack

Issue:
CORB problem from S3 service
We are having problems playing audio files from the HTML5 audio component from the S3 service.

Steps to reproduce it:

1.- Upload some mp3 file to S3 service
2.- Use signed url in some HTML5 audio component like
```
    <audio controls="controls">
        <source src="http://localstack:4572/local-bucket/test.mp3?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-Token=xxx%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx%2F20190416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190416T150211Z&X-Amz-SignedHeaders=host&X-Amz-Expires=172800&X-Amz-Signature=xxx" type="audio/mpeg">
    </audio>
```

When we try to reproduce it we get CORB problem from Chrome,

```
Cross-Origin Read Blocking (CORB) blocked cross-origin response http://localstack:4572/local-bucket/test.mp3?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-Token=xxx%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx%2F20190416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190416T150211Z&X-Amz-SignedHeaders=host&X-Amz-Expires=172800&X-Amz-Signature=xxx with MIME type text/html. See https://www.chromestatus.com/feature/5629709824032768 for more details.
```

Doing some test we realize that the http calls done by audio component to audio asset returns a wrong Content-Type header.

First call,

Request,
```
Request URL: http://localstack:4572/local-bucket/test.mp3?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-Token=xxx%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx%2F20190416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190416T150211Z&X-Amz-SignedHeaders=host&X-Amz-Expires=172800&X-Amz-Signature=xxx
Request Method: GET
Status Code: 200 OK
Remote Address: xx.xx.xx.xx:4572
Referrer Policy: no-referrer-when-downgrade
```

Response,
```
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Length: 6584528
Content-Md5: RpfskSSCIEXm7I9lGuMsBA==
Content-Type: audio/mpeg
Date: Wed, 17 Apr 2019 14:27:05 GMT
ETag: "4697ec9124822045e6ec8f651ae32c04"
last-modified: Wed, 17 Apr 2019 14:24:23 GMT
Server: BaseHTTP/0.3 Python/2.7.15
```

Second call, with a partial content,

Request,
```
Request URL: http://localstack:4572/local-bucket/test.mp3?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-Token=xxx%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx%2F20190416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190416T150211Z&X-Amz-SignedHeaders=host&X-Amz-Expires=172800&X-Amz-Signature=xxx
Request Method: GET
Status Code: 206 Partial Content
Remote Address: xx.xx.xx.xx:4572
Referrer Policy: no-referrer-when-downgrade
```

Response,
```
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Length: 6584528
Content-Range: bytes 0-6584527/6584528
Content-Type: text/html; charset=utf-8
Date: Wed, 17 Apr 2019 14:27:05 GMT
Server: BaseHTTP/0.3 Python/2.7.15
```

Here, we are getting Content-Type: text/html and this causes the CORB problem.

Trying the same from a real S3 service:

First call,

Request,

```
Request URL: https://xxx.s3.amazonaws.com/xxx.mp3
Request Method: GET
Status Code: 304 Not Modified
Remote Address: xx.xx.xx.xx:443
Referrer Policy: no-referrer-when-downgrade
```

Response,

```
Date: Wed, 17 Apr 2019 14:34:37 GMT
ETag: "5d79d70934b79a527383c87a7d22dc3a"
Last-Modified: Wed, 17 Apr 2019 10:21:17 GMT
Server: AmazonS3
x-amz-id-2: 1tX9Ni8GfNVZNBrK7KtdamP+6fxeoTIo0pMzLlLMGiIoAGEM3qpm12UoPktkjDxLiLWj16v2hbo=
x-amz-meta-gid: 33
x-amz-meta-mode: 33277
x-amz-meta-mtime: 1555496476
x-amz-meta-uid: 33
x-amz-request-id: 9AA70366CB6D9C73
x-amz-version-id: null
```

Second call, with partial content,

Request,

```
Request URL: https://xxx.s3.amazonaws.com/xxx.mp3
Request Method: GET
Status Code: 206 Partial Content
Remote Address: xx.xx.xx.xx:443
Referrer Policy: no-referrer-when-downgrade
```

Response,

```
Accept-Ranges: bytes
Content-Length: 5952573
Content-Range: bytes 0-5952572/5952573
Content-Type: audio/mpeg
Date: Wed, 17 Apr 2019 14:34:37 GMT
ETag: "5d79d70934b79a527383c87a7d22dc3a"
Last-Modified: Wed, 17 Apr 2019 10:21:17 GMT
Server: AmazonS3
x-amz-id-2: xxx
x-amz-meta-gid: 33
x-amz-meta-mode: 33277
x-amz-meta-mtime: 1555496476
x-amz-meta-uid: 33
x-amz-request-id: 9AA70366CB6D9C73
x-amz-version-id: null
```

In summary, when a partial content request is made it is always served as text/html and this generates the CORB problem.

We also try to set a CORS policy including `X-Content-Type-Options: nosniff` header via ExposeHeaders in order to prevent CORB check but the service doesn't allow this header.

We believe that the solution could be to send the correct `Content-Type` header or in case this is impossible, always include the `X-Content-Type-Options: nosniff` header in all the responses of the S3 service.


Discussion:
same here. would be great if we get a fix
Thanks for the detailed report @ganeko @qasimzee . Do you think you could provide a small example - e.g., a sequence of `awslocal` commands, to help reproduce this issue? Thanks
@whummer it’s there in the steps to reproduce on top. Basically:
1. You upload a file
2. Use it in video or audio tag
@qasimzee Sure, those are the high-level steps. :) But I guess that's the part I'd be more interested in:
```
1. You upload a file
```
There are many ways to upload a file, how exactly is this being done? How is the content type `audio/mpeg` determined, and passed along as a header to the S3 API? Which tool, SDK function, or CLI command is being used to upload the file?

Maybe I'm missing something, but you could make the maintainers' lives a lot easier if a small programmatic sample could be provided to reproduce the concrete steps. Thanks /cc @ganeko
> We believe that the solution could be to send the correct Content-Type header

@ganeko This is in fact possible in the latest version - please note the `--content-type` in the example below:
```
$ awslocal s3 mb s3://test
$ awslocal s3 cp --content-type 'audio/mpeg' test.mp3 s3://test/test.mp3
$ curl -sD - localhost:4572/test/test.mp3 -o /dev/null
HTTP/1.1 200 OK
Server: BaseHTTP/0.6 Python/3.7.4
Content-Type: audio/mpeg
...
```

Closing this issue as "cannot reproduce" - please report here if the problem persists or you cannot get this resolved by specifying the object content type on upload. Thanks
Hi @whummer ,

You can reproduce the problem following this example,

1.- Upload a test mp3 file (setting or not the content-type, the problem is the same)

```
aws s3 mb s3://test --endpoint-url=http://localhost:4572
aws s3 cp test.mp3 s3://test/test.mp3 --endpoint-url=http://localhost:4572
```

2.- Requesting the entire file the content type works fine

```
curl -X HEAD -i http://localhost:4572/test/test.mp3\?AWSAccessKeyId\=XXXXX\&Signature\=xxx\&Expires\=1551891961   
```

> HTTP/1.1 200 OK
> Server: BaseHTTP/0.6 Python/3.7.5
> Date: Mon, 09 Dec 2019 12:32:57 GMT
> Content-Type: audio/mpeg
> ...

3.- Requesting partial content (as the browsers do with HTML 5 audio and video tags) you get the wrong content type

```
curl --header "Range: bytes=0-20" -i http://localhost:4572/test/test.mp3\?AWSAccessKeyId\=XXXXX\&Signature\=xxx\&Expires\=1551891961 
```

> HTTP/1.1 206 Partial Content
> Server: BaseHTTP/0.6 Python/3.7.5
> Date: Mon, 09 Dec 2019 12:36:58 GMT
> content-range: bytes 0-20/6584528
> Content-Type: text/html; charset=utf-8
> Content-Length: 21
> ...

And this causes the CORB problem.
I have the same issue. Audio & video file
By the way, the problem is with tag audio and video with Chrome only (CORB) feature security.
With Firefox, it works.
Here a rugh example
// Server side (Nodejs)
const getUploadSignedUrl = (filename, contentType) => {
  const s3 = new aws.S3({
    accessKeyId: process.env.accessKeyId,
    secretAccessKey: process.env.secretAccessKey,
    endpoint: 'http://localhost:4572',
    s3ForcePathStyle: true
  });

  const url = s3.getSignedUrl('putObject', {
    Bucket: process.env.bucket,
    Key: fileName,
    ContentType: contentType
  });
  console.log('url', url);
  return url;
};

return getUploadSignedUrl('my-video.mp4', 'video/mp4'); // example

// Client Side
export async function uploadFile (file) {
  const url = await httpPost(`${ENDPOINT_URL}/getUploadSignedUrl`, {
    fileName: file.name,
    contentType: file.type
  })

  // Put the file there
  fetch(url, {
    method: 'PUT',
    body: file
  })
};

// the html 
< video controls playsinline>
  < source src="http://localhost:4572/bucket-name/my-video.mp4" type="video/mp4" / >
</ video >

I think you don't define "content-type" when we get the file and by default it's "text/html".
In any case, in good practices, you must always return the content-type when you get a file.
And here, the content-type is in the parameters of the url so you can store it with the file.

Resolving patch:
diff --git a/localstack/services/s3/s3_listener.py b/localstack/services/s3/s3_listener.py
--- a/localstack/services/s3/s3_listener.py
+++ b/localstack/services/s3/s3_listener.py
@@ -354,6 +354,19 @@ def fix_location_constraint(response):
         remove_xml_preamble(response)
 
 
+def fix_range_content_type(bucket_name, path, headers, response):
+    # Fix content type for Range requests - https://github.com/localstack/localstack/issues/1259
+    if 'Range' not in headers:
+        return
+
+    s3_client = aws_stack.connect_to_service('s3')
+    key_name = get_key_name(path, headers)
+    result = s3_client.head_object(Bucket=bucket_name, Key=key_name)
+    content_type = result['ContentType']
+    if response.headers.get('Content-Type') == 'text/html; charset=utf-8':
+        response.headers['Content-Type'] = content_type
+
+
 def remove_xml_preamble(response):
     """ Removes <?xml ... ?> from a response content """
     response._content = re.sub(r'^<\?[^\?]+\?>', '', response._content)
@@ -556,16 +569,28 @@ def normalize_bucket_name(bucket_name):
     return bucket_name
 
 
+def get_key_name(path, headers):
+    parsed = urlparse.urlparse(path)
+    path_parts = parsed.path.lstrip('/').split('/', 1)
+
+    if uses_path_addressing(headers):
+        return path_parts[1]
+    return path_parts[0]
+
+
+def uses_path_addressing(headers):
+    host = headers['host']
+    return host.startswith(HOSTNAME) or host.startswith(HOSTNAME_EXTERNAL)
+
+
 def get_bucket_name(path, headers):
     parsed = urlparse.urlparse(path)
 
     # try pick the bucket_name from the path
     bucket_name = parsed.path.split('/')[1]
 
-    host = headers['host']
-
-    # is the hostname not starting a bucket name?
-    if host.startswith(HOSTNAME) or host.startswith(HOSTNAME_EXTERNAL):
+    # is the hostname not starting with a bucket name?
+    if uses_path_addressing(headers):
         return normalize_bucket_name(bucket_name)
 
     # matches the common endpoints like
@@ -585,6 +610,7 @@ def get_bucket_name(path, headers):
 
     # if any of the above patterns match, the first captured group
     # will be returned as the bucket name
+    host = headers['host']
     for pattern in [common_pattern, dualstack_pattern, legacy_patterns]:
         match = pattern.match(host)
         if match:
@@ -886,6 +912,7 @@ def return_response(self, method, path, data, headers, response):
             append_last_modified_headers(response=response)
             append_list_objects_marker(method, path, data, response)
             fix_location_constraint(response)
+            fix_range_content_type(bucket_name, path, headers, response)
 
             # Remove body from PUT response on presigned URL
             # https://github.com/localstack/localstack/issues/1317


Test patch:
diff --git a/tests/integration/test_s3.py b/tests/integration/test_s3.py
--- a/tests/integration/test_s3.py
+++ b/tests/integration/test_s3.py
@@ -244,23 +244,30 @@ def test_s3_put_presigned_url_metadata(self):
         # clean up
         self._delete_bucket(bucket_name, [object_key])
 
-    def test_s3_get_response_content_type_same_as_upload(self):
+    def test_s3_get_response_content_type_same_as_upload_and_range(self):
         bucket_name = 'test-bucket-%s' % short_uid()
         self.s3_client.create_bucket(Bucket=bucket_name)
 
         # put object
-        object_key = 'key-by-hostname'
+        object_key = '/foo/bar/key-by-hostname'
+        content_type = 'foo/bar; charset=utf-8'
         self.s3_client.put_object(Bucket=bucket_name,
                                   Key=object_key,
-                                  Body='something',
-                                  ContentType='text/html; charset=utf-8')
+                                  Body='something ' * 20,
+                                  ContentType=content_type)
         url = self.s3_client.generate_presigned_url(
             'get_object', Params={'Bucket': bucket_name, 'Key': object_key}
         )
 
         # get object and assert headers
         response = requests.get(url, verify=False)
-        self.assertEqual(response.headers['content-type'], 'text/html; charset=utf-8')
+        self.assertEqual(response.headers['content-type'], content_type)
+
+        # get object using range query and assert headers
+        response = requests.get(url, headers={'Range': 'bytes=0-18'}, verify=False)
+        self.assertEqual(response.headers['content-type'], content_type)
+        self.assertEqual(to_str(response.content), 'something something')
+
         # clean up
         self._delete_bucket(bucket_name, [object_key])