Repository: getmoto/moto

Issue:
Unable to include Access-Control-Expose-Header = "ETag" in CORS response
Hi everyone! I'm using Moto to mock an S3 bucket from an end to end test for a web application. Our tests use something similar to Selenium to control the browser so we're interacting with the Moto standalone server via HTTP.

Moto version: 2.2.2 (Moto standalone server)
How I installed it:
1. sudo apt install python-pip
2. sudo -H pip install "moto[s3]"
3. pip install flask
4. pip install flask_cors

Environment: pipenv with Python 3.8.0 running on an Ubuntu 18.04.5 LTS machine.

The issue is that when uploading a file to the bucket, Moto fails because it can't find the ETag. I've found that the browser refuses to set the ETag on the request because it hasn't received an ExposeHeader on the CORS response.

Here are the steps I'm following:
1. Starting Moto with `moto_server s3 -s -H 0.0.0.0`
2. Creating the bucket with Amazon's S3 SDK in PHP:
 
![Screenshot at 2021-08-25 16-08-50](https://user-images.githubusercontent.com/4132918/130850714-ac556115-052c-433b-bcaf-5bec0103946e.png)

3. Adding the CORS configuration to the bucket. For some reason Amazon's SDK is stripping the "ExposedHeader" from the PutBucketCors (I added some prints to check and it's not reaching Moto) so for now I'm sending it via CURL. Here's the request and how Moto shows the rule in /moto-api:

![postman_curl](https://user-images.githubusercontent.com/4132918/130851307-2eef6353-9081-4abf-af98-1430af525330.png)
![moto_api](https://user-images.githubusercontent.com/4132918/130851323-56eac57d-e24d-4866-97df-0c210d5ed074.png)

4. Uploading the file from the web application that we want to test. It uses EvaporateJS to communicate with Amazon, so I changed it to point to Moto. Here's the preflight request:

![preflight](https://user-images.githubusercontent.com/4132918/130851722-959607ab-eb83-4b9f-861b-df63cf214c15.png)

You can see that there's no Expose-Header on the response, so on the next request the browser refuses to set the ETag. This causes Moto to fail with the following stack:

```10.0.6.1 - - [25/Aug/2021 19:18:30] "POST /mytestbucket/temp/8a6dc338bf4d974cbc68c204f1e42f16586b9fd454dc2a57995e920fe989ff30?uploadId=LwLkW061R0WHC0Jsgx1xhZWsfXeCpfvY5kycVGgUnhKUD5UtIaqsB5n2JQ HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/werkzeug/serving.py", line 319, in run_wsgi
    execute(self.server.app)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/werkzeug/serving.py", line 308, in execute
    application_iter = app(environ, start_response)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/server.py", line 181, in __call__
    return backend_app(environ, start_response)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 2088, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.handle_exception(e)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/core/utils.py", line 148, in __call__
    result = self.callback(request, request.url, {})
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/s3/responses.py", line 1005, in key_or_control_response
    response = self._key_response(request, full_url, headers)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/s3/responses.py", line 1176, in _key_response
    return self._key_response_post(request, body, bucket_name, query, key_name)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/s3/responses.py", line 1792, in _key_response_post
    value, etag = multipart.complete(body)
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/s3/models.py", line 314, in complete
    for pn, etag in body:
  File "/home/sb-3/.local/share/virtualenvs/moto-ol9WsJxn/lib/python3.8/site-packages/moto/s3/responses.py", line 1766, in _complete_multipart_body
    yield (pn, p.getElementsByTagName("ETag")[0].firstChild.wholeText) 
AttributeError: 'NoneType' object has no attribute 'wholeText' 
```

I've searched the existing issues but found nothing quite matching this. I can avoid the problem entirely if I just change the part of moto/server.py (line 222) that adds CORS with flask_cors from `CORS(backend_app)` to `CORS(backend_app, expose_headers="ETag")` but I suspect that it isn't the right way to go.

It looks like the CORS rule I set isn't being applied when responding to the preflight request. I'm not sure if this is an actual bug, I don't have a ton of experience with HTTP and CORS so I might be missing something. Any help is greatly appreciated.

Thank you all in advance.



Discussion:
Hi @dcappellini, looks like that is a bug indeed. We persist the CORS configuration, but then don't actually do anything with the `exposed_headers` argument.

> 
>  I can avoid the problem entirely if I just change the part of moto/server.py (line 222) that adds CORS with flask_cors from CORS(backend_app) to CORS(backend_app, expose_headers="ETag") but I suspect that it isn't the right way to go.
> 

It's a workaround 🙂 But ideally we only add this to the S3 responses, and only the headers that are set. Should be doable, just a bit more work involved - there's no way of handling CORS-specific logic on a method/service level atm.


Resolving patch:
diff --git a/moto/s3/models.py b/moto/s3/models.py
--- a/moto/s3/models.py
+++ b/moto/s3/models.py
@@ -1015,8 +1015,8 @@ def set_cors(self, rules):
             assert isinstance(rule.get("AllowedHeader", []), list) or isinstance(
                 rule.get("AllowedHeader", ""), str
             )
-            assert isinstance(rule.get("ExposedHeader", []), list) or isinstance(
-                rule.get("ExposedHeader", ""), str
+            assert isinstance(rule.get("ExposeHeader", []), list) or isinstance(
+                rule.get("ExposeHeader", ""), str
             )
             assert isinstance(rule.get("MaxAgeSeconds", "0"), str)
 
@@ -1034,8 +1034,8 @@ def set_cors(self, rules):
                     rule["AllowedMethod"],
                     rule["AllowedOrigin"],
                     rule.get("AllowedHeader"),
-                    rule.get("ExposedHeader"),
-                    rule.get("MaxAgeSecond"),
+                    rule.get("ExposeHeader"),
+                    rule.get("MaxAgeSeconds"),
                 )
             )
 
diff --git a/moto/s3/responses.py b/moto/s3/responses.py
--- a/moto/s3/responses.py
+++ b/moto/s3/responses.py
@@ -1,6 +1,7 @@
 import io
 import os
 import re
+from typing import List, Union
 
 from botocore.awsrequest import AWSPreparedRequest
 
@@ -258,9 +259,10 @@ def ambiguous_response(self, request, full_url, headers):
     def bucket_response(self, request, full_url, headers):
         self.method = request.method
         self.path = self._get_path(request)
-        self.headers = request.headers
-        if "host" not in self.headers:
-            self.headers["host"] = urlparse(full_url).netloc
+        # Make a copy of request.headers because it's immutable
+        self.headers = dict(request.headers)
+        if "Host" not in self.headers:
+            self.headers["Host"] = urlparse(full_url).netloc
         try:
             response = self._bucket_response(request, full_url, headers)
         except S3ClientError as s3error:
@@ -315,6 +317,8 @@ def _bucket_response(self, request, full_url, headers):
             return self._bucket_response_delete(body, bucket_name, querystring)
         elif method == "POST":
             return self._bucket_response_post(request, body, bucket_name)
+        elif method == "OPTIONS":
+            return self._bucket_response_options(bucket_name)
         else:
             raise NotImplementedError(
                 "Method {0} has not been implemented in the S3 backend yet".format(
@@ -342,6 +346,64 @@ def _bucket_response_head(self, bucket_name, querystring):
             return 404, {}, ""
         return 200, {}, ""
 
+    def _set_cors_headers(self, bucket):
+        """
+        TODO: smarter way of matching the right CORS rule:
+        See https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors.html
+
+        "When Amazon S3 receives a preflight request from a browser, it evaluates
+        the CORS configuration for the bucket and uses the first CORSRule rule
+        that matches the incoming browser request to enable a cross-origin request."
+        This here just uses all rules and the last rule will override the previous ones
+        if they are re-defining the same headers.
+        """
+
+        def _to_string(header: Union[List[str], str]) -> str:
+            # We allow list and strs in header values. Transform lists in comma-separated strings
+            if isinstance(header, list):
+                return ", ".join(header)
+            return header
+
+        for cors_rule in bucket.cors:
+            if cors_rule.allowed_methods is not None:
+                self.headers["Access-Control-Allow-Methods"] = _to_string(
+                    cors_rule.allowed_methods
+                )
+            if cors_rule.allowed_origins is not None:
+                self.headers["Access-Control-Allow-Origin"] = _to_string(
+                    cors_rule.allowed_origins
+                )
+            if cors_rule.allowed_headers is not None:
+                self.headers["Access-Control-Allow-Headers"] = _to_string(
+                    cors_rule.allowed_headers
+                )
+            if cors_rule.exposed_headers is not None:
+                self.headers["Access-Control-Expose-Headers"] = _to_string(
+                    cors_rule.exposed_headers
+                )
+            if cors_rule.max_age_seconds is not None:
+                self.headers["Access-Control-Max-Age"] = _to_string(
+                    cors_rule.max_age_seconds
+                )
+
+        return self.headers
+
+    def _bucket_response_options(self, bucket_name):
+        # Return 200 with the headers from the bucket CORS configuration
+        self._authenticate_and_authorize_s3_action()
+        try:
+            bucket = self.backend.head_bucket(bucket_name)
+        except MissingBucket:
+            return (
+                403,
+                {},
+                "",
+            )  # AWS S3 seems to return 403 on OPTIONS and 404 on GET/HEAD
+
+        self._set_cors_headers(bucket)
+
+        return 200, self.headers, ""
+
     def _bucket_response_get(self, bucket_name, querystring):
         self._set_action("BUCKET", "GET", querystring)
         self._authenticate_and_authorize_s3_action()
@@ -1034,9 +1096,10 @@ def key_or_control_response(self, request, full_url, headers):
         # Key and Control are lumped in because splitting out the regex is too much of a pain :/
         self.method = request.method
         self.path = self._get_path(request)
-        self.headers = request.headers
-        if "host" not in self.headers:
-            self.headers["host"] = urlparse(full_url).netloc
+        # Make a copy of request.headers because it's immutable
+        self.headers = dict(request.headers)
+        if "Host" not in self.headers:
+            self.headers["Host"] = urlparse(full_url).netloc
         response_headers = {}
 
         try:
diff --git a/moto/server.py b/moto/server.py
--- a/moto/server.py
+++ b/moto/server.py
@@ -18,7 +18,7 @@
 import moto.backend_index as backend_index
 from moto.core.utils import convert_flask_to_httpretty_response
 
-HTTP_METHODS = ["GET", "POST", "PUT", "DELETE", "HEAD", "PATCH"]
+HTTP_METHODS = ["GET", "POST", "PUT", "DELETE", "HEAD", "PATCH", "OPTIONS"]
 
 
 DEFAULT_SERVICE_REGION = ("s3", "us-east-1")


Test patch:
diff --git a/tests/test_s3/test_server.py b/tests/test_s3/test_server.py
--- a/tests/test_s3/test_server.py
+++ b/tests/test_s3/test_server.py
@@ -153,7 +153,10 @@ def test_s3_server_post_unicode_bucket_key():
 
 
 def test_s3_server_post_cors():
+    """Test default CORS headers set by flask-cors plugin"""
     test_client = authenticated_client()
+    # Create the bucket
+    test_client.put("/", "http://tester.localhost:5000/")
 
     preflight_headers = {
         "Access-Control-Request-Method": "POST",
@@ -167,7 +170,6 @@ def test_s3_server_post_cors():
     assert res.status_code in [200, 204]
 
     expected_methods = set(["DELETE", "PATCH", "PUT", "GET", "HEAD", "POST", "OPTIONS"])
-    assert set(res.headers["Allow"].split(", ")) == expected_methods
     assert (
         set(res.headers["Access-Control-Allow-Methods"].split(", ")) == expected_methods
     )
@@ -178,3 +180,60 @@ def test_s3_server_post_cors():
     res.headers.should.have.key("Access-Control-Allow-Headers").which.should.equal(
         "origin, x-requested-with"
     )
+
+
+def test_s3_server_post_cors_exposed_header():
+    """Test that we can override default CORS headers with custom bucket rules"""
+    # github.com/spulec/moto/issues/4220
+
+    cors_config_payload = """<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+  <CORSRule>
+    <AllowedOrigin>https://example.org</AllowedOrigin>
+    <AllowedMethod>HEAD</AllowedMethod>
+    <AllowedMethod>GET</AllowedMethod>
+    <AllowedMethod>PUT</AllowedMethod>
+    <AllowedMethod>POST</AllowedMethod>
+    <AllowedMethod>DELETE</AllowedMethod>
+    <AllowedHeader>*</AllowedHeader>
+    <ExposeHeader>ETag</ExposeHeader>
+    <MaxAgeSeconds>3000</MaxAgeSeconds>
+  </CORSRule>
+</CORSConfiguration>
+    """
+
+    test_client = authenticated_client()
+    preflight_headers = {
+        "Access-Control-Request-Method": "POST",
+        "Access-Control-Request-Headers": "origin, x-requested-with",
+        "Origin": "https://localhost:9000",
+    }
+    # Returns 403 on non existing bucket
+    preflight_response = test_client.options(
+        "/", "http://testcors.localhost:5000/", headers=preflight_headers
+    )
+    assert preflight_response.status_code == 403
+
+    # Create the bucket
+    test_client.put("/", "http://testcors.localhost:5000/")
+    res = test_client.put(
+        "/?cors", "http://testcors.localhost:5000", data=cors_config_payload
+    )
+    assert res.status_code == 200
+
+    cors_res = test_client.get("/?cors", "http://testcors.localhost:5000")
+    assert b"<ExposedHeader>ETag</ExposedHeader>" in cors_res.data
+
+    preflight_response = test_client.options(
+        "/", "http://testcors.localhost:5000/", headers=preflight_headers
+    )
+    assert preflight_response.status_code == 200
+    expected_cors_headers = {
+        "Access-Control-Allow-Methods": "HEAD, GET, PUT, POST, DELETE",
+        "Access-Control-Allow-Origin": "https://example.org",
+        "Access-Control-Allow-Headers": "*",
+        "Access-Control-Expose-Headers": "ETag",
+        "Access-Control-Max-Age": "3000",
+    }
+    for header_name, header_value in expected_cors_headers.items():
+        assert header_name in preflight_response.headers
+        assert preflight_response.headers[header_name] == header_value