Repository: localstack/localstack

Issue:
Edge Router doesn't handle S3 Presigned URL POSTs properly
# Bug Report

# Detailed description
This is a similar issue to #2329, specific to [S3 Presigned URLs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html). Using the edge router port (`:4566`), it is possible to use `generate_presigned_post`, but attempting to use the resultant URL to upload a file to localstack's S3 fails with a 404.

## Expected behavior
Localstack's edge router port should accept POST requests with an S3 Presigned URL (generated from that same Localstack instance) in the same way that the old S3 port (`:4572`) does.

## Actual behavior
While the old S3 port (`:4572`) handles this fine, the edge router responds with a 404 and a message similar to the following:

```
2020-05-24T15:36:54:INFO:localstack.services.edge: Unable to find forwarding rule for host "localhost:4566", path "/local-job-documents", target header "", auth header ""
```

(borrowed from @philippmalkov's comment on #2329 )

```
aws_1  | 2020-05-27T16:24:38:INFO:localstack.services.edge: Unable to find forwarding rule for host "aws:4566", path "/test-bucket", target header "", auth header ""
```

(our observed case with the below setup)

# Steps to reproduce

## Command used to start LocalStack
```
# Used in docker-compose.yml as:
  aws:
    image: localstack/localstack-light
    environment:
      - SERVICES=dynamodb,s3
      - HOSTNAME_EXTERNAL=aws
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
    expose:
      - 4566
      - 4572
    volumes:
      - ./.localstack:/tmp/localstack
```

## Client code (AWS SDK code snippet, or sequence of "awslocal" commands)
Client code is a Django app using `boto3` to generate a Presigned URL and then upload a file to it using `requests`, effectively identical to the examples in [the boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html)


Discussion:
@matthewarmand I think the fix for https://github.com/localstack/localstack/issues/2442 should have fixed this - in my testing presigned URLs work correctly on `localhost:4566` now. This now works for me:

```
$ echo "hello" > test.txt
$ alias awslocal="aws --endpoint-url http://localhost:4566"
$ awslocal s3 mb s3://testbucket
$ url=$(awslocal s3 presign s3://testbucket/test.txt)
$ curl --upload-file test.txt $url
$ awslocal s3 ls s3://testbucket
2020-06-01 10:38:58          6 test.txt
```

It'd be worth re-testing your app with the newest version :) 
@mcintyre94 Thanks for the response and the fix, I noticed that did fix some things, in particular the `PUT` case. However I'm still noticing a bit of weirdness with the `POST` on the edge router.

I've put together a [small Django app](https://github.com/matthewarmand/localstack-2464-example) which illustrates the issue I'm seeing. Anyone should be able to clone that repo and use `docker-compose` to run it locally pretty easily using the instructions in the `README.md`.

It's entirely possible some piece of config is incorrect or I'm missing a parameter or setting, so please let me know if that's the case. But it seems to be that the `POST` functionality is still not quite right on the edge router... Using pretty much the same test case I get successful results POSTing to `4572` and live S3, but still not `4566`.

I've done my best to document that test app so its easy to use and reproduce, let me know if further explanation or context is needed, or if any improvement could be made to make the debugging process easier.
Also out of curiosity, is there an issue or some documentation you can point me to on the rationalization for building the edge router functionality rather than using dedicated ports for each service? For a container whose purpose is to mock AWS services as faithfully as possible, it feels a bit like it adds more complexity and/or room for error than it might be worth, considering AWS services in the wild all live at different URLs / ports anyway.
@matthewarmand Nice repro, I get the same issue - fails on post 4566 only and everything else works. Apologies - I assumed it was fixed because the minimal CLI repro started working, think you've shown that put is fixed but post isn't yet.

To clarify I'm not a maintainer and don't have any special insight, I only commented here because it looked like a dupe of an issue I thought was fixed and I was hoping your issue had been fixed :) I will say that I like the edge port because I do a lot of my experimenting/prototyping in the AWS CLI and being able to just do `alias awslocal="aws --endpoint-url=http://localhost:4566"` and then use any service by just swapping `aws` for `awslocal` is much easier than having to look up the ports for every individual service. It's also nice when I need to talk to multiple AWS services to just use the same endpoint URL for all of them.

FWIW there's this issue where it was initially proposed - https://github.com/localstack/localstack/issues/151
>I only commented here because it looked like a dupe of an issue I thought was fixed and I was hoping your issue had been fixed

I agree that it was definitely related, and was completely reasonable to think it would be fixed, so I appreciate the input. Looks like there's still an issue on the `POST` side (or potentially how I'm trying to use it), so hopefully that can be resolved.

Thanks for the extra context on the edge port too, seeing the original issue is helpful as well. It still seems to me like it should be treated more as convenience than the main workflow, ie the service-specific ports should still be kept around if possible, since its such a divergence from what the container is trying to replicate; but that's just my $0.02 :)

Resolving patch:
diff --git a/localstack/services/edge.py b/localstack/services/edge.py
--- a/localstack/services/edge.py
+++ b/localstack/services/edge.py
@@ -111,6 +111,16 @@ def get_api_from_headers(headers, path=None):
     return result[0], result_before[1] or result[1], path, host
 
 
+def is_s3_form_data(data_bytes):
+    if(to_bytes('key=') in data_bytes):
+        return True
+
+    if(to_bytes('Content-Disposition: form-data') in data_bytes and to_bytes('name="key"') in data_bytes):
+        return True
+
+    return False
+
+
 def get_port_from_custom_rules(method, path, data, headers):
     """ Determine backend port based on custom rules. """
 
@@ -128,8 +138,8 @@ def get_port_from_custom_rules(method, path, data, headers):
         if method == 'PUT':
             # assume that this is an S3 PUT bucket request with URL path `/<bucket>`
             return config.PORT_S3
-        if method == 'POST' and to_bytes('key=') in data_bytes:
-            # assume that this is an S3 POST request with form parameters in the body
+        if method == 'POST' and is_s3_form_data(data_bytes):
+            # assume that this is an S3 POST request with form parameters or multipart form in the body
             return config.PORT_S3
 
     if path == '/' and to_bytes('QueueName=') in data_bytes:


Test patch:
diff --git a/tests/integration/test_edge.py b/tests/integration/test_edge.py
--- a/tests/integration/test_edge.py
+++ b/tests/integration/test_edge.py
@@ -15,6 +15,7 @@ def test_invoke_apis_via_edge(self):
 
         if is_api_enabled('s3'):
             self._invoke_s3_via_edge(edge_url)
+            self._invoke_s3_via_edge_multipart_form(edge_url)
         if is_api_enabled('kinesis'):
             self._invoke_kinesis_via_edge(edge_url)
         if is_api_enabled('dynamodbstreams'):
@@ -66,3 +67,23 @@ def _invoke_s3_via_edge(self, edge_url):
         result = io.BytesIO()
         client.download_fileobj(bucket_name, object_name, result)
         self.assertEqual('file_content_123', to_str(result.getvalue()))
+
+    def _invoke_s3_via_edge_multipart_form(self, edge_url):
+        client = aws_stack.connect_to_service('s3', endpoint_url=edge_url)
+        bucket_name = 'edge-%s' % short_uid()
+        object_name = 'testobject'
+        object_data = b'testdata'
+
+        client.create_bucket(Bucket=bucket_name)
+        presigned_post = client.generate_presigned_post(bucket_name, object_name)
+
+        files = {'file': object_data}
+        r = requests.post(presigned_post['url'], data=presigned_post['fields'], files=files)
+        self.assertEqual(r.status_code, 204)
+
+        result = io.BytesIO()
+        client.download_fileobj(bucket_name, object_name, result)
+        self.assertEqual(to_str(object_data), to_str(result.getvalue()))
+
+        client.delete_object(Bucket=bucket_name, Key=object_name)
+        client.delete_bucket(Bucket=bucket_name)
diff --git a/tests/unit/test_edge.py b/tests/unit/test_edge.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_edge.py
@@ -0,0 +1,59 @@
+import unittest
+from localstack.services.edge import is_s3_form_data
+
+
+class EdgeServiceTest(unittest.TestCase):
+
+    def test_data_contains_key_equal_is_true(self):
+        data_bytes = b'AWSAccessKeyId=someId&policy=somePolicy&key=someKey&signature=someSig'
+        self.assertEqual(is_s3_form_data(data_bytes), True)
+
+    def test_s3_form_data_is_true(self):
+        data_bytes = b"""--28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="Content-Type"
+
+        text/plain
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="key"
+
+        log.txt
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="AWSAccessKeyId"
+
+        AKIAIG7AH67ANH3GAWPQ
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="policy"
+
+        somePolicy
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="signature"
+
+        hTahNRfuCxL5HEKdhXxJPwvC6IQ=
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="file"; filename="file"
+
+        Hello World!
+        --28f72589b2be0c9de84386b52c615990--
+        """
+        self.assertEqual(is_s3_form_data(data_bytes), True)
+
+    def test_other_query_params_is_false(self):
+        data_bytes = b'AWSAccessKeyId=someId&policy=somePolicy&param=value&signature=someSig'
+        self.assertEqual(is_s3_form_data(data_bytes), False)
+
+    def test_other_form_data_is_false(self):
+        data_bytes = b"""--28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="AWSAccessKeyId"
+
+        AKIAIG7AH67ANH3GAWPQ
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="policy"
+
+        somePolicy
+        --28f72589b2be0c9de84386b52c615990
+        Content-Disposition: form-data; name="signature"
+
+        hTahNRfuCxL5HEKdhXxJPwvC6IQ=
+        --28f72589b2be0c9de84386b52c615990--
+        """
+        self.assertEqual(is_s3_form_data(data_bytes), False)