Repository: localstack/localstack

Issue:
Getting "The specified bucket does not exist" when running Localstack inside docker-compose
Hi there,
Although Localstack looks quite promising, since a couple of weeks ago our team is a bit blocked due to an issue we are experiencing with CloudFormation.
Any help would be more than welcome.

Having a docker-compose like this:

```
version: '3'

services:
  gateway:
    image: app
    depends_on:
      - localstack
    environment:
      - AWS_ACCESS_KEY_ID=localkey
      - AWS_SECRET_ACCESS_KEY=localsecret

  localstack:
    container_name: localstack
    image: localstack/localstack
    expose:
      - "4567-4599"
    ports:
      - "4567-4599:4567-4599"
    environment:
      - SERVICES=iam, lambda, s3, cloudformation, sqs, sts
      - LAMBDA_EXECUTOR=docker
      - DOCKER_HOST=unix:///var/run/docker.sock
      - LAMBDA_REMOTE_DOCKER=true
      - HOSTNAME_EXTERNAL=localstack
      - HOSTNAME=localstack
      - DEBUG=1
    volumes:
      - "/tmp/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
```

Gateway container it is just a container with several tools pre-installed like (awscli ... ) to be used as an entry point for the inner network.

From that gateway container we can successfully run :
```
aws --endpoint-url=http://localstack:4572 s3api create-bucket --bucket serverlessdeploy --region us-east-1 --create-bucket-configuration LocationConstraint=us-east-1
```

Even upload object to it (hitting http://localstack:4572 with the following parameters):
```
{
  Body: <Buffer 7b 22 41 57 53 54 65 6d 70 6c 61 74 65 46 6f 72 6d 61 74 56 65 72 73 69 6f 6e 22 3a 22 32 30 31 30 2d 30 39 2d 30 39 22 2c 22 44 65 73 63 72 69 70 74 ... 2768 more bytes>,
  Bucket: 'mybucket',
  Key: 'whatever/local/1584035954811-2020-03-12T17:59:14.811Z/compiled-cloudformation-template.json',
  ContentType: 'application/json',
  Metadata: { filesha256: 'h003I5m1ox/MQ+ia+9zyvGQ38kOrHTI62EVft/J6aIc=' }
}
```

But when we try to create an stack hitting CloudFormation (http://localstack:4581) with the following params:
```
{
  StackName: 'whatever',
  OnFailure: 'DELETE',
  Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM', [length]: 2 ],
  Parameters: [ [length]: 0 ],
  TemplateURL: 'http://localstack:4572/mybucket/whatever/local/1584035954811-2020-03-12T17:59:14.811Z/compiled-cloudformation-template.json',
  Tags: [ { Key: 'STAGE', Value: 'local' }, [length]: 1 ]
}
```

We receive the following error:
```
"CreateStack" (404) POST /: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error>\n    <Code>NoSuchBucket</Code>\n    <Message>The specified bucket does not exist</Message>\n    <BucketName>localstack:4572</BucketName>\n    <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID>\n</Error>'
```


## Additional Info:
 
Docker daemon it is not in the host machine

Operating System: Linux/4.14.146-119.123.amzn2.x86_64 
Localstack Version: latest (0.10.8)
Aws cli Version:  aws-cli/1.18.20 Python/3.5.3 botocore/1.15.20
Docker Version: 19.03.7, build 7141c199a2

## Edit:
If we run same steps locally (having docker daemon in the host machine) we face the following *different* issue:
```
HTTP 502
```
And the stack:
```
2020-03-13T12:45:02:ERROR:localstack.services.generic_proxy: Error forwarding request: Unable to fetch template body (code 403) from URL http://localstack:4572/mybucket/whatever/local/1584103485885-2020-03-13T12:44:45.885Z/compiled-cloudformation-template.json Traceback (most recent call last):
  File "/opt/code/localstack/localstack/services/generic_proxy.py", line 247, in forward
    listener_result = listener.forward_request(method=method,
  File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_listener.py", line 172, in forward_request
    modified_request = transform_template(req_data)
  File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_listener.py", line 70, in transform_template
    template_body = get_template_body(req_data)
  File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_listener.py", line 115, in get_template_body
    raise Exception('Unable to fetch template body (code %s) from URL %s' % (status_code, url))
Exception: Unable to fetch template body (code 403) from URL http://localstack:4572/mybucket/whatever/local/1584103485885-2020-03-13T12:44:45.885Z/compiled-cloudformation-template.json
```

Thanks in advance.



Discussion:
In my opinion there is clearly an internal networking issue inside the docker container Localstack/localstack even more if we take into account that running *localstack start* locally having the daemon also in the same host all goes good.
I also face a similar problem. The communication between cloudformation and S3 seems to be a bit problem. ·No such bucket· often appears. Or where am I doing wrong to cause this error?
Thanks for reporting @miguepintor .

The error you posted indicates a permission denied error (HTTP status code 403), not a networking error per se.
```
Exception: Unable to fetch template body (code 403) from URL http://localstack:4572/mybucket/whatever/local/1584103485885-2020-03-13T12:44:45.885Z/compiled-cloudformation-template.json
```

In fact, we've recently pushed a change in #2148 that could be related to this issue. Can you please do a `docker pull localstack/localstack` and give it another try? Thanks
I also have this exact issue. My configuration is almost identical to @miguepintor.

I'm using serverless framework + localstack, and trying to deploy to localstack, but keep getting hung up on the Cloudformation create/update steps with the same error.

It seems to specifically be an issue with specifying `TemplateUrl` (`TemplateBody` still works for example).

I tried pulling the latest localstack image @whummer, but still getting the same error.
@whummer I tried with the latest:

```
$ docker pull localstack/localstack:latest
latest: Pulling from localstack/localstack
Digest: sha256:f4658065fc07b24a7a8fd42620658bed39a5ebbd92fb039ca4b4dd1dd2ad76ba
```

And same issue:

```
Error response for CloudFormation action "CreateStack" (404) POST /: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error>\n    <Code>NoSuchBucket</Code>\n    <Message>The specified bucket does not exist</Message>\n    <BucketName>localstack:4572</BucketName>\n    <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID>\n</Error>'
```

Also @whummer take into account that is a 404 not a 403. Maybe this could help us to identify the issue.

Many thanks for your support.
Did some work trying to debug this today. The issue seems to be in the backend Cloudformation moto server (maybe that was obvious?).

I pulled down localstack and have been running and modifying a local version to try and debug it. I added a bunch of logging statements in `generic_proxy.py` to see what's going on.

I can see that the requests made to create the bucket and objects all succeed at `http://127.0.0.1:4563`, which is the host and port for the backend moto server. Then the `POST` made to the backend cloudformation server throws the error. 

The TemplateUrl contains the custom hostname (`localstack` in this case), and the frontend s3 port (`4572`). So I'm wondering if the moto server is having trouble resolving that host and port since it's being run inside of the docker container.
Is fix available for this? I am facing the same error. Specified bucket doesnot exist when trying to use localstack's cloudformation with serverless-localstack plugin.
Request to localstack experts to please resolve this issue.
Thanks for the update @miguepintor @david-mk-lawrence @amar02041988 . I was able to reproduce this issue, and believe it should be fixed via #2182 . Can you please do a `docker pull localstack/localstack` and give it another try? Please report here if the problem persists. Thanks!
Doing a pull. Will test and let you know.
@whummer I tried again with:
```
Pulling localstack (localstack/localstack:)...
latest: Pulling from localstack/localstack
b9bba2329739: Pull complete
Digest: sha256:f02c93fd0ea277c87bfb550e10a6602af49bf1b6ac070c90ea8a264682c2ba89
```
The problem is still reproducible :(

```
localstack    | 2020-03-21T17:11:48:DEBUG:localstack.services.cloudformation.cloudformation_listener: Download CloudFormation template content from local S3: mybucket - whatever/local/1584810707991-2020-03-21T17:11:47.991Z/compiled-cloudformation-template.json

localstack    | 2020-03-21T17:11:48:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "CreateStack" (404) POST /: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error>\n    <Code>NoSuchBucket</Code>\n    <Message>The specified bucket does not exist</Message>\n    <BucketName>localstack:4572</BucketName>\n    <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID>\n</Error>'
```

Thanks for your support.
@whummer taking a look into the trace I can see this:
`<BucketName>localstack:4572</BucketName>\n`
Is that correct? shouldn't be "mybucket" or "localstack:4572/mybucket" ?
@whummer Its failing with same error.

Resolving patch:
diff --git a/localstack/services/cloudformation/cloudformation_listener.py b/localstack/services/cloudformation/cloudformation_listener.py
--- a/localstack/services/cloudformation/cloudformation_listener.py
+++ b/localstack/services/cloudformation/cloudformation_listener.py
@@ -89,7 +89,7 @@ def load(self):
             os.environ['AWS_DEFAULT_REGION'] = aws_stack.get_region()
         try:
             transformed = transform_sam(parsed, {}, MockPolicyLoader())
-            return transformed
+            return json.dumps(transformed)
         finally:
             os.environ.pop('AWS_DEFAULT_REGION', None)
             if region_before is not None:
@@ -123,7 +123,7 @@ def get_template_body(req_data):
 
 def is_local_service_url(url):
     candidates = ('localhost', config.LOCALSTACK_HOSTNAME, config.HOSTNAME_EXTERNAL, config.HOSTNAME)
-    return any('://%s:' % host in url for host in candidates)
+    return url and any('://%s:' % host in url for host in candidates)
 
 
 def is_real_s3_url(url):
@@ -178,8 +178,8 @@ def forward_request(self, method, path, data, headers):
                 if stack_name:
                     if stack_name.startswith('arn:aws:cloudformation'):
                         run_fix = True
-                        stack_name = re.sub(r'arn:aws:cloudformation:[^:]+:[^:]+:stack/([^/]+)(/.+)?',
-                                            r'\1', stack_name)
+                        pattern = r'arn:aws:cloudformation:[^:]+:[^:]+:stack/([^/]+)(/.+)?'
+                        stack_name = re.sub(pattern, r'\1', stack_name)
                 if run_fix:
                     stack_names = [stack_name] if stack_name else self._list_stack_names()
                     client = aws_stack.connect_to_service('cloudformation')
@@ -198,11 +198,15 @@ def forward_request(self, method, path, data, headers):
                 do_replace_url = is_real_s3_url(req_data.get('TemplateURL'))
                 if do_replace_url:
                     req_data['TemplateURL'] = convert_s3_to_local_url(req_data['TemplateURL'])
-                modified_request = transform_template(req_data)
-                if modified_request:
+                url = req_data.get('TemplateURL', '')
+                is_custom_local_endpoint = is_local_service_url(url) and '://localhost:' not in url
+                modified_template_body = transform_template(req_data)
+                if not modified_template_body and is_custom_local_endpoint:
+                    modified_template_body = get_template_body(req_data)
+                if modified_template_body:
                     req_data.pop('TemplateURL', None)
-                    req_data['TemplateBody'] = json.dumps(modified_request)
-                if modified_request or do_replace_url:
+                    req_data['TemplateBody'] = modified_template_body
+                if modified_template_body or do_replace_url:
                     data = urlparse.urlencode(req_data, doseq=True)
                     return Request(data=data, headers=headers, method=method)
 
diff --git a/localstack/services/cloudformation/cloudformation_starter.py b/localstack/services/cloudformation/cloudformation_starter.py
--- a/localstack/services/cloudformation/cloudformation_starter.py
+++ b/localstack/services/cloudformation/cloudformation_starter.py
@@ -709,24 +709,22 @@ def cf_describe_stack_events(self):
 
     # patch FakeStack.initialize_resources
 
-    def initialize_resources(self):
+    def run_dependencies_deployment_loop(stack, action):
         def set_status(status):
-            self._add_stack_event(status)
-            self.status = status
-
-        self.resource_map.create()
-        self.output_map.create()
+            stack._add_stack_event(status)
+            stack.status = status
 
         def run_loop(*args):
             # NOTE: We're adding this additional loop, as it seems that in some cases moto
             #   does not consider resource dependencies (e.g., if a "DependsOn" resource property
             #   is defined). This loop allows us to incrementally resolve such dependencies.
-            resource_map = self.resource_map
+            resource_map = stack.resource_map
             unresolved = {}
             for i in range(MAX_DEPENDENCY_DEPTH):
+                LOG.debug('Running CloudFormation stack deployment loop iteration %s' % (i + 1))
                 unresolved = getattr(resource_map, '_unresolved_resources', {})
                 if not unresolved:
-                    set_status('CREATE_COMPLETE')
+                    set_status('%s_COMPLETE' % action)
                     return resource_map
                 resource_map._unresolved_resources = {}
                 for resource_id, resource_details in unresolved.items():
@@ -736,14 +734,25 @@ def run_loop(*args):
                     # looks like no more resources can be resolved -> bail
                     LOG.warning('Unresolvable dependencies, there may be undeployed stack resources: %s' % unresolved)
                     break
-            set_status('CREATE_FAILED')
+            set_status('%s_FAILED' % action)
             raise Exception('Unable to resolve all CloudFormation resources after traversing ' +
                 'dependency tree (maximum depth %s reached): %s' % (MAX_DEPENDENCY_DEPTH, unresolved.keys()))
 
         # NOTE: We're running the loop in the background, as it might take some time to complete
         FuncThread(run_loop).start()
 
+    def initialize_resources(self):
+        self.resource_map.create()
+        self.output_map.create()
+        run_dependencies_deployment_loop(self, 'CREATE')
+
+    def update(self, *args, **kwargs):
+        stack_update_orig(self, *args, **kwargs)
+        run_dependencies_deployment_loop(self, 'UPDATE')
+
     FakeStack.initialize_resources = initialize_resources
+    stack_update_orig = FakeStack.update
+    FakeStack.update = update
 
     # patch Kinesis Stream get_cfn_attribute(..) method in moto
     def Kinesis_Stream_get_cfn_attribute(self, attribute_name):
diff --git a/localstack/utils/aws/aws_models.py b/localstack/utils/aws/aws_models.py
--- a/localstack/utils/aws/aws_models.py
+++ b/localstack/utils/aws/aws_models.py
@@ -74,7 +74,7 @@ def read(self, amount=-1, shard='shardId-000000000001'):
                         print(next_entry['Records'][0]['Data'])
                     record = next_entry
             except Exception as e:
-                print('Error reading from Kinesis stream "%s": %s' (self.stream_name, e))
+                print('Error reading from Kinesis stream "%s": %s' % (self.stream_name, e))
 
     def wait_for(self):
         GET_STATUS_SLEEP_SECS = 5


Test patch: