Repository: localstack/localstack

Issue:
Unable to create SQS redrive policy using CloudFormation
Hi,

I'm using the latest localstack docker image which is currently `0.9.5`.

The following error occurs when trying to use CloudFormation to create Dead Letter Queue and Queue in same template.
`An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the CreateStack operation: Could not find DLQ for arn:aws:sqs:elasticmq:000000000000:test-dlq`

Example template
```yaml
AWSTemplateFormatVersion: 2010-09-09
Resources:
  DeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: 'test-dlq'
      DelaySeconds: 0
      MaximumMessageSize: 262144 # 256KB
      MessageRetentionPeriod: 1209600 # 14 Days
      VisibilityTimeout: 120 # 2 minutes
  Queue:
    Type: AWS::SQS::Queue
    DependsOn: DeadLetterQueue
    Properties:
      QueueName: 'test-queue'
      DelaySeconds: 0
      MaximumMessageSize: 262144 # 256KB
      MessageRetentionPeriod: 345600 # 4 Days
      RedrivePolicy:
        deadLetterTargetArn:
          Fn::GetAtt:
            - DeadLetterQueue
            - Arn
        maxReceiveCount: 1
      VisibilityTimeout: 120 # 2 minutes
```

Note, I even tried to use the `DependsOn` property to force it to create the DLQ first but to no avail.

Command used with `awslocal`...
```shell
awslocal cloudformation create-stack --stack-name test1 --template-body file://./queues.template.dlq.yml
```
Using the same template in AWS CloudFormation works as expected.

*Debug log output*
```
localstack_1  | 2019-07-10T10:35:16:DEBUG:__main__: Currently updating stack resource test1/DeadLetterQueue: None 
localstack_1  | 2019-07-10T10:35:16:DEBUG:__main__: Deploying CloudFormation resource: {'Type': 'AWS::SQS::Queue', 'Properties': {'QueueName': 'test-dlq', 'DelaySeconds': 0, 'MaximumMessageSize': 262144, 'MessageRetentionPeriod': 1209600, 'VisibilityTimeout': 120}}
localstack_1  | 2019-07-10T10:35:16:DEBUG:localstack.utils.cloudformation.template_deployer: Deploying resource type "SQS::Queue" id "DeadLetterQueue"
localstack_1  | 2019-07-10T10:35:16:DEBUG:localstack.utils.cloudformation.template_deployer: Request for creating resource type "SQS::Queue": {'QueueName': 'test-dlq'}
localstack_1  | 10:35:16.440 [elasticmq-akka.actor.default-dispatcher-3] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue QueueData(test-dlq,MillisVisibilityTimeout(30000),PT0S,PT0S,2019-07-10T10:35:16.408Z,2019-07-10T10:35:16.408Z,None,false,false,None,None,Map()) 
localstack_1  | 2019-07-10T10:35:16:DEBUG:__main__: Currently updating stack resource test1/Queue: None
localstack_1  | 2019-07-10T10:35:16:ERROR:__main__: Unable to parse and create resource "Queue": 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?> 
localstack_1  | <Response>
localstack_1  | <Errors>
localstack_1  | <Error> 
localstack_1  | <Code>AWS.SimpleQueueService.NonExistentQueue</Code>
localstack_1  | <Message>Could not find DLQ for arn:aws:sqs:elasticmq:000000000000:test-dlq</Message>
localstack_1  |
localstack_1  | </Error>
localstack_1  | </Errors>
localstack_1  | <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID> 
localstack_1  | </Response>
localstack_1  | 2019-07-10T10:35:16:INFO:werkzeug: 127.0.0.1 - - [10/Jul/2019 10:35:16] "POST / HTTP/1.1" 400 -
localstack_1  | 2019-07-10T10:35:16:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response from CloudFormation (400) POST /: b'<?xml version="1.0" encoding="UTF-8"?>\n  <Response>\n    <Errors>\n      <Error>\n        <Code>AWS.SimpleQueueService.NonExistentQueue</Code>\n        <Message>Could not find DLQ for arn:aws:sqs:elasticmq:000000000000:test-dlq</Message>\n        \n      </Error>\n    </Errors>\n  <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID>\n</Response>'
```

Can you help? Thanks.


Discussion:
I am facing exactly the same issue, running on MacOS with fish and virtualenv. Tried to force elasticMQ downgrade but with no sucess.
I did some digging into this, and added some debugging statements into `moto`:

In `moto/sqs/models.py#266`:
```
        f.write("looking for a queue matching %s\n" % (self.redrive_policy['deadLetterTargetArn']))
        for queue in sqs_backends[self.region].queues.values():
            f.write("found a queue: %s\n" % (queue.queue_arn))
            if queue.queue_arn == self.redrive_policy['deadLetterTargetArn']:
```

And the result:
```
$ cat /tmp/moto.log
looking for a queue matching arn:aws:sqs:us-east-1:000000000000:dlq
found a queue: arn:aws:sqs:us-east-1:123456789012:dlq
```

Notice the mismatched account IDs. 
Did some more digging. 

My specific use case is that I am using the Serverless Framework to define a dead letter queue (DLQ) and another queue that uses that DLQ in its redrive policy. I believe what is happening is that the Serverless Framework creates and uploads a CloudFormation template that contains a redrive policy that looks like this:

```
RedrivePolicy:
  deadLetterTargetArn:
    Fn::GetAtt:
      - dlq
      - Arn
```

Localstack forwards a CFN deployment request to Moto, which then downloads the Serverless-generated CFN template from S3. As this template is processed, and resources are created, `Fn::GetAtt[dlq, Arn]` is eventually evaluated by Localstack, and Localstack returns the ARN, which contains the account ID `000000000000` baked into it. This Arn is eventually passed to Moto in an `sqs:CreateQueue` call. Moto then compares this ARN against queues that it is aware of internally, and fails to find any match.

Since Localstack and Moto seem to call back and forth to each other, I might not have some of these interactions right.


Resolving patch:
diff --git a/localstack/services/cloudformation/cloudformation_starter.py b/localstack/services/cloudformation/cloudformation_starter.py
--- a/localstack/services/cloudformation/cloudformation_starter.py
+++ b/localstack/services/cloudformation/cloudformation_starter.py
@@ -1,6 +1,7 @@
 import sys
 import logging
 import traceback
+import six
 from moto.s3 import models as s3_models
 from moto.iam import models as iam_models
 from moto.sqs import models as sqs_models
@@ -14,9 +15,9 @@
 from boto.cloudformation.stack import Output
 from moto.cloudformation.exceptions import ValidationError, UnformattedGetAttTemplateException
 from localstack import config
-from localstack.constants import DEFAULT_PORT_CLOUDFORMATION_BACKEND
+from localstack.constants import DEFAULT_PORT_CLOUDFORMATION_BACKEND, TEST_AWS_ACCOUNT_ID, MOTO_ACCOUNT_ID
 from localstack.utils.aws import aws_stack
-from localstack.utils.common import short_uid, FuncThread
+from localstack.utils.common import FuncThread, short_uid, recurse_object, clone
 from localstack.stepfunctions import models as sfn_models
 from localstack.services.infra import (
     get_service_protocol, start_proxy_for_service, do_run, canonicalize_api_names)
@@ -55,6 +56,17 @@ def start_cloudformation(port=None, asynchronous=False, update_listener=None):
         return thread
 
 
+def set_moto_account_ids(resource_json):
+    def fix_ids(obj):
+        if isinstance(obj, dict):
+            for key, value in obj.items():
+                if 'arn' in key.lower() and isinstance(value, six.string_types):
+                    obj[key] = value.replace(TEST_AWS_ACCOUNT_ID, MOTO_ACCOUNT_ID)
+        return obj
+
+    return recurse_object(resource_json, fix_ids)
+
+
 def apply_patches():
     """ Apply patches to make LocalStack seamlessly interact with the moto backend.
         TODO: Eventually, these patches should be contributed to the upstream repo! """
@@ -140,8 +152,12 @@ def _parse_and_create_resource(logical_id, resource_json, resources_map, region_
                     return resource
 
         if not resource:
+            # fix resource ARNs, make sure to convert account IDs 000000000000 to 123456789012
+            resource_json_arns_fixed = clone(resource_json)
+            set_moto_account_ids(resource_json_arns_fixed)
             # create resource definition and store CloudFormation metadata in moto
-            resource = parse_and_create_resource_orig(logical_id, resource_json, resources_map, region_name)
+            resource = parse_and_create_resource_orig(logical_id,
+                resource_json_arns_fixed, resources_map, region_name)
             # Fix for moto which sometimes hard-codes region name as 'us-east-1'
             if hasattr(resource, 'region_name') and resource.region_name != region_name:
                 LOG.debug('Updating incorrect region from %s to %s' % (resource.region_name, region_name))
@@ -238,7 +254,7 @@ def update_resource_id(resource, new_id, props, region_name):
             LOG.warning('Unexpected resource type when updating ID: %s' % type(resource))
 
     def update_physical_resource_id(resource):
-        phys_res_id = getattr(resource, 'physical_resource_id') if hasattr(resource, 'physical_resource_id') else None
+        phys_res_id = getattr(resource, 'physical_resource_id', None)
         if not phys_res_id:
             if isinstance(resource, lambda_models.LambdaFunction):
                 func_arn = aws_stack.lambda_function_arn(resource.function_name)
diff --git a/localstack/utils/cloudformation/template_deployer.py b/localstack/utils/cloudformation/template_deployer.py
--- a/localstack/utils/cloudformation/template_deployer.py
+++ b/localstack/utils/cloudformation/template_deployer.py
@@ -17,6 +17,19 @@
 # list of resource types that can be updated
 UPDATEABLE_RESOURCES = ['Lambda::Function', 'ApiGateway::Method']
 
+
+def str_or_none(o):
+    return o if o is None else json.dumps(o) if isinstance(o, (dict, list)) else str(o)
+
+
+def select_attributes(obj, attrs):
+    result = {}
+    for attr in attrs:
+        if obj.get(attr) is not None:
+            result[attr] = str_or_none(obj.get(attr))
+    return result
+
+
 # maps resource types to functions and parameters for creation
 RESOURCE_TO_FUNCTION = {
     'S3::Bucket': {
@@ -34,7 +47,12 @@
             'boto_client': 'client',
             'function': 'create_queue',
             'parameters': {
-                'QueueName': ['QueueName', PLACEHOLDER_RESOURCE_NAME]
+                'QueueName': ['QueueName', PLACEHOLDER_RESOURCE_NAME],
+                'Attributes': lambda params, **kwargs: select_attributes(params,
+                    ['DelaySeconds', 'MaximumMessageSize', 'MessageRetentionPeriod',
+                     'VisibilityTimeout', 'RedrivePolicy']
+                ),
+                'tags': 'Tags'
             }
         }
     },
@@ -614,6 +632,7 @@ def deploy_resource(resource_id, resources, stack_name):
                     prop_value = resource_props.get(prop_key)
                 if prop_value is not None:
                     params[param_key] = prop_value
+
             tmp_value = params.get(param_key)
             if tmp_value is not None:
                 params[param_key] = resolve_refs_recursively(stack_name, tmp_value, resources)
@@ -621,6 +640,7 @@ def deploy_resource(resource_id, resources, stack_name):
         # hack: convert to boolean
         if params.get(param_key) in ['True', 'False']:
             params[param_key] = params.get(param_key) == 'True'
+
     # assign default value if empty
     params = common.merge_recursive(defaults, params)
     # convert data types (e.g., boolean strings to bool)


Test patch: