Repository: localstack/localstack

Issue:
bug: format of message when using SNS Topic as a Lambda DLQ
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

The format of the message received when using an SNS topic as a lambda DLQ diverges from AWS'.
What is missing is the whole event as received by the lambda to be passed as the `Message` field of the SNS message, and not only the `Message` sent to the lambda. 
Current response from Localstack using snapshot mechanism:

```json
{
  "Body": {
    "Message": {
      "raise_error": 1
    },
    "MessageAttributes": {
      "ErrorCode": {
        "Type": "String",
        "Value": "200"
      },
      "ErrorMessage": {
        "Type": "String",
        "Value": "Lambda process returned with error. Result: {\"errorType\":\"Exception\",\"errorMessage\":\"Test exception (this is intentional)\",\"stackTrace\":[\"  File \\\"/var/task/handler.py\\\", line 126, in handler\\n    raise Exception(\\\"Test exception (this is intentional)\\\")\\n\"]}. Output:\n[32mSTART RequestId: 0bf50aba-256b-11b8-8e3a-93690a4d8bcc Version: $LATEST[0m\nLambda log message - print function\n[INFO]\t2022-08-08T13:36:16.615Z\t0bf50aba-256b-11b8-8e3a-93690a4d8bcc\tLambda log message - logging module\n\n[ERROR] Exception: Test exception (this is intentional)\nTraceback (most recent call last):\n\u00a0\u00a0File \"/var/task/handler.py\", line 126, in handler\n\u00a0\u00a0\u00a0\u00a0raise Exception(\"Test exception (this is intentional)\")\n[32mEND RequestId: 0bf50aba-256b-11b8-8e3a-93690a4d8bcc[0m\n[32mREPORT RequestId: 0bf50aba-256b-11b8-8e3a-93690a4d8bcc\tInit Duration: 361.22 ms\tDuration: 6.06 ms\tBilled Duration: 7 ms\tMemory Size: 1536 MB\tMax Memory Used: 35 MB\t[0m"
      },
      "RequestID": {
        "Type": "String",
        "Value": "<request-id:1>"
      }
    },
    "MessageId": "<uuid:1>",
    "Signature": "<signature>",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.<region>.amazonaws.com/SimpleNotificationService-<signing-cert-file:1>",
    "Timestamp": "date",
    "TopicArn": "arn:aws:sns:<region>:111111111111:<resource:1>",
    "Type": "Notification",
    "UnsubscribeURL": "<unsubscribe-domain>/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:<region>:111111111111:<resource:1>:<resource:2>"
  },
  "MD5OfBody": "<md5-hash>",
  "MessageId": "<uuid:2>",
  "ReceiptHandle": "<receipt-handle:1>"
}
 ```

### Expected Behavior

Example message from AWS using the snapshot mechanism:
```json
{
  "Body": {
    "Message": {
      "Records": [
        {
          "EventSource": "aws:sns",
          "EventSubscriptionArn": "arn:aws:sns:<region>:111111111111:<resource:4>:<resource:1>",
          "EventVersion": "1.0",
          "Sns": {
            "Message": {
              "raise_error": 1
            },
            "MessageAttributes": {},
            "MessageId": "<uuid:1>",
            "Signature": "<signature>",
            "SignatureVersion": "1",
            "SigningCertUrl": "https://sns.<region>.amazonaws.com/SimpleNotificationService-<signing-cert-file:1>",
            "Subject": null,
            "Timestamp": "date",
            "TopicArn": "arn:aws:sns:<region>:111111111111:<resource:4>",
            "Type": "Notification",
            "UnsubscribeUrl": "<unsubscribe-domain>/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:<region>:111111111111:<resource:4>:<resource:1>"
          }
        }
      ]
    },
    "MessageAttributes": {
      "ErrorCode": {
        "Type": "Number",
        "Value": "200"
      },
      "ErrorMessage": {
        "Type": "String",
        "Value": "Test exception (this is intentional)"
      },
      "RequestID": {
        "Type": "String",
        "Value": "<request-id:1>"
      }
    },
    "MessageId": "<uuid:2>",
    "Signature": "<signature>",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.<region>.amazonaws.com/SimpleNotificationService-<signing-cert-file:1>",
    "Timestamp": "date",
    "TopicArn": "arn:aws:sns:<region>:111111111111:<resource:3>",
    "Type": "Notification",
    "UnsubscribeURL": "<unsubscribe-domain>/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:<region>:111111111111:<resource:3>:<resource:2>"
  },
  "MD5OfBody": "<md5-hash>",
  "MessageId": "<uuid:3>",
  "ReceiptHandle": "<receipt-handle:1>"
}
```

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

Run the SNS test `tests.integration.test_sns.TestSNSProvider.test_sns_topic_as_lambda_dead_letter_queue`, the scenario is already reproduced. 

### Environment

```markdown
- OS: macOS Monterey 12.5
- LocalStack: latest (1.0.4dev)
```


### Anything else?

_No response_


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).

Resolving patch:
diff --git a/localstack/services/awslambda/lambda_api.py b/localstack/services/awslambda/lambda_api.py
--- a/localstack/services/awslambda/lambda_api.py
+++ b/localstack/services/awslambda/lambda_api.py
@@ -431,7 +431,7 @@ def process_sns_notification(
                     # TODO Add a more sophisticated solution with an actual signature
                     # Hardcoded
                     "Signature": "EXAMPLEpH+..",
-                    "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-000000000.pem",
+                    "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem",
                     "UnsubscribeUrl": unsubscribe_url,
                     "MessageAttributes": message_attributes,
                 },
diff --git a/localstack/services/sns/provider.py b/localstack/services/sns/provider.py
--- a/localstack/services/sns/provider.py
+++ b/localstack/services/sns/provider.py
@@ -439,6 +439,7 @@ def get_topic_attributes(
         self, context: RequestContext, topic_arn: topicARN
     ) -> GetTopicAttributesResponse:
         moto_response = call_moto(context)
+        # todo fix some attributes by moto, see snapshot
         return GetTopicAttributesResponse(**moto_response)
 
     def publish_batch(
@@ -637,6 +638,7 @@ def get_subscription_attributes(
         sub = get_subscription_by_arn(subscription_arn)
         if not sub:
             raise NotFoundException(f"Subscription with arn {subscription_arn} not found")
+        # todo fix some attributes by moto see snapshot
         return GetSubscriptionAttributesResponse(Attributes=sub)
 
     def list_subscriptions(
diff --git a/localstack/utils/aws/dead_letter_queue.py b/localstack/utils/aws/dead_letter_queue.py
--- a/localstack/utils/aws/dead_letter_queue.py
+++ b/localstack/utils/aws/dead_letter_queue.py
@@ -92,25 +92,22 @@ def _prepare_messages_to_dlq(source_arn: str, event: Dict, error) -> List[Dict]:
             }
         )
     elif ":lambda:" in source_arn:
-        if event.get("Records") and "sns" in event["Records"][0]["EventSource"]:
-            for record in event["Records"]:
-                sns_rec = record.get("Sns", {})
-                message_attrs = {**sns_rec.get("MessageAttributes"), **custom_attrs}
-                messages.append(
-                    {
-                        "Id": sns_rec.get("MessageId"),
-                        "MessageBody": sns_rec.get("Message"),
-                        "MessageAttributes": message_attrs,
-                    }
-                )
-        else:
-            messages.append(
-                {
-                    "Id": str(uuid.uuid4()),
-                    "MessageBody": json.dumps(event),
-                    "MessageAttributes": custom_attrs,
-                }
-            )
+        custom_attrs["ErrorCode"]["DataType"] = "Number"
+        # not sure about what type of error can come here
+        try:
+            error_message = json.loads(error.result)["errorMessage"]
+            custom_attrs["ErrorMessage"]["StringValue"] = error_message
+        except (ValueError, KeyError):
+            # using old behaviour
+            custom_attrs["ErrorMessage"]["StringValue"] = str(error)
+
+        messages.append(
+            {
+                "Id": str(uuid.uuid4()),
+                "MessageBody": json.dumps(event),
+                "MessageAttributes": custom_attrs,
+            }
+        )
     # make sure we only have printable strings in the message attributes
     for message in messages:
         if message.get("MessageAttributes"):


Test patch:
diff --git a/tests/integration/awslambda/test_lambda_sqs_integration.py b/tests/integration/awslambda/test_lambda_sqs_integration.py
--- a/tests/integration/awslambda/test_lambda_sqs_integration.py
+++ b/tests/integration/awslambda/test_lambda_sqs_integration.py
@@ -4,9 +4,15 @@
 
 import pytest
 
-from localstack.services.awslambda.lambda_utils import LAMBDA_RUNTIME_PYTHON38
+from localstack.services.awslambda.lambda_utils import (
+    LAMBDA_RUNTIME_PYTHON37,
+    LAMBDA_RUNTIME_PYTHON38,
+)
+from localstack.testing.aws.util import is_aws_cloud
 from localstack.utils.strings import short_uid
 from localstack.utils.sync import retry
+from tests.integration.awslambda.functions import lambda_integration
+from tests.integration.awslambda.test_lambda import TEST_LAMBDA_FUNCTION_PREFIX, TEST_LAMBDA_PYTHON
 
 THIS_FOLDER = os.path.dirname(os.path.realpath(__file__))
 LAMBDA_SQS_INTEGRATION_FILE = os.path.join(THIS_FOLDER, "functions", "lambda_sqs_integration.py")
@@ -239,3 +245,67 @@ def test_redrive_policy_with_failing_lambda(
     dlq_response = sqs_client.receive_message(QueueUrl=event_dlq_url, WaitTimeSeconds=15)
     assert "Messages" in dlq_response
     snapshot.match("dlq_response", dlq_response)
+
+
+@pytest.mark.aws_validated
+def test_sqs_queue_as_lambda_dead_letter_queue(
+    sqs_client,
+    lambda_client,
+    lambda_su_role,
+    create_lambda_function,
+    sqs_create_queue,
+    sqs_queue_arn,
+    snapshot,
+):
+    snapshot.add_transformer(
+        [
+            # MessageAttributes contain the request id, messes the hash
+            snapshot.transform.key_value(
+                "MD5OfMessageAttributes",
+                value_replacement="<md5-hash>",
+                reference_replacement=False,
+            ),
+            snapshot.transform.jsonpath(
+                "$..Messages..MessageAttributes.RequestID.StringValue", "request-id"
+            ),
+        ]
+    )
+
+    dlq_queue_url = sqs_create_queue()
+    dlq_queue_arn = sqs_queue_arn(dlq_queue_url)
+
+    function_name = f"{TEST_LAMBDA_FUNCTION_PREFIX}-{short_uid()}"
+    lambda_creation_response = create_lambda_function(
+        func_name=function_name,
+        handler_file=TEST_LAMBDA_PYTHON,
+        runtime=LAMBDA_RUNTIME_PYTHON37,
+        role=lambda_su_role,
+        DeadLetterConfig={"TargetArn": dlq_queue_arn},
+    )
+    snapshot.match(
+        "lambda-response-dlq-config",
+        lambda_creation_response["CreateFunctionResponse"]["DeadLetterConfig"],
+    )
+
+    # invoke Lambda, triggering an error
+    payload = {lambda_integration.MSG_BODY_RAISE_ERROR_FLAG: 1}
+    lambda_client.invoke(
+        FunctionName=function_name,
+        Payload=json.dumps(payload),
+        InvocationType="Event",
+    )
+
+    def receive_dlq():
+        result = sqs_client.receive_message(
+            QueueUrl=dlq_queue_url, MessageAttributeNames=["All"], VisibilityTimeout=0
+        )
+        assert len(result["Messages"]) > 0
+        return result
+
+    # check that the SQS queue used as DLQ received the error from the lambda
+    # on AWS, event retries can be quite delayed, so we have to wait up to 6 minutes here
+    # reduced retries when using localstack to avoid tests flaking
+    retries = 120 if is_aws_cloud() else 3
+    messages = retry(receive_dlq, retries=retries, sleep=3)
+
+    snapshot.match("messages", messages)
diff --git a/tests/integration/awslambda/test_lambda_sqs_integration.snapshot.json b/tests/integration/awslambda/test_lambda_sqs_integration.snapshot.json
--- a/tests/integration/awslambda/test_lambda_sqs_integration.snapshot.json
+++ b/tests/integration/awslambda/test_lambda_sqs_integration.snapshot.json
@@ -198,5 +198,44 @@
         }
       }
     }
+  },
+  "tests/integration/awslambda/test_lambda_sqs_integration.py::test_sqs_queue_as_lambda_dead_letter_queue": {
+    "recorded-date": "10-08-2022, 11:10:19",
+    "recorded-content": {
+      "lambda-response-dlq-config": {
+        "TargetArn": "arn:aws:sqs:<region>:111111111111:<resource:1>"
+      },
+      "messages": {
+        "Messages": [
+          {
+            "Body": {
+              "raise_error": 1
+            },
+            "MD5OfBody": "<m-d5-of-body:1>",
+            "MD5OfMessageAttributes": "<md5-hash>",
+            "MessageAttributes": {
+              "ErrorCode": {
+                "DataType": "Number",
+                "StringValue": "200"
+              },
+              "ErrorMessage": {
+                "DataType": "String",
+                "StringValue": "Test exception (this is intentional)"
+              },
+              "RequestID": {
+                "DataType": "String",
+                "StringValue": "<request-id:1>"
+              }
+            },
+            "MessageId": "<uuid:1>",
+            "ReceiptHandle": "<receipt-handle:1>"
+          }
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      }
+    }
   }
 }
diff --git a/tests/integration/test_sns.py b/tests/integration/test_sns.py
--- a/tests/integration/test_sns.py
+++ b/tests/integration/test_sns.py
@@ -623,16 +623,6 @@ def check_subscription():
         assert poll_condition(check_subscription, timeout=5)
 
     @pytest.mark.aws_validated
-    @pytest.mark.skip_snapshot_verify(
-        paths=[
-            "$..Messages..Body.Message.raise_error",  # check lambda delivery format into DLQ
-            "$..Messages..MessageAttributes.ErrorCode.Type",
-            "$..Messages..MessageAttributes.ErrorMessage.Value",
-        ]
-    )
-    @pytest.mark.xfail(
-        reason="Behaviour and format diverges from AWS, see https://github.com/localstack/localstack/issues/6613"
-    )
     def test_sns_topic_as_lambda_dead_letter_queue(
         self,
         sns_client,
@@ -646,8 +636,6 @@ def test_sns_topic_as_lambda_dead_letter_queue(
         sns_create_sqs_subscription,
         snapshot,
     ):
-        # todo check the skipped paths, as the format is very wrong
-        snapshot.add_transformer(snapshot.transform.lambda_api())
         snapshot.add_transformer(
             snapshot.transform.jsonpath(
                 "$..Messages..MessageAttributes.RequestID.Value", "request-id"
@@ -720,13 +708,6 @@ def receive_dlq():
 
         snapshot.match("messages", messages)
 
-        # as we ignore some fields, still assert manually the presence of certain fields
-        msg_body = messages["Messages"][0]["Body"]
-        msg_attrs = msg_body["MessageAttributes"]
-        assert "RequestID" in msg_attrs
-        assert "ErrorCode" in msg_attrs
-        assert "ErrorMessage" in msg_attrs
-
     @pytest.mark.only_localstack
     def test_redrive_policy_http_subscription(
         self,