Repository: localstack/localstack

Issue:
bug: SNS batch publish with SQS subscription fails when MessageStructure json is used
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Messages containing a MessageStructure set to "json" are not delivered to SQS when these are published on SNS using 'publish-batch'

### Expected Behavior

Messages do make it to SQS.


### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    docker run localstack/localstack

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

##### Creating SNS toppic/SQS queue/subscription

```
$ ./awslocal sns create-topic --name 'my-sns-topic'
{
    "TopicArn": "arn:aws:sns:us-east-1:000000000000:my-sns-topic"
}
```

```
$ ./awslocal sqs create-queue --queue-name 'my-sqs-queue'
{
    "QueueUrl": "http://localhost:4566/000000000000/my-sqs-queue"
}
```

```
$ ./awslocal sqs get-queue-attributes --queue-url "http://localhost:4566/000000000000/my-sqs-queue" --attribute-names QueueArn
{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:us-east-1:000000000000:my-sqs-queue"
    }
}
```

```
$ ./awslocal sns subscribe --topic-arn "arn:aws:sns:us-east-1:000000000000:my-sns-topic" --protocol sqs --notification-endpoint "arn:aws:sqs:us-east-1:000000000000:my-sqs-queue"
{
    "SubscriptionArn": "arn:aws:sns:us-east-1:000000000000:my-sns-topic:2e8368a4-b3b3-4106-abc7-84a22eeb69af"
}
```

##### Verify using 'publish'

```
$ ./awslocal sns publish --topic-arn "arn:aws:sns:us-east-1:000000000000:my-sns-topic" --message-structure 'JSON' --message '{"default": "default message", "sqs": "sqs message"}'
{
    "MessageId": "e5bc1165-51bb-4710-9e65-eed1516ce229"
}
```

```
$ ./awslocal sqs receive-message --queue-url "http://localhost:4566/000000000000/my-sqs-queue"
{
    "Messages": [
        {
            "MessageId": "b997e242-687b-4a2e-befb-d2dd8169f80c",
            "ReceiptHandle": "ZGM5YTAyY2UtYWI5ZC00ZDg5LWFjOTMtYzk2NmFkYmNiY2NlIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktc3FzLXF1ZXVlIGI5OTdlMjQyLTY4N2ItNGEyZS1iZWZiLWQyZGQ4MTY5ZjgwYyAxNjg2NTc5MDAxLjk0MDQ2MTQ=",
            "MD5OfBody": "32fc7c66865504d74e50c8c646ad1156",
            "Body": "{\"Type\": \"Notification\", \"MessageId\": \"2057f9f0-b8b9-4fb2-80c4-0dd6cf3e2dd2\", \"TopicArn\": \"arn:aws:sns:us-east-1:000000000000:my-sns-topic\", \"Message\": \"sqs message\", \"Timestamp\": \"2023-06-12T14:09:57.879Z\", \"SignatureVersion\": \"1\", \"Signature\": \"EXAMPLEpH+..\", \"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem\", \"UnsubscribeURL\": \"http://localhost:4566/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:000000000000:my-sns-topic:2e8368a4-b3b3-4106-abc7-84a22eeb69af\"}"
        }
    ]
}
```

The SQS message contains "sqs message" as message. This is the expected result so that's good.

##### Verify 'publish-batch' (without MessageStructure)

```
$ ./awslocal sns publish-batch --topic-arn "arn:aws:sns:us-east-1:000000000000:my-sns-topic" --publish-batch-request-entries '[{"Id": "1", "Message": "sqs batch 1"}]'
{
    "Successful": [
        {
            "Id": "1",
            "MessageId": "bf04ebfa-193d-42f1-afe6-aadf3fb9d314"
        }
    ],
    "Failed": []
}
```

```
./awslocal sqs receive-message --queue-url "http://localhost:4566/000000000000/my-sqs-queue"
{
    "Messages": [
        {
            "MessageId": "54036176-ee15-4426-a571-295a5d5f6657",
            "ReceiptHandle": "ZGJkMjBmOWMtZmQ2Mi00YTMwLTk5YTktNzNkY2M4YjFlNmU1IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktc3FzLXF1ZXVlIDU0MDM2MTc2LWVlMTUtNDQyNi1hNTcxLTI5NWE1ZDVmNjY1NyAxNjg2NTc5Mjc1LjQ1MzExNw==",
            "MD5OfBody": "1bc38fc8c7d7a7a480784f4ea1bb37f3",
            "Body": "{\"Type\": \"Notification\", \"MessageId\": \"bf04ebfa-193d-42f1-afe6-aadf3fb9d314\", \"TopicArn\": \"arn:aws:sns:us-east-1:000000000000:my-sns-topic\", \"Message\": \"sqs batch 1\", \"Timestamp\": \"2023-06-12T14:14:30.161Z\", \"SignatureVersion\": \"1\", \"Signature\": \"EXAMPLEpH+..\", \"SigningCertURL\": \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem\", \"UnsubscribeURL\": \"http://localhost:4566/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:000000000000:my-sns-topic:2e8368a4-b3b3-4106-abc7-84a22eeb69af\"}"
        }
    ]
}
```

The SQS message contains "sqs  batch 1" as message. This is the expected result so that's good.

##### Verify 'publish-batch' (with MessageStructure)

```
$ ./awslocal sns publish-batch --topic-arn "arn:aws:sns:us-east-1:000000000000:my-sns-topic" --publish-batch-request-entries '[{"Id": "1", "MessageStructure": "json", "Message": "{\"default\": \"default batch 2\", \"sqs\": \"sqs batch 2\"}"}]'
{
    "Successful": [
        {
            "Id": "1",
            "MessageId": "29b26566-79a8-40eb-8abc-37fd42232dcc"
        }
    ],
    "Failed": []
}
```

```
$ ./awslocal sqs receive-message --queue-url "http://localhost:4566/000000000000/my-sqs-queue"
```

There are no messages on the queue.
This is **unexpected**.
The **expected** result is a message with content "sqs batch 2".



### Environment

```markdown
- OS: Debian
- LocalStack:
  - LocalStack version: 2.1.1.dev
  - LocalStack build date: 2023-06-10
  - LocalStack build git hash: 710f950c
```


### Anything else?

### Relevant logging

```
DEBUG --- [   asgi_gw_2] l.services.sns.publisher   : Topic 'arn:aws:sns:us-east-1:000000000000:my-sns-topic' batch publishing 1 messages to subscribed 'arn:aws:sqs:us-east-1:000000000000:my-sqs-queue' with protocol 'sqs' (subscription 'arn:aws:sns:us-east-1:000000000000:my-sns-topic:2e8368a4-b3b3-4106-abc7-84a22eeb69af')
INFO --- [   asgi_gw_2] localstack.request.aws     : AWS sns.PublishBatch => 200
```

It shows that the SNS publish-batch succeeded and then it attempted to deliver to SQS.
There are **no messages** about the actual delivery to SQS and/or no failure logged.
I.e. it doesn't log the result of the `AWS sqs.SendMessageBatch` call (which it does do in the other cases)


### Debugging

In SqsBatchTopicPublisher the "self.prepare_message(message_ctx, subscriber)" call fails.
It throws an exception but it appears this isn't logged, adding a try except around the for loop in SqsBatchTopicPublisher::_publish and resending the message:

```
Traceback (most recent call last):
  File "/opt/code/localstack/localstack/services/sns/publisher.py", line 318, in _publish
    message_body = self.prepare_message(message_ctx, subscriber)
  File "/opt/code/localstack/localstack/services/sns/publisher.py", line 108, in prepare_message
    return create_sns_message_body(message_context, subscriber)
  File "/opt/code/localstack/localstack/services/sns/publisher.py", line 777, in create_sns_message_body
    message_content = message_context.message_content(protocol)
  File "/opt/code/localstack/localstack/services/sns/models.py", line 71, in message_content
    return self.message.get(protocol, self.message.get("default"))
AttributeError: 'str' object has no attribute 'get'
```

Adding some logging in the 'create_sns_message_body' function:
```
message_context=SnsMessage(type='Notification', message='{"default": "default batch 2", "sqs": "sqs batch 2"}', message_attributes={}, message_structure='json', subject=None, message_deduplication_id=None, message_group_id=None, token=None, message_id='29b26566-79a8-40eb-8abc-37fd42232dcc', is_fifo=False, sequencer_number=None)
```
The `message_context.message` is a string (JSON) but the code appears to expect a dictionary?





Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).

Resolving patch:
diff --git a/localstack/services/sns/provider.py b/localstack/services/sns/provider.py
--- a/localstack/services/sns/provider.py
+++ b/localstack/services/sns/provider.py
@@ -159,22 +159,13 @@ def publish_batch(
                 "Two or more batch entries in the request have the same Id."
             )
 
-        if is_fifo := (".fifo" in topic_arn):
-            if not all(["MessageGroupId" in entry for entry in publish_batch_request_entries]):
-                raise InvalidParameterException(
-                    "Invalid parameter: The MessageGroupId parameter is required for FIFO topics"
-                )
-            topic = self._get_topic(topic_arn, context)
-            if topic.content_based_deduplication == "false":
-                if not all(
-                    ["MessageDeduplicationId" in entry for entry in publish_batch_request_entries]
-                ):
-                    raise InvalidParameterException(
-                        "Invalid parameter: The topic should either have ContentBasedDeduplication enabled or MessageDeduplicationId provided explicitly",
-                    )
-
-        # TODO: implement SNS MessageDeduplicationId and ContentDeduplication checks
         response: PublishBatchResponse = {"Successful": [], "Failed": []}
+
+        # TODO: write AWS validated tests with FilterPolicy and batching
+        # TODO: find a scenario where we can fail to send a message synchronously to be able to report it
+        # right now, it seems that AWS fails the whole publish if something is wrong in the format of 1 message
+
+        message_contexts = []
         for entry in publish_batch_request_entries:
             message_attributes = entry.get("MessageAttributes", {})
             if message_attributes:
@@ -186,21 +177,39 @@ def publish_batch(
             if entry.get("MessageStructure") == "json":
                 try:
                     message = json.loads(entry.get("Message"))
+                    # Keys in the JSON object that correspond to supported transport protocols must have
+                    # simple JSON string values.
+                    # Non-string values will cause the key to be ignored.
+                    message = {
+                        key: field for key, field in message.items() if isinstance(field, str)
+                    }
                     if "default" not in message:
                         raise InvalidParameterException(
                             "Invalid parameter: Message Structure - No default entry in JSON message body"
                         )
+                    entry["Message"] = message  # noqa
                 except json.JSONDecodeError:
                     raise InvalidParameterException(
                         "Invalid parameter: Message Structure - JSON message body failed to parse"
                     )
 
-        # TODO: write AWS validated tests with FilterPolicy and batching
-        # TODO: find a scenario where we can fail to send a message synchronously to be able to report it
-        # right now, it seems that AWS fails the whole publish if something is wrong in the format of 1 message
+            if is_fifo := (".fifo" in topic_arn):
+                if not all(["MessageGroupId" in entry for entry in publish_batch_request_entries]):
+                    raise InvalidParameterException(
+                        "Invalid parameter: The MessageGroupId parameter is required for FIFO topics"
+                    )
+                topic = self._get_topic(topic_arn, context)
+                if topic.content_based_deduplication == "false":
+                    if not all(
+                        [
+                            "MessageDeduplicationId" in entry
+                            for entry in publish_batch_request_entries
+                        ]
+                    ):
+                        raise InvalidParameterException(
+                            "Invalid parameter: The topic should either have ContentBasedDeduplication enabled or MessageDeduplicationId provided explicitly",
+                        )
 
-        message_contexts = []
-        for entry in publish_batch_request_entries:
             msg_ctx = SnsMessage.from_batch_entry(entry, is_fifo=is_fifo)
             message_contexts.append(msg_ctx)
             success = PublishBatchResultEntry(
@@ -210,6 +219,7 @@ def publish_batch(
             if is_fifo:
                 success["SequenceNumber"] = msg_ctx.sequencer_number
             response["Successful"].append(success)
+
         publish_ctx = SnsBatchPublishContext(
             messages=message_contexts,
             store=store,


Test patch:
diff --git a/tests/integration/test_sns.py b/tests/integration/test_sns.py
--- a/tests/integration/test_sns.py
+++ b/tests/integration/test_sns.py
@@ -1325,6 +1325,7 @@ def test_publish_sqs_from_sns(
         }
 
     @pytest.mark.aws_validated
+    @pytest.mark.skip_snapshot_verify(paths=["$..Attributes.SubscriptionPrincipal"])
     def test_publish_batch_messages_from_sns_to_sqs(
         self, sns_create_topic, sqs_create_queue, sns_create_sqs_subscription, snapshot, aws_client
     ):
@@ -1368,11 +1369,15 @@ def test_publish_batch_messages_from_sns_to_sqs(
                     "Id": "4",
                     "Message": "Test Message without subject",
                 },
+                {
+                    "Id": "5",
+                    "Message": json.dumps({"default": "test default", "sqs": "test sqs"}),
+                    "MessageStructure": "json",
+                },
             ],
         )
         snapshot.match("publish-batch", publish_batch_response)
 
-        message_ids_received = set()
         messages = []
 
         def get_messages():
@@ -1380,23 +1385,19 @@ def get_messages():
             sqs_response = aws_client.sqs.receive_message(
                 QueueUrl=queue_url,
                 WaitTimeSeconds=1,
-                VisibilityTimeout=10,
+                VisibilityTimeout=0,
                 MessageAttributeNames=["All"],
                 AttributeNames=["All"],
             )
             for message in sqs_response["Messages"]:
-                if message["MessageId"] in message_ids_received:
-                    aws_client.sqs.delete_message(
-                        QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]
-                    )
-                    continue
-
-                message_ids_received.add(message["MessageId"])
                 messages.append(message)
+                aws_client.sqs.delete_message(
+                    QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]
+                )
 
-            assert len(messages) == 4
+            assert len(messages) == 5
 
-        retry(get_messages, retries=3, sleep=1)
+        retry(get_messages, retries=10, sleep=0.1)
         # we need to sort the list (the order does not matter as we're not using FIFO)
         messages.sort(key=itemgetter("Body"))
         snapshot.match("messages", {"Messages": messages})
@@ -1780,6 +1781,19 @@ def test_publish_batch_exceptions(
             )
         snapshot.match("no-dedup-id", e.value.response)
 
+        with pytest.raises(ClientError) as e:
+            aws_client.sns.publish_batch(
+                TopicArn=topic_arn,
+                PublishBatchRequestEntries=[
+                    {
+                        "Id": "1",
+                        "Message": json.dumps({"sqs": "test sqs"}),
+                        "MessageStructure": "json",
+                    }
+                ],
+            )
+        snapshot.match("no-default-key-json", e.value.response)
+
     @pytest.mark.aws_validated
     def test_subscribe_to_sqs_with_queue_url(
         self, sns_create_topic, sqs_create_queue, sns_subscription, snapshot
diff --git a/tests/integration/test_sns.snapshot.json b/tests/integration/test_sns.snapshot.json
--- a/tests/integration/test_sns.snapshot.json
+++ b/tests/integration/test_sns.snapshot.json
@@ -1141,7 +1141,7 @@
     }
   },
   "tests/integration/test_sns.py::TestSNSProvider::test_publish_batch_messages_from_sns_to_sqs": {
-    "recorded-date": "09-08-2022, 11:36:59",
+    "recorded-date": "12-06-2023, 18:15:59",
     "recorded-content": {
       "sub-attrs-raw-true": {
         "Attributes": {
@@ -1152,6 +1152,7 @@
           "Protocol": "sqs",
           "RawMessageDelivery": "true",
           "SubscriptionArn": "arn:aws:sns:<region>:111111111111:<resource:3>:<resource:2>",
+          "SubscriptionPrincipal": "<sub-principal>",
           "TopicArn": "arn:aws:sns:<region>:111111111111:<resource:3>"
         },
         "ResponseMetadata": {
@@ -1161,10 +1162,6 @@
       },
       "publish-batch": {
         "Failed": [],
-        "ResponseMetadata": {
-          "HTTPHeaders": {},
-          "HTTPStatusCode": 200
-        },
         "Successful": [
           {
             "Id": "1",
@@ -1181,8 +1178,16 @@
           {
             "Id": "4",
             "MessageId": "<uuid:4>"
+          },
+          {
+            "Id": "5",
+            "MessageId": "<uuid:5>"
           }
-        ]
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
       },
       "messages": {
         "Messages": [
@@ -1202,7 +1207,7 @@
                 "StringValue": "19.12"
               }
             },
-            "MessageId": "<uuid:5>",
+            "MessageId": "<uuid:6>",
             "ReceiptHandle": "<receipt-handle:1>"
           },
           {
@@ -1225,7 +1230,7 @@
                 "StringValue": "109.12"
               }
             },
-            "MessageId": "<uuid:6>",
+            "MessageId": "<uuid:7>",
             "ReceiptHandle": "<receipt-handle:2>"
           },
           {
@@ -1237,7 +1242,7 @@
             },
             "Body": "Test Message without attribute",
             "MD5OfBody": "<md5-hash>",
-            "MessageId": "<uuid:7>",
+            "MessageId": "<uuid:8>",
             "ReceiptHandle": "<receipt-handle:3>"
           },
           {
@@ -1249,15 +1254,27 @@
             },
             "Body": "Test Message without subject",
             "MD5OfBody": "<md5-hash>",
-            "MessageId": "<uuid:8>",
+            "MessageId": "<uuid:9>",
             "ReceiptHandle": "<receipt-handle:4>"
+          },
+          {
+            "Attributes": {
+              "ApproximateFirstReceiveTimestamp": "timestamp",
+              "ApproximateReceiveCount": "1",
+              "SenderId": "<sender-id>",
+              "SentTimestamp": "timestamp"
+            },
+            "Body": "test sqs",
+            "MD5OfBody": "<md5-hash>",
+            "MessageId": "<uuid:10>",
+            "ReceiptHandle": "<receipt-handle:5>"
           }
         ]
       }
     }
   },
   "tests/integration/test_sns.py::TestSNSProvider::test_publish_batch_exceptions": {
-    "recorded-date": "09-08-2022, 11:35:46",
+    "recorded-date": "12-06-2023, 18:22:01",
     "recorded-content": {
       "no-group-id": {
         "Error": {
@@ -1302,6 +1319,17 @@
           "HTTPHeaders": {},
           "HTTPStatusCode": 400
         }
+      },
+      "no-default-key-json": {
+        "Error": {
+          "Code": "InvalidParameter",
+          "Message": "Invalid parameter: Message Structure - No default entry in JSON message body",
+          "Type": "Sender"
+        },
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 400
+        }
       }
     }
   },