Repository: localstack/localstack

Issue:
bug: SES sending email not firing events to SNS topic
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

I am trying to receive SES events in a SNS topic using localstack. A queue is connected to the SNS topic in order to get messages.
As an example, with SES we send an email, we use a config set that says, that whenever we send an email, the delivery event is fired to an SNS topic.
Queue listens to the SNS topic, we get this information polling from the queue.

### Expected Behavior

When sending email via SES in localstack, the delivery event gets fired to the SNS topic.

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

I started localstack with docker-compose:

```
version: '3.0'

services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - SERVICES=sqs,sns,ses
      - AWS_DEFAULT_REGION=eu-central-1
      - EDGE_PORT=4566
    ports:
      - '4566-4597:4566-4597'
    volumes:
      - ./localstack_setup.sh:/docker-entrypoint-initaws.d/setup.sh
      - '${TMPDIR:-/tmp/localstack}:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'
networks:
  default:
    name: development-network
    external: true

```

## verify email address
`aws --endpoint-url=http://localhost:4566 ses verify-email-identity --email-address sender@test.com --profile test-profile --region eu-central-1 --output table | cat`

## create queue for SNS - SES events
`aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name ses_events_queue2 --attributes '{"SqsManagedSseEnabled": "false"}' --profile test-profile --region eu-central-1 --output table | cat`

## update queue policy
```
aws --endpoint-url=http://localhost:4566 sqs set-queue-attributes --queue-url http://localhost:4566/000000000000/ses_events_queue --attributes '{"Policy" : "{\"Id\": \"Policy1564523767951\",\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"Stmt1564523766749\",\"Action\": \"sqs:*\",\"Effect\": \"Allow\",\"Resource\": \"arn:aws:sqs:eu-central-1:078250808814:ses_events_queue2\",\"Condition\": {\"ArnEquals\": {\"aws:SourceArn\": \"arn:aws:sns:eu-central-1:078250808814:ses_events_topic\"}},\"Principal\": \"*\"}]}"}' --profile test-profile --region eu-central-1 --output table | cat

aws --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/000000000000/ses_events_queue --attribute-names All --profile test-profile --region eu-central-1 --output table | cat
```

## create SNS topic for SES events
`aws --endpoint-url=http://localhost:4566 sns create-topic --name ses_events_topic --region eu-central-1 --profile test-profile --output table`

## subscribe queue to topic
`aws --endpoint-url=http://localhost:4566 sns subscribe --topic-arn arn:aws:sns:eu-central-1:000000000000:ses_events_topic --protocol sqs --notification-endpoint arn:aws:sqs:eu-central-1:000000000000:ses_events_queue --profile test-profile --region eu-central-1 --output table | cat`

## create config set
`aws --endpoint-url=http://localhost:4566 ses create-configuration-set --configuration-set "{\"Name\":\"ses_config_set\"}" --profile test-profile --region eu-central-1 --output table | cat`

## create event destination
`aws --endpoint-url=http://localhost:4566 ses create-configuration-set-event-destination --configuration-set-name ses_config_set --event-destination '{"Name":"some_name2","Enabled":true,"MatchingEventTypes":["send","bounce","delivery","open","click"],"SNSDestination":{"TopicARN":"arn:aws:sns:eu-central-1:078250808814:ses_events_topic"}}' --profile test-profile --region eu-central-1 --output table | cat`

## send email via ses with created set
```
aws ses send-email --endpoint-url=http://localhost:4566 --destination '{"ToAddresses":["receiver@mail.com"]}' --message '{"Subject":{"Data":"foo subject"},"Body":{"Text":{"Data":"saml body"}}}' --configuration-set-name ses_config_set --from 'sender@test.com' --profile test-profile --region eu-central-1 --output table | cat
```

When I check messages in queue, it's empty.

I use same commands pointing to a sandbox in aws, and everything worked. In the aws GUI, I could create the sns, the queue, the config sets using same commands as in localstack, fire the event, and I could see the event being sent to the queue.

In localstack is not working and it's very difficult to check which part is not working.


### Environment

```markdown
- OS: macOS Monterey Version 12.6
- LocalStack: https://hub.docker.com/r/localstack/localstack
```


### Anything else?

_No response_


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).
Hello, @simonrw 

first of all, thanks a lot for your great work with this PR. 

I have been trying it in our app and indeed we are getting event notification (at least for send and delivered status which are the ones I can test), but unless I'm missing something, the email messageId is not being sent as part of the "Message" key, therefore I cannot associate the event notification with the email it belongs.

Can you please take a look and let me know if I'm missing something? 

I'm guessing the issue is here: for example, when building the event_payload https://github.com/localstack/localstack/blob/9770dc75074043909ce373902cc1b6fa54560b56/localstack/services/ses/provider.py#L377

This is the event notification I'm getting from Localstack:

```json
{
   "Type":"Notification",
   "MessageId":"b64a7b57-6358-48c5-8927-2a06e820c5f7",
   "TopicArn":"arn:aws:sns:eu-central-1:000000000000:ses_events_topic",
   "Message":"{\"eventType\": \"Delivery\", \"mail\": {\"timestamp\": \"2022-11-25T09:25:24.231419+00:00\", \"source\": \"Sender <sender@test.com>\", \"sourceArn\": \"arn:aws:ses:eu-central-1:000000000000:identity/Sender <sender@test.com>\", \"sendingAccountId\": \"000000000000\", \"destination\": [\"Recipient <recipient@test.com>\"]}, \"delivery\": {\"recipients\": [\"Recipient <recipient@test.com>\"], \"timestamp\": \"2022-11-25T09:25:24.231419+00:00\"}}",
   "Timestamp":"2022-11-25T09:25:24.238Z",
   "SignatureVersion":"1",
   "Signature":"EXAMPLEpH+..",
   "SigningCertURL":"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem",
   "UnsubscribeURL":"http://localhost:4566/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-central-1:000000000000:ses_events_topic:a13fc864-9f10-4687-a6a0-74a65eb353c8",
   "Subject":"Amazon SES Email Event Notification"
```
And this is an example from AWS:

```json
{
  "Type" : "Notification",
  "MessageId" : "4c7e77c2-a65b-502a-8092-2c3b218ad866",
  "TopicArn" : "arn:aws:sns:eu-central-1:xxxxxxx:ses_events_topic",
  "Subject" : "Amazon SES Email Event Notification",
  "Message" : "{\"eventType\":\"Open\",\"mail\":{\"timestamp\":\"2022-11-24T13:25:09.124Z\",\"source\":\"ernesto.gimeno@xxx.com\",\"sendingAccountId\":\"xxxxxxx\",\"messageId\":\"01070184a9d04384-54923915-eddf-4934-a4aa-d9521e1a8255-000000\",\"destination\":[\"ergisan@gmail.com\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"From\",\"value\":\"ernesto.gimeno@xxx.com\"},{\"name\":\"To\",\"value\":\"xxxx@gmail.com\"},{\"name\":\"Subject\",\"value\":\"foo subject\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"Content-Type\",\"value\":\"text/html; charset=UTF-8\"},{\"name\":\"Content-Transfer-Encoding\",\"value\":\"7bit\"},{\"name\":\"Message-ID\",\"value\":\"null\"}],\"commonHeaders\":{\"from\":[\"ernesto.gimeno@xxx.com\"],\"to\":[\"ergisan@gmail.com\"],\"messageId\":\"01070184a9d04384-54923915-eddf-4934-a4aa-d9521e1a8255-000000\",\"subject\":\"foo subject\"},\"tags\":{\"ses:operation\":[\"SendEmail\"],\"ses:configuration-set\":[\"ses_config_set\"],\"ses:source-ip\":[\"3.73.66.62\"],\"ses:from-domain\":[\"xxx.com\"],\"ses:caller-identity\":[\"xx-admin\"]}},\"open\":{\"timestamp\":\"2022-11-24T14:54:09.301Z\",\"userAgent\":\"Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)\",\"ipAddress\":\"66.249.93.147\"}}\n",
  "Timestamp" : "2022-11-24T14:54:09.410Z",
  "SignatureVersion" : "1",
  "Signature" : "S/jaH6UcECs1W2e+h5APv/8nTmQn4MbrZsgeB4d70V+ui1szYNKMh66wABh3i3MWXTTfMao1dV5o6VD0UVtriBO3LEvD9x5rLlUGvRHEVNjyJpkL+AbEdCoZxSH5ObfDpsC6rduHGL+utuWDvRwsifKWzHPv98rtBJSMYafCzPCY6bY3sGXVUEkdIDR1vqIsPCoobGHp3A7wm3+BGaJaSogdhfrbI7RtXH2bb8kZ/Nn29Tg043ajrtquSmEbElCM27w53Ye2XLB2pbaqGPKGKn03FvEZeaxr00cbbfY8cjBLzQSM6dwRYmZRqwhPJskKwtys6cwLuITLi5xLwnyihw==",
  "SigningCertURL" : "https://sns.eu-central-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",
  "UnsubscribeURL" : "https://sns.eu-central-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-central-1:xxxxxx:ses_events_topic:c5810a4e-f8e6-41a6-bfca-b5b7e9dfe33e"
}
```

Thanks in advance!

Resolving patch:
diff --git a/localstack/services/ses/provider.py b/localstack/services/ses/provider.py
--- a/localstack/services/ses/provider.py
+++ b/localstack/services/ses/provider.py
@@ -1,3 +1,4 @@
+import dataclasses
 import json
 import logging
 import os
@@ -244,8 +245,13 @@ def send_email(
             if not sns_destination_arn:
                 continue
 
-            emitter.emit_send_event(source, destination["ToAddresses"], sns_destination_arn)
-            emitter.emit_delivery_event(source, destination["ToAddresses"], sns_destination_arn)
+            payload = SNSPayload(
+                message_id=response["MessageId"],
+                sender_email=source,
+                destination_addresses=destination["ToAddresses"],
+            )
+            emitter.emit_send_event(payload, sns_destination_arn)
+            emitter.emit_delivery_event(payload, sns_destination_arn)
 
         text_part = message["Body"].get("Text", {}).get("Data")
         html_part = message["Body"].get("Html", {}).get("Data")
@@ -352,11 +358,17 @@ def publish(self, TopicArn: str, Message: str, Subject: Optional[str] = None):
         ...
 
 
+@dataclasses.dataclass(frozen=True)
+class SNSPayload:
+    message_id: str
+    sender_email: Address
+    destination_addresses: AddressList
+
+
 class SNSEmitter:
     def __init__(
         self,
         context: RequestContext,
-        client: Optional[SNSClient] = None,
     ):
         self.context = context
 
@@ -369,19 +381,18 @@ def emit_create_configuration_set_event_destination_test_message(
             Message="Successfully validated SNS topic for Amazon SES event publishing.",
         )
 
-    def emit_send_event(
-        self, sender_email: Address, destination_addresses: AddressList, sns_topic_arn: str
-    ):
+    def emit_send_event(self, payload: SNSPayload, sns_topic_arn: str):
         now = datetime.now(tz=timezone.utc)
 
         event_payload = {
             "eventType": "Send",
             "mail": {
                 "timestamp": now.isoformat(),
-                "source": sender_email,
-                "sourceArn": f"arn:aws:ses:{self.context.region}:{self.context.account_id}:identity/{sender_email}",
+                "source": payload.sender_email,
+                "sourceArn": f"arn:aws:ses:{self.context.region}:{self.context.account_id}:identity/{payload.sender_email}",
                 "sendingAccountId": self.context.account_id,
-                "destination": destination_addresses,
+                "destination": payload.destination_addresses,
+                "messageId": payload.message_id,
             },
             "send": {},
         }
@@ -392,22 +403,21 @@ def emit_send_event(
             Subject="Amazon SES Email Event Notification",
         )
 
-    def emit_delivery_event(
-        self, sender_email: Address, destination_addresses: AddressList, sns_topic_arn: str
-    ):
+    def emit_delivery_event(self, payload: SNSPayload, sns_topic_arn: str):
         now = datetime.now(tz=timezone.utc)
 
         event_payload = {
             "eventType": "Delivery",
             "mail": {
                 "timestamp": now.isoformat(),
-                "source": sender_email,
-                "sourceArn": f"arn:aws:ses:{self.context.region}:{self.context.account_id}:identity/{sender_email}",
+                "source": payload.sender_email,
+                "sourceArn": f"arn:aws:ses:{self.context.region}:{self.context.account_id}:identity/{payload.sender_email}",
                 "sendingAccountId": self.context.account_id,
-                "destination": destination_addresses,
+                "destination": payload.destination_addresses,
+                "messageId": payload.message_id,
             },
             "delivery": {
-                "recipients": destination_addresses,
+                "recipients": payload.destination_addresses,
                 "timestamp": now.isoformat(),
             },
         }


Test patch:
diff --git a/tests/integration/test_ses.py b/tests/integration/test_ses.py
--- a/tests/integration/test_ses.py
+++ b/tests/integration/test_ses.py
@@ -235,7 +235,6 @@ def test_clone_receipt_rule_set(self, ses_client):
             "$..Message.delivery.processingTimeMillis",
             "$..Message.delivery.reportingMTA",
             "$..Message.delivery.smtpResponse",
-            "$..Message.mail.messageId",
             "$..Message.mail.commonHeaders",
             "$..Message.mail.headers",
             "$..Message.mail.headersTruncated",
@@ -267,6 +266,7 @@ def test_ses_sns_topic_integration(
             [
                 snapshot.transform.regex(sender_email_address, "<sender-email-address>"),
                 snapshot.transform.regex(recipient_email_address, "<recipient-email-address>"),
+                snapshot.transform.key_value("messageId"),
             ]
             + snapshot.transform.sns_api()
         )
@@ -308,4 +308,17 @@ def test_ses_sns_topic_integration(
         )
 
         messages = sqs_receive_num_messages(sqs_queue, 3)
+        # the messages may arrive out of order so sort
+
+        def sort_fn(message):
+            if "Successfully validated" in message["Message"]:
+                return 0
+            elif json.loads(message["Message"])["eventType"] == "Send":
+                return 1
+            elif json.loads(message["Message"])["eventType"] == "Delivery":
+                return 2
+            else:
+                raise ValueError("bad")
+
+        messages.sort(key=sort_fn)
         snapshot.match("messages", messages)
diff --git a/tests/integration/test_ses.snapshot.json b/tests/integration/test_ses.snapshot.json
--- a/tests/integration/test_ses.snapshot.json
+++ b/tests/integration/test_ses.snapshot.json
@@ -1,6 +1,6 @@
 {
   "tests/integration/test_ses.py::TestSES::test_ses_sns_topic_integration": {
-    "recorded-date": "21-11-2022, 09:38:18",
+    "recorded-date": "25-11-2022, 12:16:13",
     "recorded-content": {
       "messages": [
         {
@@ -26,7 +26,7 @@
               "source": "<sender-email-address>",
               "sourceArn": "arn:aws:ses:<region>:111111111111:identity/<sender-email-address>",
               "sendingAccountId": "111111111111",
-              "messageId": "01070184998d7f5f-9821ae1c-7289-498a-9e6f-1baf98cc547d-000000",
+              "messageId": "<message-id:1>",
               "destination": [
                 "<recipient-email-address>"
               ],
@@ -64,7 +64,7 @@
                 "to": [
                   "<recipient-email-address>"
                 ],
-                "messageId": "01070184998d7f5f-9821ae1c-7289-498a-9e6f-1baf98cc547d-000000",
+                "messageId": "<message-id:1>",
                 "subject": "foo subject"
               },
               "tags": {
@@ -72,13 +72,13 @@
                   "SendEmail"
                 ],
                 "ses:configuration-set": [
-                  "config-set-f721b02c"
+                  "config-set-90b751e6"
                 ],
                 "ses:source-ip": [
                   "80.189.216.182"
                 ],
                 "ses:from-domain": [
-                  "localstack.cloud"
+                  "gmail.com"
                 ],
                 "ses:caller-identity": [
                   "localstack-testing"
@@ -105,7 +105,7 @@
               "source": "<sender-email-address>",
               "sourceArn": "arn:aws:ses:<region>:111111111111:identity/<sender-email-address>",
               "sendingAccountId": "111111111111",
-              "messageId": "01070184998d7f5f-9821ae1c-7289-498a-9e6f-1baf98cc547d-000000",
+              "messageId": "<message-id:1>",
               "destination": [
                 "<recipient-email-address>"
               ],
@@ -143,7 +143,7 @@
                 "to": [
                   "<recipient-email-address>"
                 ],
-                "messageId": "01070184998d7f5f-9821ae1c-7289-498a-9e6f-1baf98cc547d-000000",
+                "messageId": "<message-id:1>",
                 "subject": "foo subject"
               },
               "tags": {
@@ -151,13 +151,13 @@
                   "SendEmail"
                 ],
                 "ses:configuration-set": [
-                  "config-set-f721b02c"
+                  "config-set-90b751e6"
                 ],
                 "ses:source-ip": [
                   "80.189.216.182"
                 ],
                 "ses:from-domain": [
-                  "localstack.cloud"
+                  "gmail.com"
                 ],
                 "ses:caller-identity": [
                   "localstack-testing"
@@ -169,11 +169,11 @@
             },
             "delivery": {
               "timestamp": "date",
-              "processingTimeMillis": 497,
+              "processingTimeMillis": 1043,
               "recipients": [
                 "<recipient-email-address>"
               ],
-              "smtpResponse": "250 2.0.0 OK  1669023498 q4-20020adffec4000000b002368e4d5d74si6016172wrs.528 - gsmtp",
+              "smtpResponse": "250 2.0.0 OK  1669378573 f10-20020a5d64ca000000b0024207ba48d6si799363wri.326 - gsmtp",
               "reportingMTA": "b224-7.smtp-out.<region>.amazonses.com"
             }
           },