Repository: localstack/localstack

Issue:
bug: KeyError: 'Token' while subscribing to SNS topic
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

I have localstack+terraform running in our CI system that creates several SQS queues, SNS topics, and subscribing the SQS queues to the SNS topics.

Intermittently but frequently (about 35% of the time), this fails with the log message:

```
error waiting for SNS Topic Subscription (arn:aws:sns:us-east-1:000000000000:xometry-app-partner-registration:0e2fde2b-34af-49ad-812b-8fff16b63790) confirmation: timeout while waiting for state to become 'false' (last state: 'true', timeout: 2m0s)
```

I turned on DEBUG within localstack and dug this back to the following error/stack:

```
KeyError: 'Token'
  if v["Token"] == token and v["TopicArn"] == topic_arn:
File "/opt/code/localstack/localstack/services/sns/provider.py", line 517, in confirm_subscription
  self.confirm_subscription(context, topic_arn, subscription_token)
File "/opt/code/localstack/localstack/services/sns/provider.py", line 787, in subscribe
  return self.fn(*args, **kwargs)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
  result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
  return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
  skeleton_response = self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 91, in __call__
  handler(chain, context, response)
File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
  handler(self, self.context, response)
File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
  Traceback (most recent call last):
```


### Expected Behavior

Heh, I expect the subscription to work without an error stack.

Or... if the failure is expected, for the API call to fail with an appropriate error message so I can retry.

### How are you starting LocalStack?

Custom (please describe below)

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

I'm using the localstack helm chart.

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

`terraform init && terraform apply -var "localstack_url=http://localstack.$KUBE_NS:4566" -auto-approve`


### Environment

```markdown
- OS: Kube cluster, Linux-based x86-64
- LocalStack: installed via helm chart, version 1.0.4 from docker hub
```


### Anything else?

The terraform workspace that reproduces the problem is attached... I don't know whether we could reduce the problem by reducing the number of topics/queues being created.
[localstack-terraform.zip](https://github.com/localstack/localstack/files/9479165/localstack-terraform.zip)



Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).

Resolving patch:
diff --git a/localstack/services/sns/provider.py b/localstack/services/sns/provider.py
--- a/localstack/services/sns/provider.py
+++ b/localstack/services/sns/provider.py
@@ -528,7 +528,7 @@ def confirm_subscription(
         sns_backend = SNSBackend.get()
         sub_arn = None
         for k, v in sns_backend.subscription_status.items():
-            if v["Token"] == token and v["TopicArn"] == topic_arn:
+            if v.get("Token") == token and v["TopicArn"] == topic_arn:
                 v["Status"] = "Subscribed"
                 sub_arn = k
         for k, v in sns_backend.sns_subscriptions.items():


Test patch: