Repository: localstack/localstack

Issue:
bug: SQS reports that "MessageGroupId" is required when it is not
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

When calling `ReceiveMessage` on a FIFO queue, I _sometimes_ get HTTP 400 with the message: 

```
  [Aws\Sqs\Exception\SqsException]                                                                                                    
  Error executing "ReceiveMessage" on "http://localhost:4566/queue/us-west-2/000000000000/commands.fifo"; AWS HTTP error: Client er   
  ror: `POST http://localhost:4566/queue/us-west-2/000000000000/commands.fifo` resulted in a `400 Bad Request` response:              
  <?xml version='1.0' encoding='utf-8'?>                                                                                              
  <ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Code>Mi (truncated...)                                    
   MissingParameter (client): The request must contain the parameter MessageGroupId. - <?xml version='1.0' encoding='utf-8'?>         
  <ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Code>MissingParameter</Code><Message>The request must c   
  ontain the parameter MessageGroupId.</Message><Type>Sender</Type></Error><RequestId>88L3LXATEGVS0JF5S9BZDEQA95ELASZCKL4EJM5NCX9Y3   
  DHMP2RK</RequestId></ErrorResponse>  
```

There doesn't seem to be a particular pattern to it, and restarting localstack makes it go away. 

### Expected Behavior

`RecieveMessage` should never require a `MessageGroupId` parameter, because such a parameter [does not exist][1].

[1]: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

```
docker compose up -d
```

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

My calling code is (PHP SDK):

```php
        $options = [
            'QueueUrl' => $this->config->url,
            'WaitTimeSeconds' => 4,
            'MaxNumberOfMessages' => 10,
            'VisibilityTimeout' => 10,
        ];

        $result = $this->client->receiveMessage($options);
```

### Environment

```markdown
- OS: Mac OS 12.3
- LocalStack: 0.14.3
```


### Anything else?

_No response_


Discussion:
Hey Woody @shadowhand, thanks for reporting. Could you please share the code you use to create the queue and to send the messages?

If you could reproduce it with `awslocal sqs` cli commands that would be extremely helpful, or at least the SDK invocations.

Moreover: what version of the PHP SDK are you using?

For now you can start localstack with `PROVIDER_OVERRIDE_SQS=legacy` to use the old (deprecated) provider to make it work again.
AWS SDK version 3.222.15

I can't really provide the info that we are using to create the queue, because we are reading the values directly out of `serverless.yml` and then manipulating the data to create the queue parameters. This particular queue has the following config:

```yml
    CommandsQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: commands.fifo
        FifoQueue: true
        ContentBasedDeduplication: true
        VisibilityTimeout: 60
        KmsMasterKeyId: ${self:custom.config.QueueKms}
        RedrivePolicy:
          maxReceiveCount: 3
          deadLetterTargetArn: !GetAtt CommandsFailedQueue.Arn
```
```
docker exec -it localstack awslocal --region=us-west-2 sqs receive-message --queue-url "http://localhost:4566/queue/us-west-2/000000000000/commands.fifo" --wait-time-seconds 4 --max-number-of-messages 1 --visibility-timeout 4

An error occurred (MissingParameter) when calling the ReceiveMessage operation: The request must contain the parameter MessageGroupId.
```

And then after about 3-4 tries of the same `awslocal` command I will get:

```
{
    "Messages": [
        {
            "MessageId": "3bd25bf4-8e1f-4491-84e4-577949b69e8c",
            "ReceiptHandle": "oN4dKWyfod9HXzeLtjUbzEFfxU1QNQ5dxMm7FjbnVR91LPM0kDUpK6skPMmDahns3NN8ROIDzKoeh2KSgByl4pdyArTDyWlw87SQAIXFxs3zZTQxYJKnYxmpDzLn4rqfPzEACC8rkmtevhGVCSxS79Eodi8xLkJ4RPOtKugLuGBv=",
            "MD5OfBody": "f39fbc626e4b8d4cd97b2c4404a91105",
            "Body": "/* snip */"
        }
    ]
}
```
Is it possible that the new SQS backend is not thread safe? Because we have a cron job operating in the background that performs the ReceiveMessage every minute. 
I'm trying to avoid reverting to `PROVIDER_OVERRIDE_SQS=legacy` backend because it is incredibly slow and often times out for us, causing tests to fail transiently.

Also refs https://github.com/localstack/localstack/issues/6102, though the issues may not be related, but it does require me to include the `--region` in the `awslocal` call.
i'll start some investigations. since the `MissingParameter` on `MessageGroupId` is only raised on `send_message` events, and you are having the issues on `receive_message`, i'm assuming it has something to do with the combination of fifo content-based dedup and the redrive policy.
@thrau I looked through the SQS API and didn't see anything obvious that could raise this error. Is it possible that it is being raised when the visibility timeout causes a FIFO message to be returned to the queue? It just seems like an impossible error to encounter with a ReceiveMessage operation.
@shadowhand could you also provide the config of the `CommandsFailedQueue` DLQ?
```yml
    CommandsFailedQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: commands-failed.fifo
        FifoQueue: true
        ContentBasedDeduplication: true
        MessageRetentionPeriod: 1209600 # 14 days
        KmsMasterKeyId: ${self:custom.config.QueueKms}
```
the reason it may be occurring in `ReceiveMessage` is because we are potentially doing a `put` to a dead-letter-queue here https://github.com/localstack/localstack/blob/46adaab4c308dee6cffe2edc2ad806131dbcab7a/localstack/services/sqs/provider.py#L1082, and if that doesn't have the same de-duplication strategy the error may be raised.

but it seems your DLQ also has contend-based dedup, so :shrug: will check
Ah, that is interesting. Is it possible that when the message is pulled up:

https://github.com/localstack/localstack/blob/46adaab4c308dee6cffe2edc2ad806131dbcab7a/localstack/services/sqs/provider.py#L1070-L1073

That it is not including the `MessageGroupId` attribute, and therefore failing when trying to deliver the (FIFO) DLQ?
Your scenario seems very possible to me, and I am guessing this is the line where the error is raised:

https://github.com/localstack/localstack/blob/46adaab4c308dee6cffe2edc2ad806131dbcab7a/localstack/services/sqs/provider.py#L533

Resolving patch:
diff --git a/localstack/services/sqs/provider.py b/localstack/services/sqs/provider.py
--- a/localstack/services/sqs/provider.py
+++ b/localstack/services/sqs/provider.py
@@ -1113,7 +1113,11 @@ def _dead_letter_check(
             dl_queue = self._require_queue_by_arn(context, dead_letter_target_arn)
             # TODO: this needs to be atomic?
             dead_message = std_m.message
-            dl_queue.put(message=dead_message)
+            dl_queue.put(
+                message=dead_message,
+                message_deduplication_id=std_m.message_deduplication_id,
+                message_group_id=std_m.message_group_id,
+            )
             queue.remove(std_m.message["ReceiptHandle"])
             return True
         else:


Test patch:
diff --git a/tests/integration/test_sqs.py b/tests/integration/test_sqs.py
--- a/tests/integration/test_sqs.py
+++ b/tests/integration/test_sqs.py
@@ -59,6 +59,18 @@ def queue_exists(sqs_client, queue_url: str) -> bool:
         raise
 
 
+def get_queue_arn(sqs_client, queue_url: str) -> str:
+    """
+    Returns the given Queue's ARN. Expects the Queue to exist.
+
+    :param sqs_client: the boto3 client
+    :param queue_url: the queue URL
+    :return: the QueueARN
+    """
+    response = sqs_client.get_queue_attributes(QueueUrl=queue_url, AttributeNames=["QueueArn"])
+    return response["Attributes"]["QueueArn"]
+
+
 class TestSqsProvider:
     @pytest.mark.only_localstack
     def test_get_queue_url_contains_request_host(self, sqs_client, sqs_create_queue, monkeypatch):
@@ -750,9 +762,7 @@ def test_standard_queue_cannot_have_fifo_suffix(self, sqs_create_queue):
     def test_redrive_policy_attribute_validity(self, sqs_create_queue, sqs_client):
         dl_queue_name = f"dl-queue-{short_uid()}"
         dl_queue_url = sqs_create_queue(QueueName=dl_queue_name)
-        dl_target_arn = sqs_client.get_queue_attributes(
-            QueueUrl=dl_queue_url, AttributeNames=["QueueArn"]
-        )["Attributes"]["QueueArn"]
+        dl_target_arn = get_queue_arn(sqs_client, dl_queue_url)
         queue_name = f"queue-{short_uid()}"
         queue_url = sqs_create_queue(QueueName=queue_name)
         valid_max_receive_count = "42"
@@ -1082,6 +1092,50 @@ def test_dead_letter_queue_execution(
         result_recv = sqs_client.receive_message(QueueUrl=dl_queue_url, VisibilityTimeout=0)
         assert result_recv["Messages"][0]["Body"] == json.dumps(payload)
 
+    @pytest.mark.aws_validated
+    def test_dead_letter_queue_with_fifo_and_content_based_deduplication(
+        self, sqs_client, sqs_create_queue
+    ):
+        dlq_url = sqs_create_queue(
+            QueueName=f"test-dlq-{short_uid()}.fifo",
+            Attributes={
+                "FifoQueue": "true",
+                "ContentBasedDeduplication": "true",
+                "MessageRetentionPeriod": "1209600",
+            },
+        )
+        dlq_arn = get_queue_arn(sqs_client, dlq_url)
+
+        queue_url = sqs_create_queue(
+            QueueName=f"test-queue-{short_uid()}.fifo",
+            Attributes={
+                "FifoQueue": "true",
+                "ContentBasedDeduplication": "true",
+                "VisibilityTimeout": "60",
+                "RedrivePolicy": json.dumps({"deadLetterTargetArn": dlq_arn, "maxReceiveCount": 2}),
+            },
+        )
+
+        response = sqs_client.send_message(
+            QueueUrl=queue_url, MessageBody="foobar", MessageGroupId="1"
+        )
+        message_id = response["MessageId"]
+
+        # receive the messages twice, which is the maximum allwed
+        sqs_client.receive_message(QueueUrl=queue_url, WaitTimeSeconds=1, VisibilityTimeout=0)
+        sqs_client.receive_message(QueueUrl=queue_url, WaitTimeSeconds=1, VisibilityTimeout=0)
+        # after this receive call the message should be in the DLQ
+        sqs_client.receive_message(QueueUrl=queue_url, WaitTimeSeconds=1, VisibilityTimeout=0)
+
+        # check the DLQ
+        response = sqs_client.receive_message(QueueUrl=dlq_url, WaitTimeSeconds=10)
+        assert (
+            len(response["Messages"]) == 1
+        ), f"invalid number of messages in DLQ response {response}"
+        message = response["Messages"][0]
+        assert message["MessageId"] == message_id
+        assert message["Body"] == "foobar"
+
     def test_dead_letter_queue_max_receive_count(self, sqs_client, sqs_create_queue):
         queue_name = f"queue-{short_uid()}"
         dead_letter_queue_name = f"dl-queue-{short_uid()}"