Repository: localstack/localstack

Issue:
bug: can't receive message from fifo queue with the same MessageGroupId
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

with python boto3 client

- send a message to fifo queue with MessageGroupId='1'
- receive the message
- delete the message
- send another message to the queue with MessageGroupId='1'
- try to receive the message, get none
- check message availability with get_queue_attributes, get `{'ApproximateNumberOfMessages': '1', 'ApproximateNumberOfMessagesNotVisible': '0', 'ApproximateNumberOfMessagesDelayed': '0', ...`

### Expected Behavior

expecting to receive message from the queue

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

docker-compose up


### Environment

```markdown
- OS:MacOS Sonoma 14.2.1
- LocalStack: latest
```


### Anything else?

I tried to run with Rosetta, no result


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).
Saw an issue similar to this in our builds a few weeks ago. The symptom was that our tests would fail apparently because only one queue message was ever delivered. The localstack-pro version was `3.0.3.dev20240105221043` in the docker logs of the builds that failed. We switched to use the `stable` tag as a quick fix at the time, and the issue went away. The issue resurfaced today (we are still using the `stable` tag) and the version number in the docker logs is being reported as `3.1.0`. Changing the image version from `stable` down to `3.0.2` brings us back to the expected behavior and our tests pass again.

I can confirm with testing that using a randomized string for the MessageGroupId in each message will allow more than one queue message to flow through a given FIFO queue, while using a static value makes it so that only the first message will be delivered.
I don't know if can be helpful but I have created a minimal spring boot project with the described scenarios in this [repository](https://github.com/classicPintus/spring-boot-sqs-localstack)
Just echoing what @schraitle-pomelo mentioned - our tests failed as only one message gets delivered. In those, the `MessageGroupId` was the same for the messages sent. Using a unique `MessageGroupId` for each message allows more than one message to be delivered through the FIFO queue.

For now sticking to the `3.0.2` version.

**Edit**: The issue seems to have been introduced in commit https://github.com/localstack/localstack/commit/367ff3309bc406bdb8023bdbb6cf8bd801649a46 - removing the added check by using this (naive) Dockerfile makes our tests pass again:

```Dockerfile
FROM localstack/localstack:stable
RUN sed -i '1019,1020d' /opt/code/localstack/localstack/services/sqs/models.py
```

Then use this image in an updated `docker-compose.yaml`:
```diff
diff --git a/docker-compose.yaml b/docker-compose.yaml
index 6d14eca..3375093 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -2,7 +2,8 @@ version: '3.3'
 services:
   localstack:
     # REMEMBER TO CHANGE docker-compose.yml AND .github/workflows/build.yml AT THE SAME TIME
-    image: localstack/localstack:stable
+    # image: localstack/localstack:stable
+    build: .
     environment:
       SERVICES: dynamodb,sqs
       DEFAULT_REGION: eu-central-1
```

Not sure about the internals going on here, but hopefully this helps :)
okay, version 3.0.2 works just fine, thanks a lot, I'm going close the issue now
@Nenavsegda it's still a valid bug, I'd suggest to reopen please. Otherwise I'll create a new issue. I can also repro it:

```shell
root@5bdd86475de1:/opt/code/localstack# awslocal sqs create-queue --queue-name test.fifo --attributes FifoQueue=true
{
    "QueueUrl": "http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo"
}
root@5bdd86475de1:/opt/code/localstack# awslocal sqs send-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo --message-body '{"hello": "world"}' --message-group-id foo --message-deduplication-id 1
{
    "MD5OfMessageBody": "49dfdd54b01cbcd2d2ab5e9e5ee6b9b9",
    "MessageId": "f00e4e5c-e98d-42a9-b8a3-b14d4f690a4e",
    "SequenceNumber": "14661116923572387840"
}
root@5bdd86475de1:/opt/code/localstack# awslocal sqs receive-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo
{
    "Messages": [
        {
            "MessageId": "f00e4e5c-e98d-42a9-b8a3-b14d4f690a4e",
            "ReceiptHandle": "MzU1NzMyZmQtNTkzMC00MWY5LWE4MGUtNTNiNmFlZGJjZDIzIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6dGVzdC5maWZvIGYwMGU0ZTVjLWU5OGQtNDJhOS1iOGEzLWIxNGQ0ZjY5MGE0ZSAxNzA2Nzc4NjQ5LjIxOTQ3",
            "MD5OfBody": "49dfdd54b01cbcd2d2ab5e9e5ee6b9b9",
            "Body": "{\"hello\": \"world\"}"
        }
    ]
}
root@5bdd86475de1:/opt/code/localstack# awslocal sqs send-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo --message-body '{"foo": "bar"}' --message-group-id foo --message-deduplication-id 2
{
    "MD5OfMessageBody": "94232c5b8fc9272f6f73a1e36eb68fcf",
    "MessageId": "71e31a0a-17d9-4e83-ac90-fd42382669b1",
    "SequenceNumber": "14661116923572387841"
}
root@5bdd86475de1:/opt/code/localstack# awslocal sqs receive-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo
root@5bdd86475de1:/opt/code/localstack# awslocal sqs send-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo --message-body '{"foo": "bar"}' --message-group-id bar --message-deduplication-id 3
{
    "MD5OfMessageBody": "94232c5b8fc9272f6f73a1e36eb68fcf",
    "MessageId": "caefe7cc-4355-4779-9b64-c1249d9bd4b9",
    "SequenceNumber": "14661116923572387842"
}
root@5bdd86475de1:/opt/code/localstack# awslocal sqs receive-message --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/test.fifo
{
    "Messages": [
        {
            "MessageId": "caefe7cc-4355-4779-9b64-c1249d9bd4b9",
            "ReceiptHandle": "ODg0NTI0NTQtOWM5MS00OTlmLWIzMWItODFjNTZlMThhZTBiIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6dGVzdC5maWZvIGNhZWZlN2NjLTQzNTUtNDc3OS05YjY0LWMxMjQ5ZDliZDRiOSAxNzA2Nzc4NjYzLjc5MjI3MDI=",
            "MD5OfBody": "94232c5b8fc9272f6f73a1e36eb68fcf",
            "Body": "{\"foo\": \"bar\"}"
        }
    ]
}
```
Hello and thanks for your reports! Could you please indicate if your test cases are working against AWS?

@jbauers, it looks like this should be the intended behavior per the AWS documentation:
> When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

If you could confirm that LocalStack is behaving differently than AWS here, that would be great. Thanks! 

Hi @bentsku, thanks for the quick reply. Can confirm that this is in fact expected behaviour:

```
➜  ~ aws sqs create-queue --queue-name test.fifo --attributes FifoQueue=true
➜  ~ aws sqs send-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo --message-body '{"hello": "world"}' --message-group-id foo --message-deduplication-id 1
➜  ~ aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo
➜  ~ aws sqs send-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo --message-body '{"foo": "bar"}' --message-group-id foo --message-deduplication-id 2
➜  ~ aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo
➜  ~ aws sqs send-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo --message-body '{"foo": "bar"}' --message-group-id bar --message-deduplication-id 3
➜  ~ aws sqs receive-message --queue-url https://sqs.eu-central-1.amazonaws.com/0123456789/test.fifo
```

☝️ yields the same result as `awslocal` + `localstack`. The caveat is that as per [1], **Lambda deletes the message** from the SQS queue after successful processing - so if you're using the SQS client directly in your handler, without LocalStack Lambda, you'll get different results depending on whether you're running tests with LocalStack or in an AWS + Lambda environment.

We'll update our tests - sorry for the noise, hope this helps others :)

[1] https://docs.aws.amazon.com/en_gb/lambda/latest/dg/with-sqs.html
I'm not sure this expected behavior, I'm sending unique messages with the same MessageGroupId.
Only the first request to the FIFO queue returns pending messages.

When the messages are deleted (Acknowledged) localstack SQS does not return the next batch of messages for that MessageGroupId, instead they appear as Available Messages but are never pulled.

![image](https://github.com/localstack/localstack/assets/1643692/e0080ed2-62f7-4379-b0ec-ca62d6de6147)

When targeting a real SQS FIFO queue, it will return new messages for this MessageGroupId when the previous messages are deleted.

Version: `3.1.1.dev`
Hey all,

> When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

According to this documentation @bentsku already posted, if there are two messages with the same group id and the first one is deleted (aka. acknowledged), the second one should be returned. This is not the case in localstack 3.1.0, but in localstack 3.0.2 and also in AWS. Like @jamescarter-le already pointed out in localstack 3.1.0 further messages group id are just not returned at all. This is not expected and a bug. @jbauers first example demonstrates it quite well.

just as additional context, here are changes we made to FIFO SQS queue handling
* https://github.com/localstack/localstack/pull/8238 (2.3)
* https://github.com/localstack/localstack/pull/9975 (3.1)

It seems the behavior we have right now is still not 100% correct. i don't have time to look into it right now, but will see whether someone can work on this in the team.

i think we need to do more testing of AWS behavior, but automated testing of ordering for FIFO queues and visibility timeout is very tricky. so a more manual testing approach would probably be better. perhaps it's also just a silly little thing missing somewhere

cc @steffyP 
I also get messages stuck from the same `MessageGroupId`, only the first one is delivered.
Reproduced it today on the latest version.
And can confirm that a custom image with [this](https://github.com/localstack/localstack/commit/367ff3309bc406bdb8023bdbb6cf8bd801649a46) change removed (like mentioned [here](https://github.com/localstack/localstack/issues/10107#issuecomment-1919078135)) is working fine, subsequent MessageGroupId messages are delivered.
> ☝️ yields the same result as `awslocal` + `localstack`. The caveat is that as per [1], **Lambda deletes the message** from the SQS queue after successful processing - so if you're using the SQS client directly in your handler, without LocalStack Lambda, you'll get different results depending on whether you're running tests with LocalStack or in an AWS + Lambda environment.

i think we do that, but should double check. cc @joe4dev 
> > ☝️ yields the same result as `awslocal` + `localstack`. The caveat is that as per [1], **Lambda deletes the message** from the SQS queue after successful processing - so if you're using the SQS client directly in your handler, without LocalStack Lambda, you'll get different results depending on whether you're running tests with LocalStack or in an AWS + Lambda environment.
> 
> i think we do that, but should double check. cc @joe4dev

Our SQS event source listener does delete the SQS messages [here](https://github.com/localstack/localstack/blob/361abbf986fb6753cf6544efb982c017521b9c1a/localstack/services/lambda_/event_source_listeners/sqs_event_source_listener.py#L199) upon successful processing using the `delete_messages` callback.


@jbauers I guess you are referring to the following Lambda event source mapping behavior, right?
> When Lambda reads a batch, the messages stay in the queue but are hidden for the length of the queue's [visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html). If your function successfully processes the batch, Lambda deletes the messages from the queue. By default, if your function encounters an error while processing a batch, all messages in that batch become visible in the queue again. For this reason, your function code must be able to process the same message multiple times without unintended side effects.

Source: https://docs.aws.amazon.com/en_gb/lambda/latest/dg/with-sqs.html

@jbauers Related to the LocalStack Lambda integration, did you observe any difference between AWS and LocalStack?
@joe4dev Yes, exactly. We're only using LocalStack SQS, **not** the Lambda integration, in our tests, therefore have an inconsistent setup compared to running on AWS, where we do use Lambda + SQS, and then obviously need to account for the Lambda integration handling message deletion on AWS vs. that not happening automatically in our tests. I'm therefore unable to comment on this:
> Related to the LocalStack Lambda integration, did you observe any difference between AWS and LocalStack?

What I can say is that when we purge the queue "manually" in our tests, as we do now, subsequent messages do get delivered. Long-term we'll probably start using LocalStack SQS + Lambda, but I didn't dig into this yet - for the time being our setup is working fine and as expected.

I'd refer to @jamescarter-le, as he appears to have a different experience, but probably also a different setup :)


I'm also experiencing the same issue. When using LocalStack SQS fifo queue. I am not able to receive messages with the same `MessageGroupId` after deleting the first one (or when processing in batch after deleting the ones in first batch)

I tested it with `aws-sdk-go-v2` and localstack version `3.1.1.dev`. 

I did not experience this issue when running the same code against AWS SQS fifo.
We're also affected by this.

Downgraded to `3.0.2` until this is fixed, but that means we have to live with https://github.com/localstack/localstack/issues/9832 which is what prompted us to pick up `3.1.0` in the first place.

Resolving patch:
diff --git a/localstack/services/sqs/models.py b/localstack/services/sqs/models.py
--- a/localstack/services/sqs/models.py
+++ b/localstack/services/sqs/models.py
@@ -922,11 +922,7 @@ def get_message_group(self, message_group_id: str) -> MessageGroup:
         """
         with self.mutex:
             if message_group_id not in self.message_groups:
-                # a newly created message group is added to the queue immediately
-                message_group = self.message_groups[message_group_id] = MessageGroup(
-                    message_group_id
-                )
-                self.message_group_queue.put_nowait(message_group)
+                self.message_groups[message_group_id] = MessageGroup(message_group_id)
 
             return self.message_groups.get(message_group_id)
 
@@ -1018,15 +1014,20 @@ def _put_message(self, message: SqsMessage):
         message_group = self.get_message_group(message.message_group_id)
 
         with self.mutex:
+            previously_empty = message_group.empty()
             # put the message into the group
             message_group.push(message)
 
-            # if an older message becomes visible again in the queue, that message's group becomes visible also.
-            if message.receive_count < 1:
+            # new messages should not make groups visible that are currently inflight
+            if message.receive_count < 1 and message_group in self.inflight_groups:
                 return
+            # if an older message becomes visible again in the queue, that message's group becomes visible also.
             if message_group in self.inflight_groups:
                 self.inflight_groups.remove(message_group)
                 self.message_group_queue.put_nowait(message_group)
+            # if the group was previously empty, it was not yet added back to the queue
+            elif previously_empty:
+                self.message_group_queue.put_nowait(message_group)
 
     def remove_expired_messages(self):
         with self.mutex:
@@ -1080,10 +1081,9 @@ def receive(
             except Empty:
                 break
 
-            self.inflight_groups.add(group)
-
             if group.empty():
-                # this can be the case if all messages in the group are still invisible
+                # this can be the case if all messages in the group are still invisible or
+                # if all messages of a group have been processed.
                 # TODO: it should be blocking until at least one message is in the queue, but we don't
                 #  want to block the group
                 # TODO: check behavior in case it happens if all messages were removed from a group due to message
@@ -1093,6 +1093,8 @@ def receive(
                     timeout = 0
                 continue
 
+            self.inflight_groups.add(group)
+
             received_groups.add(group)
 
             block = False
@@ -1175,7 +1177,8 @@ def _on_remove_message(self, message: SqsMessage):
                         return
 
                 self.inflight_groups.remove(message_group)
-                self.message_group_queue.put_nowait(message_group)
+                if not message_group.empty():
+                    self.message_group_queue.put_nowait(message_group)
 
     def _assert_queue_name(self, name):
         if not name.endswith(".fifo"):


Test patch:
diff --git a/tests/aws/services/sqs/test_sqs.py b/tests/aws/services/sqs/test_sqs.py
--- a/tests/aws/services/sqs/test_sqs.py
+++ b/tests/aws/services/sqs/test_sqs.py
@@ -1154,6 +1154,45 @@ def test_message_retention_with_inflight(self, sqs_create_queue, aws_client, mon
         # try to delete the expired message
         aws_client.sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
 
+    @markers.aws.validated
+    def test_fifo_empty_message_groups_added_back_to_queue(
+        self, sqs_create_queue, aws_sqs_client, snapshot
+    ):
+        # https://github.com/localstack/localstack/issues/10107
+        queue_name = f"queue-{short_uid()}.fifo"
+        queue_url = sqs_create_queue(
+            QueueName=queue_name,
+            Attributes={
+                "FifoQueue": "True",
+            },
+        )
+        aws_sqs_client.send_message(
+            QueueUrl=queue_url,
+            MessageDeduplicationId="1",
+            MessageGroupId="g1",
+            MessageBody="Message 1",
+        )
+        resp = aws_sqs_client.receive_message(QueueUrl=queue_url)
+        snapshot.match("inital-fifo-receive", resp)
+
+        aws_sqs_client.delete_message(
+            QueueUrl=queue_url, ReceiptHandle=resp["Messages"][0]["ReceiptHandle"]
+        )
+
+        # call receive on the now empty message group
+        resp = aws_sqs_client.receive_message(QueueUrl=queue_url)
+        snapshot.match("empty-fifo-receive", resp)
+
+        # ensure FIFO queue stays functional
+        aws_sqs_client.send_message(
+            QueueUrl=queue_url,
+            MessageDeduplicationId="2",
+            MessageGroupId="g1",
+            MessageBody="Message 2",
+        )
+        resp = aws_sqs_client.receive_message(QueueUrl=queue_url)
+        snapshot.match("final-fifo-receive", resp)
+
     @markers.aws.needs_fixing
     @pytest.mark.skip("Needs AWS fixing and is now failing against LocalStack")
     def test_delete_message_batch_from_lambda(
diff --git a/tests/aws/services/sqs/test_sqs.snapshot.json b/tests/aws/services/sqs/test_sqs.snapshot.json
--- a/tests/aws/services/sqs/test_sqs.snapshot.json
+++ b/tests/aws/services/sqs/test_sqs.snapshot.json
@@ -2615,5 +2615,83 @@
         }
       }
     }
+  },
+  "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_empty_message_groups_added_back_to_queue[sqs]": {
+    "recorded-date": "12-02-2024, 16:08:11",
+    "recorded-content": {
+      "inital-fifo-receive": {
+        "Messages": [
+          {
+            "Body": "Message 1",
+            "MD5OfBody": "68390233272823b7adf13a1db79b2cd7",
+            "MessageId": "<uuid:1>",
+            "ReceiptHandle": "<receipt-handle:1>"
+          }
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      },
+      "empty-fifo-receive": {
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      },
+      "final-fifo-receive": {
+        "Messages": [
+          {
+            "Body": "Message 2",
+            "MD5OfBody": "88ef8f31ed540f1c4c03d5fdb06a7935",
+            "MessageId": "<uuid:2>",
+            "ReceiptHandle": "<receipt-handle:2>"
+          }
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      }
+    }
+  },
+  "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_empty_message_groups_added_back_to_queue[sqs_json]": {
+    "recorded-date": "12-02-2024, 16:08:14",
+    "recorded-content": {
+      "inital-fifo-receive": {
+        "Messages": [
+          {
+            "Body": "Message 1",
+            "MD5OfBody": "68390233272823b7adf13a1db79b2cd7",
+            "MessageId": "<uuid:1>",
+            "ReceiptHandle": "<receipt-handle:1>"
+          }
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      },
+      "empty-fifo-receive": {
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      },
+      "final-fifo-receive": {
+        "Messages": [
+          {
+            "Body": "Message 2",
+            "MD5OfBody": "88ef8f31ed540f1c4c03d5fdb06a7935",
+            "MessageId": "<uuid:2>",
+            "ReceiptHandle": "<receipt-handle:2>"
+          }
+        ],
+        "ResponseMetadata": {
+          "HTTPHeaders": {},
+          "HTTPStatusCode": 200
+        }
+      }
+    }
   }
 }
diff --git a/tests/aws/services/sqs/test_sqs.validation.json b/tests/aws/services/sqs/test_sqs.validation.json
--- a/tests/aws/services/sqs/test_sqs.validation.json
+++ b/tests/aws/services/sqs/test_sqs.validation.json
@@ -41,6 +41,12 @@
   "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_delete_message_with_expired_receipt_handle": {
     "last_validated_date": "2023-11-14T10:58:50+00:00"
   },
+  "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_empty_message_groups_added_back_to_queue[sqs]": {
+    "last_validated_date": "2024-02-12T16:08:11+00:00"
+  },
+  "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_empty_message_groups_added_back_to_queue[sqs_json]": {
+    "last_validated_date": "2024-02-12T16:08:14+00:00"
+  },
   "tests/aws/services/sqs/test_sqs.py::TestSqsProvider::test_fifo_message_attributes": {
     "last_validated_date": "2023-11-14T10:58:35+00:00"
   },