Repository: localstack/localstack

Issue:
bug: &quot; when sending replaced by quotes when receiving
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Sending a message with body:
`"&quot;&quot;"`
is received on the consumer side with body:
`""""`

### Expected Behavior

Sending a message with body:
"&quot;&quot;"
is received on the consumer side with body:
"&quot;&quot;"

### How are you starting LocalStack?

Custom (please describe below)

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

Running localstack in docker desktop on Mac using helm chart

https://localstack.github.io/helm-charts
version 0.3.7
Image version: 0.14.3

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)



### Environment

```markdown
- OS: macOS Monterey
- LocalStack: 0.14.3
```


### Anything else?

_No response_


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).
Hi @yatanasov-hs, which service is this about? Could you share detailed steps to reproduce this issue?
Ops, so sorry, this is SQS running in localstaack:
Reproduction:
Create a queue with arbitrary name

Send a message:
`aws --endpoint-url http://localhost:31000 sqs send-message --queue-url <queue_url_name> --message-body '&quot;'`

Receive a message:
`aws --endpoint-url http://localhost:31000 sqs receive-message --queue-url <queue_url_name>`

Result:
`
{
    "Messages": [
        {
            "MessageId": "675ef0fd-dbea-431d-8b06-accbbc8f44b6",
            "ReceiptHandle": "ZmJhMWI3YTctZDFkNi00ODQ4LTkxMjItOGJjYjA0MmU0NDBjIGFybjphd3M6c3FzOnVzLWVhc3QtMjowMDAwMDAwMDAwMDA6aHMtbWFpbi1kZXYtcnB0cyA2NzVlZjBmZC1kYmVhLTQzMWQtOGIwNi1hY2NiYmM4ZjQ0YjYgMTY4MDA5OTM1Ni45MDA2NDE0",
            "MD5OfBody": "eb6439de53405a48b124e7cf89ba71d3",
            "Body": "\""
        }
    ]
}
`

Note Body gets changed
Same change occurs in message attributes
Thanks for reporting @yatanasov-hs, that's a very interesting find and may actually have been the root cause for https://github.com/localstack/localstack/issues/7897
We'll investigate!
I'm having the same issue with the latest version of localstack.

Another example below

Send a message:
`aws --endpoint-url=http://localhost:4566 sqs send-message --queue-url <queue_url> --message-body '"\\""'`

Receive message:
`aws --endpoint-url=http://localhost:4566 sqs receive-message --queue-url <queue_url>`

Result:
```
{
    "Messages": [
        {
            "MessageId": "92b607f8-bf72-4897-ac27-56a13b39977c",
            "ReceiptHandle": "ZDJjN2UwYzgtMzg0Zi00MTQ1LTgzNjEtZTc5OGE2NWVjNWIyIGFybjphd3M6c3FzOmFwLXNvdXRoZWFzdC0yOjAwMDAwMDAwMDAwMDpkYXRhYmFzZS1wcm9jZXNzb3IgOTJiNjA3ZjgtYmY3Mi00ODk3LWFjMjctNTZhMTNiMzk5NzdjIDE2ODEyMDAxMTUuNzUwNjg2Ng==",
            "MD5OfBody": "643953c85fd3510e719e12a3fa85d13b",
            "Body": "\"\\\\\"\""
        }
    ]
}
```
hey @jaiew , that seems to be the same behavior as in AWS though.

i tried your example against my aws account and got this:

```
aws sqs send-message --queue-url https://queue.amazonaws.com/xxxxxxxxxxxx/test-queue-02 --message-body '"\\""'
```

```
 % aws sqs receive-message --queue-url https://queue.amazonaws.com/xxxxxxxxxxxx/test-queue-02        
{
    "Messages": [
        {
            "MessageId": "e6feb982-8bc6-479c-b83d-d339d1xxxxxx",
            "ReceiptHandle": "AQEBwBNKyqZgdC3iDWj3TAubxsvmiPPe8XrbE7m8DZ95AeWolLHdlUP/b6FYmCdh9wKgb17qYK8VRoG/ObmJHATJJA6gRBrdPRr+Lk5j1DhsXka93g/C/Psf/jpaxAeOddKy3Ea5FRezC1YeNmQnlYkFEUwOJqyF0ABPYnqoRLRVxPAnH935BPdMncb4gdxmVmzMHvbKngMBTQLJfrsuWkx4ESwYk1iNqqPPzqjWKF+Dim0V3nPZ4b05XN/amLh/bNAj7PfqznAelDf5bQ2I0c1b45DGQxxxxxxxx/IxuThnueLPLf+4Kiw2s5Lg9xb5IAD9Ku+vWo+XONuoovRK/RyeNqazmZdcFI1qjTy3YR+Lggf3BtocIfHxvZvIWlM4YFy5aD902V371ikIyOsasxxxxx==",
            "MD5OfBody": "643953c85fd3510e719e12a3fa85d13b",
            "Body": "\"\\\\\"\""
        }
    ]
}
```

what are you getting back from AWS?
The original issue by @yatanasov-hs however I was able to reproduce.

Here's the XML I get from AWS:

```xml
<?xml version="1.0"?>
<ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
    <ReceiveMessageResult>
        <Message>
            <MessageId>4b6e1539-668c-4570-8c6f-63c6a5742811</MessageId>
            <ReceiptHandle>(removed)</ReceiptHandle>
            <MD5OfBody>eb6439de53405a48b124e7cf89ba71d3</MD5OfBody>
            <Body>&amp;quot;</Body>
        </Message>
    </ReceiveMessageResult>
    <ResponseMetadata>
        <RequestId>a0726f7d-a9b6-5cb5-ae5c-33c51576ddee</RequestId>
    </ResponseMetadata>
</ReceiveMessageResponse>
```

and the one i get from localstack:
```xml
<?xml version=\'1.0\' encoding=\'utf-8\'?>\n
<ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
    <ReceiveMessageResult>
        <Message>
            <MessageId>1b1dafd8-2ef9-4b35-9d33-4cbfcf950649</MessageId>
            <MD5OfBody>eb6439de53405a48b124e7cf89ba71d3</MD5OfBody>
            <Body>&quot;</Body>
            <ReceiptHandle>(removed)</ReceiptHandle>
        </Message>
    </ReceiveMessageResult>
    <ResponseMetadata>
        <RequestId>4JX5RCIBEE73CHQVAQP2MVUYWQWRBYSZU71VGXS9LS9IK834DV94</RequestId>
    </ResponseMetadata>
</ReceiveMessageResponse>
```

note that the md5 sum of the body is correct, so it's very likely an issue with the serializer. we have addressed an issue with quoting in SQS previously with a special serializer: #5298 which probably needs amendment for other HTML characters
in fact i think this line is responsible for this: https://github.com/localstack/localstack/blob/8f196a0fd079ca5e3523dcfb518bf6cbb0600cba/localstack/aws/protocol/serializer.py#L1530

@alexrashed any thoughts on how to deal with this?
This was implemented with https://github.com/localstack/localstack/pull/5298.
The `SqsResponseSerializer` docstring explains why this is necessary:
https://github.com/localstack/localstack/blob/2c3f10855180415bd55efe20a68451c2f9ebff34/localstack/aws/protocol/serializer.py#L1503-L1512

TL/DR: Unfortunately, the limitation comes from the used XML serializer. If we want to circumvent this, we need to find another XML serialization library which can handle these special characters (maybe the landscape has changed in the meantime), or write our own.
Just an idea can body and headers be base64 encoded/decoded when going in/out sqs queue? This way you do not need to escape anything?

Resolving patch:
diff --git a/localstack/aws/protocol/serializer.py b/localstack/aws/protocol/serializer.py
--- a/localstack/aws/protocol/serializer.py
+++ b/localstack/aws/protocol/serializer.py
@@ -1513,23 +1513,28 @@ class SqsResponseSerializer(QueryResponseSerializer):
     """
 
     def _default_serialize(self, xmlnode: ETree.Element, params: str, _, name: str, __) -> None:
-        """Ensures that XML text nodes use HTML entities instead of " or \r"""
+        """
+        Ensures that we "mark" characters in the node's text which need to be specifically encoded.
+        This is necessary to easily identify these specific characters later, after the standard XML serialization is
+        done, while not replacing any other occurrences of these characters which might appear in the serialized string.
+        """
         node = ETree.SubElement(xmlnode, name)
-        node.text = str(params).replace('"', "&quot;").replace("\r", "&#xD;")
+        node.text = (
+            str(params)
+            .replace('"', '__marker__"__marker__')
+            .replace("\r", "__marker__\r__marker__")
+        )
 
     def _node_to_string(self, root: Optional[ETree.ElementTree], mime_type: str) -> Optional[str]:
-        """
-        Replaces the double-escaped HTML entities with their correct HTML entity (basically reverts the escaping in
-        the serialization of the used XML framework).
-        """
+        """Replaces the previously "marked" characters with their encoded value."""
         generated_string = super()._node_to_string(root, mime_type)
         return (
             to_bytes(
                 to_str(generated_string)
                 # Undo the second escaping of the &
-                .replace("&amp;quot;", "&quot;")
+                .replace('__marker__"__marker__', "&quot;")
                 # Undo the second escaping of the carriage return (\r)
-                .replace("&amp;#xD;", "&#xD;")
+                .replace("__marker__\r__marker__", "&#xD;")
             )
             if generated_string is not None
             else None


Test patch:
diff --git a/tests/integration/test_sqs.py b/tests/integration/test_sqs.py
--- a/tests/integration/test_sqs.py
+++ b/tests/integration/test_sqs.py
@@ -292,6 +292,14 @@ def test_send_receive_message_multiple_queues(self, sqs_create_queue, aws_client
         assert len(result["Messages"]) == 1
         assert result["Messages"][0]["Body"] == "message"
 
+    @pytest.mark.aws_validated
+    def test_send_receive_message_encoded_content(self, sqs_create_queue, aws_client):
+        queue = sqs_create_queue()
+        aws_client.sqs.send_message(QueueUrl=queue, MessageBody='"&quot;&quot;\r')
+        result = aws_client.sqs.receive_message(QueueUrl=queue)
+        assert len(result["Messages"]) == 1
+        assert result["Messages"][0]["Body"] == '"&quot;&quot;\r'
+
     @pytest.mark.aws_validated
     def test_send_message_batch(self, sqs_queue, aws_client):
         aws_client.sqs.send_message_batch(