Repository: getmoto/moto

Issue:
SQS Response Protocol Migrated from XML to JSON
Moto (installed via Docker, pip)
- Python version: 3.12.0
- Moto Version: 4.2.9

AWS Ruby SDK
- aws-sdk-core: 3.187.1
- aws-sdk-sqs: 1.67.0

According to AWS documentation, the most recent version of their SDKs utilize JSON instead of XML for SQS responses. It appears this is a backward incompatible change, so migrating to the most recent version of the various AWS SDKs will cause cause issues with Moto.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-json-faqs.html

Tests are failing in the [ruby-shoryuken/shoryuken](https://github.com/ruby-shoryuken/shoryuken/pull/753) repository due to the [XML responses](https://github.com/getmoto/moto/blob/b3c3883a78f7dc096984a06e06112c31e96c2e06/moto/sqs/responses.py#L746) in Moto


[Action run in](https://github.com/ruby-shoryuken/shoryuken/actions/runs/6946610660/job/18904539378?pr=753) ruby-shoryuken/shoryuken: 
```
JSON::ParserError:
          #   859: unexpected token at '<CreateQueueResponse><CreateQueueResult><QueueUrl>http://localhost:5000/123456789012/shoryuken-travis-StandardWorker-455c8b98-cade-4220-a7cd-8b9e5bdfed27</QueueUrl></CreateQueueResult><ResponseMetadata><RequestId>F4FSDWJ3GAHTAWXHEQ11QOW56ZOVVDXRLHTUKZLB10T1XSI17DSA</RequestId></ResponseMetadata></CreateQueueResponse>'
          #   ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/json/json_engine.rb:8:in `load'

     2.2) Failure/Error:
            queue_url = Shoryuken::Client.sqs.get_queue_url(
              queue_name: StandardWorker.get_shoryuken_options['queue']
            ).queue_url

          Aws::SQS::Errors::NotFound:
            Aws::SQS::Errors::NotFound
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/request.rb:72:in `send_request'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-sqs-1.67.0/lib/aws-sdk-sqs/client.rb:1475:in `get_queue_url'
          # ./spec/integration/launcher_spec.rb:56:in `block (3 levels) in <top (required)>'
```

Dockerfile for Moto
```
FROM python:3.12

RUN pip install 'moto[server,sqs]'
CMD moto_server sqs -H 0.0.0.0

HEALTHCHECK --interval=5s --timeout=3s \
  CMD curl --fail http://localhost:5000/?Action=ListQueues || exit 1

EXPOSE 5000
```


Discussion:
Interesting - that should have been fixed by https://github.com/getmoto/moto/pull/6331 already. Maybe the Ruby SDK uses a different header than the other SDK's to flag that we should return JSON. Will look into it later.
I can't reproduce this with the latest version of Moto.

Looking at the repository that you linked, you are using a custom Docker image: 

https://github.com/ruby-shoryuken/shoryuken/blob/d65636ebacd6a33b68cdc7f539aa9ce6a50eef9f/.github/workflows/specs.yml#L17

And that Docker image was updated 3 years ago - which makes me think that you're actually running against an old(er) version of Moto. (https://quay.io/repository/cjlarose/moto-sqs-server)
Hey @bblommers I was able to replicate this using the Dockerfile I provided in the initial post, which uses the most up to date version of Python and Moto

I've created a very basic app that uses the most up to date versions of Moto and the AWS Ruby SDK. Interestingly, the `create_queue` function runs just fine, but `delete_queue` fails due to the JSON/XML mismatch.

https://github.com/mkeemon/moto-aws-ruby-debug

Edit: To validate this further, modifying the Gemfile to downgrade  `aws-sdk-sqs` to 1.65.0 and re-running the test works.

```rb
gem 'aws-sdk-sqs', '1.65.0'
```

## aws-sdk-sqs 1.67.0
<img width="1055" alt="Screenshot 2023-11-22 at 5 01 22 PM" src="https://github.com/getmoto/moto/assets/863187/fb8736fd-b848-45b0-80c0-a37f9acdb82c">

## aws-sdk-sqs 1.65.0
<img width="858" alt="Screenshot 2023-11-22 at 5 00 37 PM" src="https://github.com/getmoto/moto/assets/863187/64f32987-b73b-47ca-accb-1ce0f453fadb">

Resolving patch:
diff --git a/moto/sqs/responses.py b/moto/sqs/responses.py
--- a/moto/sqs/responses.py
+++ b/moto/sqs/responses.py
@@ -289,7 +289,13 @@ def set_queue_attributes(self) -> str:
         queue_name = self._get_queue_name()
         self.sqs_backend.set_queue_attributes(queue_name, attribute)
 
-        return SET_QUEUE_ATTRIBUTE_RESPONSE
+        return self._empty_response(SET_QUEUE_ATTRIBUTE_RESPONSE)
+
+    def _empty_response(self, xml_response: str) -> str:
+        # 'Empty' XML response always needs some fields
+        if self.is_json():
+            return "{}"
+        return xml_response
 
     @jsonify_error
     def delete_queue(self) -> str:
@@ -298,8 +304,7 @@ def delete_queue(self) -> str:
 
         self.sqs_backend.delete_queue(queue_name)
 
-        template = self.response_template(DELETE_QUEUE_RESPONSE)
-        return template.render()
+        return self._empty_response(DELETE_QUEUE_RESPONSE)
 
     @jsonify_error
     def send_message(self) -> Union[str, TYPE_RESPONSE]:
@@ -506,8 +511,7 @@ def delete_message_batch(self) -> str:
     def purge_queue(self) -> str:
         queue_name = self._get_queue_name()
         self.sqs_backend.purge_queue(queue_name)
-        template = self.response_template(PURGE_QUEUE_RESPONSE)
-        return template.render()
+        return self._empty_response(PURGE_QUEUE_RESPONSE)
 
     @jsonify_error
     def receive_message(self) -> Union[str, TYPE_RESPONSE]:
@@ -684,8 +688,7 @@ def add_permission(self) -> str:
 
         self.sqs_backend.add_permission(queue_name, actions, account_ids, label)
 
-        template = self.response_template(ADD_PERMISSION_RESPONSE)
-        return template.render()
+        return self._empty_response(ADD_PERMISSION_RESPONSE)
 
     @jsonify_error
     def remove_permission(self) -> str:
@@ -694,8 +697,7 @@ def remove_permission(self) -> str:
 
         self.sqs_backend.remove_permission(queue_name, label)
 
-        template = self.response_template(REMOVE_PERMISSION_RESPONSE)
-        return template.render()
+        return self._empty_response(REMOVE_PERMISSION_RESPONSE)
 
     @jsonify_error
     def tag_queue(self) -> str:


Test patch:
diff --git a/.github/workflows/tests_sdk_dotnet.yml b/.github/workflows/tests_sdk_dotnet.yml
--- a/.github/workflows/tests_sdk_dotnet.yml
+++ b/.github/workflows/tests_sdk_dotnet.yml
@@ -17,7 +17,7 @@ jobs:
       run: |
         pip install build
         python -m build
-        docker run --rm -t --name motoserver -e TEST_SERVER_MODE=true -e AWS_SECRET_ACCESS_KEY=server_secret -e AWS_ACCESS_KEY_ID=server_key -v `pwd`:/moto -p 5000:5000 -v /var/run/docker.sock:/var/run/docker.sock python:3.7-buster /moto/scripts/ci_moto_server.sh &
+        docker run --rm -t --name motoserver -e TEST_SERVER_MODE=true -e AWS_SECRET_ACCESS_KEY=server_secret -e AWS_ACCESS_KEY_ID=server_key -v `pwd`:/moto -p 5000:5000 -v /var/run/docker.sock:/var/run/docker.sock python:3.10-slim /moto/scripts/ci_moto_server.sh &
         python scripts/ci_wait_for_server.py
     - uses: actions/setup-dotnet@v3
     - uses: actions/cache@v3
diff --git a/.github/workflows/tests_sdk_java.yml b/.github/workflows/tests_sdk_java.yml
--- a/.github/workflows/tests_sdk_java.yml
+++ b/.github/workflows/tests_sdk_java.yml
@@ -17,7 +17,7 @@ jobs:
       run: |
         pip install build
         python -m build
-        docker run --rm -t --name motoserver -e TEST_SERVER_MODE=true -e AWS_SECRET_ACCESS_KEY=server_secret -e AWS_ACCESS_KEY_ID=server_key -v `pwd`:/moto -p 5000:5000 -v /var/run/docker.sock:/var/run/docker.sock python:3.7-buster /moto/scripts/ci_moto_server.sh &
+        docker run --rm -t --name motoserver -e TEST_SERVER_MODE=true -e AWS_SECRET_ACCESS_KEY=server_secret -e AWS_ACCESS_KEY_ID=server_key -v `pwd`:/moto -p 5000:5000 -v /var/run/docker.sock:/var/run/docker.sock python:3.10-slim /moto/scripts/ci_moto_server.sh &
         python scripts/ci_wait_for_server.py
     - uses: actions/setup-java@v3
       with:
diff --git a/.github/workflows/tests_sdk_ruby.yml b/.github/workflows/tests_sdk_ruby.yml
new file mode 100644
--- /dev/null
+++ b/.github/workflows/tests_sdk_ruby.yml
@@ -0,0 +1,36 @@
+name: Ruby SDK test
+on: [workflow_call]
+
+jobs:
+  test:
+
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        ruby-version: ['3.2']
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Ruby ${{ matrix.ruby-version }}
+        uses: ruby/setup-ruby@ec02537da5712d66d4d50a0f33b7eb52773b5ed1
+        with:
+          ruby-version: ${{ matrix.ruby-version }}
+      - name: Set up Python 3.8
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.8"
+      - name: Start MotoServer
+        run: |
+          pip install build
+          python -m build
+          docker run --rm -t --name motoserver -e TEST_SERVER_MODE=true -e AWS_SECRET_ACCESS_KEY=server_secret -e AWS_ACCESS_KEY_ID=server_key -v `pwd`:/moto -p 5000:5000 -v /var/run/docker.sock:/var/run/docker.sock python:3.10-slim /moto/scripts/ci_moto_server.sh &
+          python scripts/ci_wait_for_server.py
+      - name: Install dependencies
+        run: cd other_langs/tests_ruby && bundle install
+      - name: Run tests
+        env:
+          AWS_ENDPOINT_URL: "http://localhost:5000"
+        run: |
+          mkdir ~/.aws && touch ~/.aws/credentials && echo -e "[default]\naws_access_key_id = test\naws_secret_access_key = test" > ~/.aws/credentials
+          cd other_langs/tests_ruby && ruby test/*
diff --git a/other_langs/tests_ruby/Gemfile b/other_langs/tests_ruby/Gemfile
new file mode 100644
--- /dev/null
+++ b/other_langs/tests_ruby/Gemfile
@@ -0,0 +1,4 @@
+source "https://rubygems.org"
+
+gem 'aws-sdk-sqs', '1.65.0'
+gem "minitest"
\ No newline at end of file
diff --git a/other_langs/tests_ruby/test/aws_sqs_test.rb b/other_langs/tests_ruby/test/aws_sqs_test.rb
new file mode 100644
--- /dev/null
+++ b/other_langs/tests_ruby/test/aws_sqs_test.rb
@@ -0,0 +1,40 @@
+require 'aws-sdk-sqs'
+require 'minitest/autorun'
+
+class AwsSqsTest < Minitest::Test
+  def test_all_sqs_actions
+    region = 'us-east-1'
+
+    sqs_client = Aws::SQS::Client.new(region: region)
+
+    sqs_client.list_queues
+
+    sqs_client.create_queue(queue_name: "q1")
+
+    sqs_client.list_queues
+
+    queue_url = sqs_client.get_queue_url(queue_name: "q1").queue_url
+
+    sqs_client.add_permission({queue_url: queue_url, label: "String", aws_account_ids: ["String"], actions: ["GetQueueUrl"]})
+
+    sqs_client.send_message(
+      queue_url: queue_url,
+      message_body: "message_body"
+    )
+
+    sqs_client.remove_permission({queue_url: queue_url, label: "String"})
+
+    sqs_client.set_queue_attributes({queue_url: queue_url, attributes: {"All" => "DelaySeconds"}})
+
+    sqs_client.receive_message(
+      queue_url: queue_url,
+      max_number_of_messages: 10,
+      attribute_names: ['All'],
+      message_attribute_names: ['All']
+    ).messages
+
+    sqs_client.purge_queue({queue_url: queue_url})
+
+    sqs_client.delete_queue(queue_url: queue_url)
+  end
+end
\ No newline at end of file