Repository: localstack/localstack

Issue:
bug: OpenSearch response with compressed content
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

While running `curl` to reach OpenSearch health endpoint, I receive a binary response instead of plain JSON.

```shell
$ curl -vvv my-domain.us-east-1.opensearch.localhost.localstack.cloud:443/_cluster/health                                                                                                                                                                                                                               
*   Trying 127.0.0.1:443...
* Connected to my-domain.us-east-1.opensearch.localhost.localstack.cloud (127.0.0.1) port 443 (#0)
> GET /_cluster/health HTTP/1.1
> Host: my-domain.us-east-1.opensearch.localhost.localstack.cloud:443
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200
< content-type: application/json; charset=UTF-8
< content-encoding: gzip
< Connection: close
< date: Tue, 01 Aug 2023 08:46:35 GMT
< server: hypercorn-h11
< Transfer-Encoding: chunked
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Failed reading the chunked-encoded stream
* Closing connection 0
```

### Expected Behavior

I expect to receive a JSON response without the need to use `--compressed` option in the `curl` command.

 curl -vvv --compressed my-domain.us-east-1.opensearch.localhost.localstack.cloud:443/_cluster/health
*   Trying 127.0.0.1:443...
* Connected to my-domain.us-east-1.opensearch.localhost.localstack.cloud (127.0.0.1) port 443 (#0)
> GET /_cluster/health HTTP/1.1
> Host: my-domain.us-east-1.opensearch.localhost.localstack.cloud:443
> User-Agent: curl/7.81.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br, zstd
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< content-type: application/json; charset=UTF-8
< content-encoding: gzip
< Connection: close
< date: Tue, 01 Aug 2023 08:47:06 GMT
< server: hypercorn-h11
< Transfer-Encoding: chunked
<
* Closing connection 0
{"cluster_name":"opensearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)
```yaml
version: '3.8'
services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack-pro:latest
    ports:
      - "127.0.0.1:53:53"
      - "127.0.0.1:53:53/udp"
      - "127.0.0.1:443:443"
      - "127.0.0.1:4510-4559:4510-4559"
      - "127.0.0.1:4566:4566"
    environment:
      - DEBUG=1
      - LS_LOG=trace
      - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-} # only required for Pro
      - DOCKER_HOST=unix:///var/run/docker.sock
      - OPENSEARCH_ENDPOINT_STRATEGY=domain
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./volume:/var/lib/localstack" # mount Docker volume
    healthcheck:
      disable: true
```

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

```shell
awslocal opensearch create-domain --domain-name my-domain
curl -vvv my-domain.us-east-1.opensearch.localhost.localstack.cloud:443/_cluster/health          
```

### Environment

```markdown
- OS: Windows 10
- LocalStack: 2.2.1.dev20230731233910
```


### Anything else?

Following the guide from the documentation page: [OpenSearch](https://docs.localstack.cloud/user-guide/aws/opensearch/)


Discussion:
Just upgraded localstack from version 1.3.0 to 2.2.0 and started having this issue.

Resolving patch:
diff --git a/localstack/http/client.py b/localstack/http/client.py
--- a/localstack/http/client.py
+++ b/localstack/http/client.py
@@ -81,10 +81,6 @@ def request(self, request: Request, server: str | None = None) -> Response:
         Very naive implementation to make the given HTTP request using the requests library, i.e., process the request
         as a client.
 
-        TODO: over time, this should become more sophisticated, specifically the use of restore_payload should only be
-         used when necessary (when the underlying stream has been consumed), and by default the stream should be
-         streamed to the destination.
-
         :param request: the request to perform
         :param server: the URL to send the request to, which defaults to the host component of the original Request.
         :return: the response.
@@ -92,43 +88,17 @@ def request(self, request: Request, server: str | None = None) -> Response:
 
         url = self._get_destination_url(request, server)
 
-        response = self.session.request(
-            method=request.method,
-            # use raw base url to preserve path url encoding
-            url=url,
-            # request.args are only the url parameters
-            params=[(k, v) for k, v in request.args.items(multi=True)],
-            headers=dict(request.headers.items()),
-            data=restore_payload(request),
-        )
-
-        final_response = Response(
-            response=response.content,
-            status=response.status_code,
-            headers=Headers(dict(response.headers)),
-        )
-        if request.method == "HEAD":
-            # for HEAD  requests we have to keep the original content-length, but it will be re-calculated when creating
-            # the final_response object
-            final_response.content_length = response.headers.get("Content-Length", 0)
-        return final_response
-
-    def close(self):
-        self.session.close()
-
-
-class SimpleStreamingRequestsClient(SimpleRequestsClient):
-    def request(self, request: Request, server: str | None = None) -> Response:
-        """
-        Very naive implementation to make the given HTTP request using the requests library, i.e., process the request
-        as a client.
-
-        :param request: the request to perform
-        :param server: the URL to send the request to, which defaults to the host component of the original Request.
-        :return: the response.
-        """
+        headers = dict(request.headers.items())
 
-        url = self._get_destination_url(request, server)
+        # urllib3 (used by requests) will set an Accept-Encoding header ("gzip,deflate")
+        # - See urllib3.util.request.ACCEPT_ENCODING
+        # - This could be circumvented by setting Accept-Encoding explicitly to None
+        # BUT: Python's http client (used by urllib3) will always add an accept header if none is given
+        # - See https://github.com/psf/requests/issues/2234 and http.client.putrequest
+        # Explicitly set `Accept-Encoding: identity` here if no `Accept-Encoding` header is set in the originating
+        # request to avoid any unused manipulations by underlying libraries.
+        if not request.headers.get("accept-encoding"):
+            headers["accept-encoding"] = "identity"
 
         response = self.session.request(
             method=request.method,
@@ -136,7 +106,7 @@ def request(self, request: Request, server: str | None = None) -> Response:
             url=url,
             # request.args are only the url parameters
             params=[(k, v) for k, v in request.args.items(multi=True)],
-            headers=dict(request.headers.items()),
+            headers=headers,
             data=restore_payload(request),
             stream=True,
         )
@@ -164,6 +134,9 @@ def request(self, request: Request, server: str | None = None) -> Response:
 
         return final_response
 
+    def close(self):
+        self.session.close()
+
 
 def make_request(request: Request) -> Response:
     """
diff --git a/localstack/services/opensearch/cluster.py b/localstack/services/opensearch/cluster.py
--- a/localstack/services/opensearch/cluster.py
+++ b/localstack/services/opensearch/cluster.py
@@ -14,7 +14,7 @@
     EngineType,
     ValidationException,
 )
-from localstack.http.client import SimpleStreamingRequestsClient
+from localstack.http.client import SimpleRequestsClient
 from localstack.http.proxy import ProxyHandler
 from localstack.services.edge import ROUTER
 from localstack.services.infra import DEFAULT_BACKEND_HOST
@@ -229,7 +229,7 @@ def register_cluster(
     forward_url = config.OPENSEARCH_CUSTOM_BACKEND or forward_url
 
     # if the opensearch security plugin is enabled, only TLS connections are allowed, but the cert cannot be verified
-    client = SimpleStreamingRequestsClient()
+    client = SimpleRequestsClient()
     client.session.verify = False
     endpoint = ProxyHandler(forward_url, client)
 
@@ -416,7 +416,6 @@ def _base_settings(self, dirs) -> CommandSettings:
             "http.publish_port": self.port,
             "transport.port": "0",
             "network.host": self.host,
-            # TODO this breaks the cluster check?!
             "http.compression": "true",
             "path.data": f'"{dirs.data}"',
             "path.repo": f'"{dirs.backup}"',
diff --git a/localstack/services/s3/virtual_host.py b/localstack/services/s3/virtual_host.py
--- a/localstack/services/s3/virtual_host.py
+++ b/localstack/services/s3/virtual_host.py
@@ -5,7 +5,6 @@
 from localstack import config
 from localstack.constants import LOCALHOST_HOSTNAME
 from localstack.http import Request, Response
-from localstack.http.client import SimpleStreamingRequestsClient
 from localstack.http.proxy import Proxy
 from localstack.runtime import hooks
 from localstack.services.edge import ROUTER
@@ -62,7 +61,6 @@ def _create_proxy(self) -> Proxy:
             forward_base_url=config.get_edge_url(),
             # do not preserve the Host when forwarding (to avoid an endless loop)
             preserve_host=False,
-            client=SimpleStreamingRequestsClient(),
         )
 
     @staticmethod


Test patch:
diff --git a/tests/aws/services/opensearch/test_opensearch.py b/tests/aws/services/opensearch/test_opensearch.py
--- a/tests/aws/services/opensearch/test_opensearch.py
+++ b/tests/aws/services/opensearch/test_opensearch.py
@@ -3,6 +3,8 @@
 import logging
 import os
 import threading
+from http.client import HTTPConnection
+from urllib.parse import urlparse
 
 import botocore.exceptions
 import pytest
@@ -29,6 +31,7 @@
 from localstack.utils.common import call_safe, poll_condition, retry
 from localstack.utils.common import safe_requests as requests
 from localstack.utils.common import short_uid, start_worker_thread
+from localstack.utils.strings import to_str
 
 LOG = logging.getLogger(__name__)
 
@@ -482,12 +485,33 @@ def test_describe_domains(self, opensearch_domain, aws_client):
         assert response["DomainStatusList"][0]["DomainName"] == opensearch_domain
 
     @markers.aws.unknown
-    def test_gzip_responses(self, opensearch_endpoint):
-        gzip_response = requests.get(opensearch_endpoint, stream=True)
+    def test_gzip_responses(self, opensearch_endpoint, monkeypatch):
+        def send_plain_request(method, url):
+            """
+            It's basically impossible to send a request without an Accept-Encoding header with Python's standard
+            lib when using http.client.
+            This function sends a plain request by directly interacting with the HTTPConnection.
+            """
+            parsed = urlparse(url)
+            connection = HTTPConnection(host=parsed.hostname, port=parsed.port)
+            connection.putrequest(method, parsed.path, skip_accept_encoding=True)
+            connection.endheaders()
+            response = connection.getresponse()
+            connection.close()
+            return response
+
+        plain_response = send_plain_request("GET", opensearch_endpoint)
+        assert "cluster_name" in to_str(plain_response.read())
+
+        # ensure that requests with the "Accept-Encoding": "gzip" header receive gzip compressed responses
+        gzip_accept_headers = {"Accept-Encoding": "gzip"}
+        gzip_response = requests.get(opensearch_endpoint, headers=gzip_accept_headers, stream=True)
         # get the raw data, don't let requests decode the response
-        raw_data = b"".join(chunk for chunk in gzip_response.raw.stream(1024, decode_content=False))
+        raw_gzip_data = b"".join(
+            chunk for chunk in gzip_response.raw.stream(1024, decode_content=False)
+        )
         # force the gzip decoding here (which would raise an exception if it's not actually gzip)
-        assert gzip.decompress(raw_data)
+        assert gzip.decompress(raw_gzip_data)
 
     @markers.aws.unknown
     def test_domain_version(self, opensearch_domain, opensearch_create_domain, aws_client):
diff --git a/tests/unit/http_/test_client.py b/tests/unit/http_/test_client.py
--- a/tests/unit/http_/test_client.py
+++ b/tests/unit/http_/test_client.py
@@ -7,7 +7,7 @@
 from requests.exceptions import SSLError
 
 from localstack.http import Request
-from localstack.http.client import SimpleRequestsClient, SimpleStreamingRequestsClient
+from localstack.http.client import SimpleRequestsClient
 from localstack.utils.ssl import create_ssl_cert
 
 
@@ -38,16 +38,15 @@ def ssl_httpserver(make_ssl_httpserver):
     server.clear()
 
 
-@pytest.mark.parametrize("client_class", [SimpleRequestsClient, SimpleStreamingRequestsClient])
 @pytest.mark.parametrize("verify", [True, False])
 @pytest.mark.parametrize("cert_env", [None, "REQUESTS_CA_BUNDLE", "CURL_CA_BUNDLE"])
-def test_http_clients_respect_verify(client_class, verify, cert_env, ssl_httpserver, monkeypatch):
+def test_http_clients_respect_verify(verify, cert_env, ssl_httpserver, monkeypatch):
     # If we want to test that a certain environment variable, setting the CA bundle, is set, we
     # just set the same path as requests uses anyway (the issues is caused just by the variables being set).
     if cert_env:
         monkeypatch.setenv(cert_env, certifi.where())
 
-    client = client_class()
+    client = SimpleRequestsClient()
     client.session.verify = verify
 
     # Configure the SSL http server fixture
diff --git a/tests/unit/http_/test_proxy.py b/tests/unit/http_/test_proxy.py
--- a/tests/unit/http_/test_proxy.py
+++ b/tests/unit/http_/test_proxy.py
@@ -7,7 +7,7 @@
 from werkzeug import Request as WerkzeugRequest
 
 from localstack.http import Request, Response, Router
-from localstack.http.client import SimpleRequestsClient, SimpleStreamingRequestsClient
+from localstack.http.client import SimpleRequestsClient
 from localstack.http.dispatcher import handler_dispatcher
 from localstack.http.hypercorn import HypercornServer
 from localstack.http.proxy import Proxy, ProxyHandler, forward
@@ -213,19 +213,18 @@ def _handler(_: WerkzeugRequest):
 
         httpserver.expect_request("").respond_with_handler(_handler)
 
-        with SimpleStreamingRequestsClient() as client:
-            proxy = Proxy(httpserver.url_for("/").lstrip("/"), client)
+        proxy = Proxy(httpserver.url_for("/").lstrip("/"))
 
-            request = Request(path="/", method="GET", headers={"Host": "127.0.0.1:80"})
+        request = Request(path="/", method="GET", headers={"Host": "127.0.0.1:80"})
 
-            response = proxy.request(request)
+        response = proxy.request(request)
 
-            if chunked:
-                assert response.headers["Transfer-Encoding"] == "chunked"
-                assert "Content-Length" not in response.headers
-            else:
-                assert response.headers["Content-Length"] == str(len(body))
-                assert "Transfer-Encoding" not in response.headers
+        if chunked:
+            assert response.headers["Transfer-Encoding"] == "chunked"
+            assert "Content-Length" not in response.headers
+        else:
+            assert response.headers["Content-Length"] == str(len(body))
+            assert "Transfer-Encoding" not in response.headers
 
 
 @pytest.mark.parametrize("consume_data", [True, False])