Repository: localstack/localstack

Issue:
bug: SQS performance regression with ASF
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Hi, I'm testing the image `localstack/localstack:0.14.5`, and setting `LEGACY_EDGE_PROXY=1` makes the following cleanup code much slower:

`for queue in client.list_queues()['QueueUrls']: client.purge_queue(QueueUrl=queue)`

`LEGACY_EDGE_PROXY=1`: `56.2 ms ± 2.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)`

`LEGACY_EDGE_PROXY=0`: `258 ms ± 17.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)`


### Expected Behavior

I expected performance to improve or not to regress.

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

# Benchmark setup:

Docker-compose file:
```
version: "3.9"
services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - DEFAULT_REGION=eu-west-1
      - SERVICES=s3,sns,sqs
      - LEGACY_EDGE_PROXY=1  # or 0
    ipc: host
```

```
# docker-compose rm -f localstack && docker-compose up localstack
```

```
import boto3

client = boto3.resource(
    'sqs',
    endpoint_url="http://localstack:4566/",
    region_name='eu-west-1',
    aws_secret_access_key="x",
    aws_access_key_id="x",
    use_ssl=False,
).meta.client

for i in range(10):
    client.create_queue(QueueName=str(i))

queues = client.list_queues()['QueueUrls']
assert len(queues) == 10
```

```
In [3]: %timeit for queue in client.list_queues()['QueueUrls']: client.purge_queue(QueueUrl=queue)
```

### Environment

```markdown
- OS: Ubuntu 22.04 LTS, Linux 5.15.0-40-generic #43-Ubuntu SMP Wed Jun 15 12:54:21 UTC 2022 x86_64 
- docker-compose 1.29.2
- Docker: 20.10.12, build 20.10.12-0ubuntu4
- LocalStack: 0.14.5
```


### Anything else?

Results don't improve over time, it's not a cold start issue.

https://github.com/localstack/localstack/issues/6398


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).
Thanks for the report! We can confirm your findings, and will work to improve performance over the coming weeks. It might take _some_ time to fix this completely.
I've been digging into this a bit more and I don't really have a good sense of what's going on yet. My current working assumption given the experiments i've done so far is that it is a concurrency issue rooted in the sync/async bridge.

I've done some experiments with different methods and also plain http requests, here are some of the findings. These are client-side latencies.
```
sqs list-queues legacy  		ms/op 29.055353829990054
sqs list-queues gateway 		ms/op 67.29019051999785

opensearch list-domains legacy		ms/op 13.264056890002394
opensearch list-domains gateway		ms/op 52.241944870002044

es list-domains legacy			ms/op 27.98722170999099
es list-domains gateway  		ms/op 102.1972499200092

GET /health legacy  			ms/op 12.459445409000182
GET /health gateway 			ms/op 10.224880224001026
```

Some observations (some of which are not reflected in the data, but gathered from individual experiments)
* running simple `GET` requests is actually faster with the new gateway.
* server-side latencies do not seem to be the issue. it has nothing to do with the handler chain per-se. my early concerns about parsing or logging taking a long time were invalidated here. i timed the requests comparing these two areas in the respective approaches, and typically they perform the same, or the gateway is even a bit faster (will collect some data to showcase that)
  * legacy:https://github.com/localstack/localstack/blob/b848ed372e7de1c83e029c479426a4f54e6a3e13/localstack/utils/server/http2_server.py#L182-L186
  * gateway: https://github.com/localstack/localstack/blob/b848ed372e7de1c83e029c479426a4f54e6a3e13/localstack/http/asgi.py#L351-L352
* running individual calls to SQS list-queues is _not_ slower. the slowdown happens when multiple requests are made in a row (indicating to me some sort of issue with queuing)
* `es` is MUCH slower than opensearch, which has to do with the fact that es calls opensearch using the boto client, so that involves another http call, obviously exacerbating the issue. that call suffers from the same problem as this client benchmark: calls take roughly 30 ms more.
* it doesn't seem to be related to SSL or the `DuplexSocket`, tried to enable/disable both, no change
i can definitely confirm now that it has something to do with the asgi/wsgi bridge. serving the gateway directly through the werkzeug server is much faster.

you can use this script to serve localstack through werkzeug:
```python
from localstack.aws.app import LocalstackAwsGateway
from localstack.aws.serving.werkzeug import serve
from localstack.logging.setup import setup_logging
from localstack.services.plugins import SERVICE_PLUGINS


def main():
    setup_logging()
    serve(LocalstackAwsGateway(SERVICE_PLUGINS), port=4567)


if __name__ == '__main__':
    main()
```

when running the sqs list-queues benchmark get the following results:
```
sqs list-queues legacy  ms/op 10.982355949963676
sqs list-queues gateway ms/op 6.484360479953466
```

it seems to be a combination of things though, since s3 does not seem to suffer from the same behavior. This benchmark script here yields similar result when using the wsgi/asgi bridge (both http and using boto).

```python
import timeit

import boto3
import requests

s3_old = boto3.client("s3", endpoint_url="http://localhost:4566")
s3_new = boto3.client("s3", endpoint_url="http://localhost:4567")

s3_old.create_bucket(Bucket="foo")
s3_new.create_bucket(Bucket="foo")

s3_old.put_object(Bucket="foo", Key="bar", Body="foobared")
s3_new.put_object(Bucket="foo", Key="bar", Body="foobared")


def send_s3_http_old():
    return requests.get("http://localhost:4566/foo/bar")


def send_s3_http_new():
    return requests.get("http://localhost:4567/foo/bar")


def send_s3_boto_old():
    return s3_old.get_object(Bucket="foo", Key="bar")


def send_s3_boto_new():
    return s3_new.get_object(Bucket="foo", Key="bar")


def main():
    print("s3 GetObject (http) legacy  ms/op", (timeit.timeit(send_s3_http_old, number=100) / 100) * 1000)
    print("s3 GetObject (http) gateway ms/op", (timeit.timeit(send_s3_http_new, number=100) / 100) * 1000)

    print("s3 GetObject (boto) legacy  ms/op", (timeit.timeit(send_s3_boto_old, number=100) / 100) * 1000)
    print("s3 GetObject (boto) gateway ms/op", (timeit.timeit(send_s3_boto_new, number=100) / 100) * 1000)


if __name__ == '__main__':
    main()
```
here's a demonstration that it has nothing to do with server-side latency. i wrapped both the asgi gateway and quart ASGI app (what's underlying the http2_server) with timer. here are the results. of server-side latency of the `sqs.ListQueues` command.

![image](https://user-images.githubusercontent.com/3996682/181918580-0044b920-e16f-47ea-b587-3d48b87a411a.png)


OK i further narrowed it down to this line here: https://github.com/localstack/localstack/blob/736e7ad490deda766b81b0ed893a0322bc9d3d88/localstack/http/asgi.py#L317

removing this leads to the following results for the list queues benchmark:
```
sqs list-queues legacy  ms/op 12.990882140002213
sqs list-queues gateway ms/op 12.340250319994084
```

when i profile the clients, i can see that the clients sending to the gateway spend 4 times longer in `recv_into` of `_socket.socket`.
this indicates to me that it has something to do with resource management, where for some reason streams/connections are not closed correctly.


my hunch that it has to do with connection management seems to have been correct:

TL/DR:

* what's happening? hypercorn/h11 connection keep-alive is slowing down boto clients (not sure why exactly yet)
* why was it working before? `http2_server.py` was forcing keep-alive to be disabled by responding with `Connection: close` headers

## connection re-use and keep-alive in the spec and localstack

[RFC 7230 6.3 Persistence](https://datatracker.ietf.org/doc/html/rfc7230#section-6.3) states that, unless the HTTP/1.1 client explicitly send the `Connection: close` header, then the server should use a persistent connection by default.

by forcing `Connection: close` here, we are not adhering to the spec. https://github.com/localstack/localstack/blob/736e7ad490deda766b81b0ed893a0322bc9d3d88/localstack/utils/server/http2_server.py#L225-L226

however this had a side effect in h11, which made clients close connections faster.

## h11 keep-alive

h11 (the protocol library underlying hypercorn) uses a state machine for connections that look like this:

<img src="https://h11.readthedocs.io/en/latest/_static/SERVER.svg">

the purple path `start_next_cycle()` was never used with `http2_server.py`. h11 considers keep-alive to be active according to the spec: if the client uses HTTP/1.1, and is not explicitly disabling it via a `Connection: close` header, then it will enable keep-alive: https://github.com/python-hyper/h11/blob/fb6c715184c6a1f29e299d2f63d722b2ea4309d5/h11/_connection.py#L72-L78

more details on re-using connections in h11 can be found in the docs: [Re-using a connection: keep-alive and pipelining](https://h11.readthedocs.io/en/latest/api.html#re-using-a-connection-keep-alive-and-pipelining)


## It's probably not our asgi/wsgi bridge

Boto clients running against a plain quart application from the same problem:

```python
from quart import Quart, request

app = Quart(__name__)

@app.route("/", methods=["POST", "GET"])
async def echo():
    data = await request.data
    await asyncio.sleep(0.005)
    return '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<ListQueuesResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ListQueuesResult /><ResponseMetadata><RequestId>Y05BFPBE0ZMIURXXPY35R57NELYBSPCNY6XBU0FR0A6ITR27LZUJ</RequestId></ResponseMetadata></ListQueuesResponse>'

app.run(port=4566)
```
running the benchmark against this shows the same or at least similar behavior.


## what to do?

well for now, the easiest fix would be to add a `ResponseHandler` to the gateway that sets the `Connection: close` header like the http2_server.py does. that's not great (breaking specs causes headaches), but since we don't seem to be having any problems doing it now, ..

figuring out what happens in boto would be interesting though to understand why connection re-use seems to be slower than dropping and re-establishing the connection after each request.


Resolving patch:
diff --git a/localstack/aws/app.py b/localstack/aws/app.py
--- a/localstack/aws/app.py
+++ b/localstack/aws/app.py
@@ -57,6 +57,7 @@ def __init__(self, service_manager: ServiceManager = None) -> None:
         self.response_handlers.extend(
             [
                 handlers.parse_service_response,
+                handlers.set_close_connection_header,
                 handlers.run_custom_response_handlers,
                 handlers.add_cors_response_headers,
                 handlers.log_response,
diff --git a/localstack/aws/handlers/__init__.py b/localstack/aws/handlers/__init__.py
--- a/localstack/aws/handlers/__init__.py
+++ b/localstack/aws/handlers/__init__.py
@@ -23,5 +23,6 @@
 # legacy compatibility handlers
 serve_edge_router_rules = legacy.EdgeRouterHandler()
 serve_default_listeners = legacy.DefaultListenerHandler()
+set_close_connection_header = legacy.set_close_connection_header
 pop_request_context = legacy.pop_request_context
 push_request_context = legacy.push_request_context
diff --git a/localstack/aws/handlers/legacy.py b/localstack/aws/handlers/legacy.py
--- a/localstack/aws/handlers/legacy.py
+++ b/localstack/aws/handlers/legacy.py
@@ -45,6 +45,18 @@ def pop_request_context(_chain: HandlerChain, _context: RequestContext, _respons
     request_context.THREAD_LOCAL.request_context = None
 
 
+def set_close_connection_header(_chain: HandlerChain, context: RequestContext, response: Response):
+    """This is a hack to work around performance issues with h11 and boto. See
+    https://github.com/localstack/localstack/issues/6557"""
+    if conn := context.request.headers.get("Connection"):
+        if conn.lower() == "keep-alive":
+            # don't set Connection: close header if keep-alive is explicitly asked for
+            return
+
+    if "Connection" not in response.headers:
+        response.headers["Connection"] = "close"
+
+
 class EdgeRouterHandler(RouterHandler):
     def __init__(self, respond_not_found=False) -> None:
         from localstack.services.edge import ROUTER


Test patch: