Repository: localstack/localstack

Issue:
HTTP header casing when localstack proxies requests
In #6778 we found that AWS forwards HTTP headers to Lambdas from clients without modifying their casing. Since users are relying on this behavior, it effectively renders a web server that lowercases headers unusable as proxy for localstack. In general, I think there is a case to be made that web servers should not manipulate the casing of request headers, so that they can function as application-layer proxy to backends that make assumptions about header casing.

The issue of header casing for HTTP/1.1 in hypercorn was addressed with https://github.com/pgjones/hypercorn/issues/66, but later reverted with https://github.com/pgjones/hypercorn/issues/77. Apparently the relevant ASGI spec is being discussed here https://github.com/django/asgiref/issues/246 where I think we should make that case.

If this is a fundamental limitation of ASGI and we cannot get this into hypercorn, then we will be forced to either
- apologize to lambda users and tell them that they need to re-write their lambda code in a way that treats headers in a case-insensitive way
- fork hypercorn and maintain our own version
- use another webserver that does not manipulate HTTP headers, or at least allows access to the raw headers in some way


Discussion:
interested in getting your opinion @whummer (in particular the lambda user perspective, i.e., option 1), and @alexrashed
Currently, it feels like the discussion is going towards keeping the spec as it is right now (or just weakening the phrase a bit).
But, even if the spec stays the same we could maybe continue using hypercorn. The recent changes were addressing the way of accessing the headers as specified by ASGI. Maybe we can find a compromise to preserving the raw headers and to have a way of accessing them outside of ASGI. No-one prevents any framework from providing a additional features than specified in the spec. :)

If we can't stick with hypercorn (because we need the header casing being preserved and we can't find a way to access the raw headers), we could search for a suitable alternative (where the landscape is somewhat limited), or we could try to create a tightly-synced fork which simply adds this feature (which I would like to avoid if possible).
I've proposed a config option approach in hypercorn to allow for raw headers, which would let us continue using upstream hypercorn :crossed_fingers: https://github.com/pgjones/hypercorn/pull/117

Resolving patch:
diff --git a/localstack/aws/serving/hypercorn.py b/localstack/aws/serving/hypercorn.py
--- a/localstack/aws/serving/hypercorn.py
+++ b/localstack/aws/serving/hypercorn.py
@@ -29,6 +29,7 @@ def serve(
     :param kwargs: any oder parameters that can be passed to the hypercorn.Config object
     """
     config = Config()
+    config.h11_pass_raw_headers = True
     config.bind = f"{host}:{port}"
     config.use_reloader = use_reloader
 
diff --git a/localstack/http/hypercorn.py b/localstack/http/hypercorn.py
--- a/localstack/http/hypercorn.py
+++ b/localstack/http/hypercorn.py
@@ -93,6 +93,7 @@ def __init__(
         """
         # build server config
         config = Config()
+        config.h11_pass_raw_headers = True
         setup_hypercorn_logger(config)
 
         bind_address = ensure_list(bind_address)
diff --git a/localstack/utils/server/http2_server.py b/localstack/utils/server/http2_server.py
--- a/localstack/utils/server/http2_server.py
+++ b/localstack/utils/server/http2_server.py
@@ -242,6 +242,7 @@ def run_app_sync(*args, loop=None, shutdown_event=None):
             kwargs["keyfile"] = key_file_name
             config.keyfile = key_file_name
         setup_quart_logging()
+        config.h11_pass_raw_headers = True
         config.bind = [f"{bind_address}:{port}" for bind_address in bind_addresses]
         config.workers = len(bind_addresses)
         loop = loop or ensure_event_loop()


Test patch:
diff --git a/tests/unit/aws/test_gateway.py b/tests/unit/aws/test_gateway.py
--- a/tests/unit/aws/test_gateway.py
+++ b/tests/unit/aws/test_gateway.py
@@ -20,6 +20,7 @@ def serve_gateway_hypercorn():
 
     def _create(gateway: Gateway) -> HypercornServer:
         config = Config()
+        config.h11_pass_raw_headers = True
         config.bind = f"localhost:{net.get_free_tcp_port()}"
         loop = asyncio.new_event_loop()
         srv = HypercornServer(AsgiGateway(gateway, event_loop=loop), config, loop=loop)
diff --git a/tests/unit/http_/conftest.py b/tests/unit/http_/conftest.py
--- a/tests/unit/http_/conftest.py
+++ b/tests/unit/http_/conftest.py
@@ -26,6 +26,7 @@ def _create(
     ) -> HypercornServer:
         if not config:
             config = Config()
+            config.h11_pass_raw_headers = True
             config.bind = f"localhost:{net.get_free_tcp_port()}"
 
         srv = HypercornServer(app, config, loop=event_loop)