Repository: sanic-org/sanic

Issue:
Maximum request header size capped at 8k
**Describe the bug**
Sending a request with a large header to a Sanic server results in 413 (Payload Too Large) response. 

This happens regardless of the `request_max_size` setting.

If I undserstand the implementation correctly, the maximum HTTP header size is hardcoded to be 8k:

https://github.com/sanic-org/sanic/blob/main/sanic/http.py#L172

**Expected behavior**
I think maximum header size should be configurable and values > 8k should be allowed.

**Environment (please complete the following information):**
 - OS: CentOS 7
 - Version 21.3.4



Discussion:
The limit is in place to mitigate DoS attacks which may easily consume all server memory and crash the entire system (or at least Sanic processes, if the admin was careful to set suitable ulimits). If you need larger headers, you are probably doing something very wrong, and as a primary solution you should be looking into fixing your application (e.g. use a session token instead of scramming everything into cookies, or use browser localStorage). The header memory use expands a lot as Sanic parses the byte buffer into Python data structures, and with a large number of parallel requests this becomes a problem. Other software such as proxies and clients may also have their own limits that affect for your application at some point.

That being said, it might still be a good idea to have it configurable for those special cases where huge headers cannot be avoided, especially in intranet services where DoS is not a concern. I believe this needs to stay at least a few kilobytes lower than `REQUEST_BUFFER_SIZE` (by default 64 KiB) to avoid deadlocks (need to check sanic/http.py).

Relevant discussion elsewhere:
* https://stackoverflow.com/questions/686217/maximum-on-http-header-values
* https://github.com/nodejs/node/issues/24692

Thanks for clarification. I believe there was no such limit in 20.12 and I was a bit surprised when I saw all those 413 errors after upgrading to 21.03. I agree that reducing the header size is the best approach and this is what I will try to do, although I believe this can sometimes be problematic, e.g. when integrating with an existing service which uses large headers.
I am okay allowing for some variance here, and in most cases am on board with letting the developer make the choice. However, I'd feel more comfortable having a max config value to guard from someone putting too high a value. It's Python, so someone that really knew the consequences and had the use case, they could still monkey patch around it. I believe some other servers out there do something similar. A default value that can be configured up to some maximum.
FWIW I've been using the approach of "design it so users who really want to change it can monkey-patch it" to tackle this request and it went well — no issues filed anymore — https://websockets.readthedocs.io/en/stable/security.html#other-limits

Resolving patch:
diff --git a/sanic/app.py b/sanic/app.py
--- a/sanic/app.py
+++ b/sanic/app.py
@@ -183,7 +183,6 @@ def __init__(
 
         if register is not None:
             self.config.REGISTER = register
-
         if self.config.REGISTER:
             self.__class__.register_app(self)
 
diff --git a/sanic/config.py b/sanic/config.py
--- a/sanic/config.py
+++ b/sanic/config.py
@@ -4,6 +4,8 @@
 from typing import Any, Dict, Optional, Union
 from warnings import warn
 
+from sanic.http import Http
+
 from .utils import load_module_from_file_location, str_to_bool
 
 
@@ -28,6 +30,7 @@
     "REAL_IP_HEADER": None,
     "REGISTER": True,
     "REQUEST_BUFFER_SIZE": 65536,  # 64 KiB
+    "REQUEST_MAX_HEADER_SIZE": 8192,  # 8 KiB, but cannot exceed 16384
     "REQUEST_ID_HEADER": "X-Request-ID",
     "REQUEST_MAX_SIZE": 100000000,  # 100 megabytes
     "REQUEST_TIMEOUT": 60,  # 60 seconds
@@ -42,12 +45,36 @@
 
 
 class Config(dict):
+    ACCESS_LOG: bool
+    EVENT_AUTOREGISTER: bool
+    FALLBACK_ERROR_FORMAT: str
+    FORWARDED_FOR_HEADER: str
+    FORWARDED_SECRET: Optional[str]
+    GRACEFUL_SHUTDOWN_TIMEOUT: float
+    KEEP_ALIVE_TIMEOUT: int
+    KEEP_ALIVE: bool
+    PROXIES_COUNT: Optional[int]
+    REAL_IP_HEADER: Optional[str]
+    REGISTER: bool
+    REQUEST_BUFFER_SIZE: int
+    REQUEST_MAX_HEADER_SIZE: int
+    REQUEST_ID_HEADER: str
+    REQUEST_MAX_SIZE: int
+    REQUEST_TIMEOUT: int
+    RESPONSE_TIMEOUT: int
+    WEBSOCKET_MAX_QUEUE: int
+    WEBSOCKET_MAX_SIZE: int
+    WEBSOCKET_PING_INTERVAL: int
+    WEBSOCKET_PING_TIMEOUT: int
+    WEBSOCKET_READ_LIMIT: int
+    WEBSOCKET_WRITE_LIMIT: int
+
     def __init__(
         self,
         defaults: Dict[str, Union[str, bool, int, float, None]] = None,
         load_env: Optional[Union[bool, str]] = True,
         env_prefix: Optional[str] = SANIC_PREFIX,
-        keep_alive: Optional[int] = None,
+        keep_alive: Optional[bool] = None,
     ):
         defaults = defaults or {}
         super().__init__({**DEFAULT_CONFIG, **defaults})
@@ -72,6 +99,8 @@ def __init__(
         else:
             self.load_environment_vars(SANIC_PREFIX)
 
+        self._configure_header_size()
+
     def __getattr__(self, attr):
         try:
             return self[attr]
@@ -80,6 +109,19 @@ def __getattr__(self, attr):
 
     def __setattr__(self, attr, value):
         self[attr] = value
+        if attr in (
+            "REQUEST_MAX_HEADER_SIZE",
+            "REQUEST_BUFFER_SIZE",
+            "REQUEST_MAX_SIZE",
+        ):
+            self._configure_header_size()
+
+    def _configure_header_size(self):
+        Http.set_header_max_size(
+            self.REQUEST_MAX_HEADER_SIZE,
+            self.REQUEST_BUFFER_SIZE - 4096,
+            self.REQUEST_MAX_SIZE,
+        )
 
     def load_environment_vars(self, prefix=SANIC_PREFIX):
         """
diff --git a/sanic/http.py b/sanic/http.py
--- a/sanic/http.py
+++ b/sanic/http.py
@@ -64,6 +64,9 @@ class Http:
     :raises RuntimeError:
     """
 
+    HEADER_CEILING = 16_384
+    HEADER_MAX_SIZE = 0
+
     __slots__ = [
         "_send",
         "_receive_more",
@@ -169,7 +172,6 @@ async def http1_request_header(self):
         """
         Receive and parse request header into self.request.
         """
-        HEADER_MAX_SIZE = min(8192, self.request_max_size)
         # Receive until full header is in buffer
         buf = self.recv_buffer
         pos = 0
@@ -180,12 +182,12 @@ async def http1_request_header(self):
                 break
 
             pos = max(0, len(buf) - 3)
-            if pos >= HEADER_MAX_SIZE:
+            if pos >= self.HEADER_MAX_SIZE:
                 break
 
             await self._receive_more()
 
-        if pos >= HEADER_MAX_SIZE:
+        if pos >= self.HEADER_MAX_SIZE:
             raise PayloadTooLarge("Request header exceeds the size limit")
 
         # Parse header content
@@ -541,3 +543,10 @@ def respond(self, response: BaseHTTPResponse) -> BaseHTTPResponse:
     @property
     def send(self):
         return self.response_func
+
+    @classmethod
+    def set_header_max_size(cls, *sizes: int):
+        cls.HEADER_MAX_SIZE = min(
+            *sizes,
+            cls.HEADER_CEILING,
+        )


Test patch:
diff --git a/tests/test_headers.py b/tests/test_headers.py
--- a/tests/test_headers.py
+++ b/tests/test_headers.py
@@ -7,6 +7,13 @@
 from sanic.http import Http
 
 
+@pytest.fixture
+def raised_ceiling():
+    Http.HEADER_CEILING = 32_768
+    yield
+    Http.HEADER_CEILING = 16_384
+
+
 @pytest.mark.parametrize(
     "input, expected",
     [
@@ -76,15 +83,75 @@ async def _receive_more():
         recv_buffer += b"123"
 
     protocol = Mock()
+    Http.set_header_max_size(1)
     http = Http(protocol)
     http._receive_more = _receive_more
-    http.request_max_size = 1
     http.recv_buffer = recv_buffer
 
     with pytest.raises(PayloadTooLarge):
         await http.http1_request_header()
 
 
+@pytest.mark.asyncio
+async def test_header_size_increased_okay():
+    recv_buffer = bytearray()
+
+    async def _receive_more():
+        nonlocal recv_buffer
+        recv_buffer += b"123"
+
+    protocol = Mock()
+    Http.set_header_max_size(12_288)
+    http = Http(protocol)
+    http._receive_more = _receive_more
+    http.recv_buffer = recv_buffer
+
+    with pytest.raises(PayloadTooLarge):
+        await http.http1_request_header()
+
+    assert len(recv_buffer) == 12_291
+
+
+@pytest.mark.asyncio
+async def test_header_size_exceeded_maxed_out():
+    recv_buffer = bytearray()
+
+    async def _receive_more():
+        nonlocal recv_buffer
+        recv_buffer += b"123"
+
+    protocol = Mock()
+    Http.set_header_max_size(18_432)
+    http = Http(protocol)
+    http._receive_more = _receive_more
+    http.recv_buffer = recv_buffer
+
+    with pytest.raises(PayloadTooLarge):
+        await http.http1_request_header()
+
+    assert len(recv_buffer) == 16_389
+
+
+@pytest.mark.asyncio
+async def test_header_size_exceeded_raised_ceiling(raised_ceiling):
+    recv_buffer = bytearray()
+
+    async def _receive_more():
+        nonlocal recv_buffer
+        recv_buffer += b"123"
+
+    protocol = Mock()
+    http = Http(protocol)
+    Http.set_header_max_size(65_536)
+    http._receive_more = _receive_more
+    http.recv_buffer = recv_buffer
+
+    with pytest.raises(PayloadTooLarge):
+        await http.http1_request_header()
+
+    assert len(recv_buffer) == 32_772
+
+
 def test_raw_headers(app):
     app.route("/")(lambda _: text(""))
     request, _ = app.test_client.get(