Repository: encode/uvicorn

Issue:
WSGI middleware should stream request body, rather than loading it all at once.
Hi! I'm deploying a Django app with uvicorn, running on k8s. Our containers were being killed, and I've found that when users upload large files, uvicorn increases memory usage, and slows to a crawl. Eventually causing an OOM.

I'm not sure where this is happening yet, and I suspect could be related to https://github.com/django/channels/pull/1251, but I did a bit more digging and I'm not totally sure. I've tried running uvicorn in wsgi mode, and completely removed channels from the django install, and I'm getting the same behavior. File uploads are being loaded into memory (rather than streaming to disk, as they should be), and upload speeds slow to a crawl. The same app running on gunicorn works fine.

The file I'm testing with is about 470mb.


Discussion:
So both channels, and uvicorn's WSGI middleware consume the entire request body into memory rather than streaming it (due to some complexities of bridging across from async on the frontend, to WSGI's threaded concurrency)

See https://github.com/encode/uvicorn/blob/master/uvicorn/middleware/wsgi.py#L76-L83

Ideally we'd rework the WSGI middleware to provide proper streaming of request bodies.
Maybe using `tempfile.SpooledTemporaryFile` instead is a simple and easy good idea.
@peterlandry @tomchristie What do you think of my newly submitted pr? Use `bytearray` to solve it.
I was looking into this for @Flauschbaellchen and wrote a demo that illustrates how hard this problem hits. While working on this I ran into #1345 all the time so I though this would also aid in fixing that issue. Of course today this wasn't reproducing anymore as the newest release fixes that bug. :+1:

Still, here is an example that can be used to compare the different ways to upload, comparing especially a2wsgi with the middle ware included with uvicorn. Instead of storing the uploads, I just compute a sha256 hash:

[issue371_demo.zip](https://github.com/encode/uvicorn/files/7998556/issue371_demo.zip)

After unpacking, you can run this like this:

```bash
$ cd issue371_demo/
$ sudo docker build .
```

When using the a2wsgi middleware for wsgi, 10 concurrent uploads using curl complete like this (uvicorn is run from gunicorn with 2 workers, wsgi middleware may use 2 threads):

```
Upload timings for wsgi/a2wsgi: [
1.3421823978424072, 1.4076666831970215, 1.4279963970184326, 1.448310375213623,
2.0521488189697266, 2.1042470932006836, 2.168593168258667, 2.179323434829712,
2.662322998046875, 2.7450664043426514].
Highest latency: 0.025380373001098633.
```

So basically 4 uploads (2 processes * 2 threads) are processed concurrently. While doing the upload, I run an asgi request concurrently to check the latency for processing ASGI requests. So here they complete after 0.025 s worst case.

Compare this to using the wsgi middleware in uvicorn:

```
Upload timings for wsgi/uvicorn: [
33.341026306152344, 33.38411903381348, 33.76080870628357,
73.60798692703247, 73.64170670509338, 73.97475504875183,
74.0065565109253, 74.37995886802673, 74.40127992630005,
74.73946642875671].
Highest latency: 0.619253396987915.
```

No idea why the uploads complete at such irregular intervals.

And, of course, using only asgi, the uploads are processed concurrently and the latency is much better:

```
Upload timings for asgi: [1.972560167312622, 2.0667288303375244, 2.0667288303375244, 2.0784835815429688, 2.5489165782928467, 2.6735644340515137, 2.724545955657959, 2.724545955657959, 2.735142707824707, 2.735142707824707].
Highest latency: 0.020055770874023438.
```

BTW: If you are adventurous, you can raise the size of the uploads. Note that the pull request https://github.com/encode/uvicorn/pull/1329 actually made matters worse in my tests because it took down my machine for many big uploads.

Simply replacing our previous naive byte concatenation with https://github.com/encode/uvicorn/pull/1329 makes a massive difference here.

Still *feasible* that an implementation that streams the request body through the WSGI adapter would be preferable, but it's not *absolutely* necessary that's the case. Sometime simple wins, just by virtue of being simple.
@tomchristie I agree that simple is preferable. But it seems you did not read my comment:

> Note that the pull request https://github.com/encode/uvicorn/pull/1329 actually made matters worse in my tests because it took down my machine for many big uploads.

I'd consider this a security issue given that a simply running a few big uploads will take down an application (or maybe just a single pod) by collecting all incoming data in memory before even handing it over the the application code.

The goal here is to replace the `WSGIMiddleware` we have by the `a2wsgi.WSGIMiddleware`.

Implementation is available on https://github.com/encode/uvicorn/pull/1303/. If someone wants to take that over, it will be really helpful.
Or... Option 1 from https://github.com/encode/uvicorn/pull/1303#issuecomment-1036074401. 

Resolving patch:
diff --git a/uvicorn/middleware/wsgi.py b/uvicorn/middleware/wsgi.py
--- a/uvicorn/middleware/wsgi.py
+++ b/uvicorn/middleware/wsgi.py
@@ -2,6 +2,7 @@
 import concurrent.futures
 import io
 import sys
+import warnings
 from collections import deque
 from typing import TYPE_CHECKING, Deque, Iterable, Optional, Tuple
 
@@ -73,8 +74,13 @@ def build_environ(
     return environ
 
 
-class WSGIMiddleware:
+class _WSGIMiddleware:
     def __init__(self, app: WSGIApp, workers: int = 10):
+        warnings.warn(
+            "Uvicorn's native WSGI implementation is deprecated, you "
+            "should switch to a2wsgi (`pip install a2wsgi`).",
+            DeprecationWarning,
+        )
         self.app = app
         self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=workers)
 
@@ -188,3 +194,9 @@ def wsgi(self, environ: Environ, start_response: StartResponse) -> None:
         }
         self.send_queue.append(empty_body)
         self.loop.call_soon_threadsafe(self.send_event.set)
+
+
+try:
+    from a2wsgi import WSGIMiddleware
+except ModuleNotFoundError:
+    WSGIMiddleware = _WSGIMiddleware  # type: ignore[misc, assignment]


Test patch:
diff --git a/tests/middleware/test_wsgi.py b/tests/middleware/test_wsgi.py
--- a/tests/middleware/test_wsgi.py
+++ b/tests/middleware/test_wsgi.py
@@ -1,12 +1,13 @@
 import io
 import sys
-from typing import TYPE_CHECKING, AsyncGenerator, List
+from typing import TYPE_CHECKING, AsyncGenerator, Callable, List
 
+import a2wsgi
 import httpx
 import pytest
 
 from uvicorn._types import Environ, StartResponse
-from uvicorn.middleware.wsgi import WSGIMiddleware, build_environ
+from uvicorn.middleware import wsgi
 
 if TYPE_CHECKING:
     from asgiref.typing import HTTPRequestEvent, HTTPScope
@@ -34,7 +35,7 @@ def echo_body(environ: Environ, start_response: StartResponse) -> List[bytes]:
     return [output]
 
 
-def raise_exception(environ: Environ, start_response: StartResponse) -> RuntimeError:
+def raise_exception(environ: Environ, start_response: StartResponse) -> List[bytes]:
     raise RuntimeError("Something went wrong")
 
 
@@ -52,9 +53,14 @@ def return_exc_info(environ: Environ, start_response: StartResponse) -> List[byt
         return [output]
 
 
+@pytest.fixture(params=[wsgi._WSGIMiddleware, a2wsgi.WSGIMiddleware])
+def wsgi_middleware(request: pytest.FixtureRequest) -> Callable:
+    return request.param
+
+
 @pytest.mark.anyio
-async def test_wsgi_get() -> None:
-    app = WSGIMiddleware(hello_world)
+async def test_wsgi_get(wsgi_middleware: Callable) -> None:
+    app = wsgi_middleware(hello_world)
     async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
         response = await client.get("/")
     assert response.status_code == 200
@@ -62,8 +68,8 @@ async def test_wsgi_get() -> None:
 
 
 @pytest.mark.anyio
-async def test_wsgi_post() -> None:
-    app = WSGIMiddleware(echo_body)
+async def test_wsgi_post(wsgi_middleware: Callable) -> None:
+    app = wsgi_middleware(echo_body)
     async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
         response = await client.post("/", json={"example": 123})
     assert response.status_code == 200
@@ -71,12 +77,12 @@ async def test_wsgi_post() -> None:
 
 
 @pytest.mark.anyio
-async def test_wsgi_put_more_body() -> None:
+async def test_wsgi_put_more_body(wsgi_middleware: Callable) -> None:
     async def generate_body() -> AsyncGenerator[bytes, None]:
         for _ in range(1024):
             yield b"123456789abcdef\n" * 64
 
-    app = WSGIMiddleware(echo_body)
+    app = wsgi_middleware(echo_body)
     async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
         response = await client.put("/", content=generate_body())
     assert response.status_code == 200
@@ -84,25 +90,25 @@ async def generate_body() -> AsyncGenerator[bytes, None]:
 
 
 @pytest.mark.anyio
-async def test_wsgi_exception() -> None:
+async def test_wsgi_exception(wsgi_middleware: Callable) -> None:
     # Note that we're testing the WSGI app directly here.
     # The HTTP protocol implementations would catch this error and return 500.
-    app = WSGIMiddleware(raise_exception)
+    app = wsgi_middleware(raise_exception)
     async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
         with pytest.raises(RuntimeError):
             await client.get("/")
 
 
 @pytest.mark.anyio
-async def test_wsgi_exc_info() -> None:
+async def test_wsgi_exc_info(wsgi_middleware: Callable) -> None:
     # Note that we're testing the WSGI app directly here.
     # The HTTP protocol implementations would catch this error and return 500.
-    app = WSGIMiddleware(return_exc_info)
+    app = wsgi_middleware(return_exc_info)
     async with httpx.AsyncClient(app=app, base_url="http://testserver") as client:
         with pytest.raises(RuntimeError):
             response = await client.get("/")
 
-    app = WSGIMiddleware(return_exc_info)
+    app = wsgi_middleware(return_exc_info)
     transport = httpx.ASGITransport(
         app=app,
         raise_app_exceptions=False,
@@ -136,6 +142,6 @@ def test_build_environ_encoding() -> None:
         "body": b"",
         "more_body": False,
     }
-    environ = build_environ(scope, message, io.BytesIO(b""))
+    environ = wsgi.build_environ(scope, message, io.BytesIO(b""))
     assert environ["PATH_INFO"] == "/文".encode("utf8").decode("latin-1")
     assert environ["HTTP_KEY"] == "value1,value2"