Repository: encode/starlette

Issue:
Memory usage streaming large responses
We've been running into memory issues when providing very large async generators to a streaming response.  We have these generators producing large (larger than memory set) responses in a way that allows us to only keep small chunks in memory at a time.  However, it looks like the BaseHTTPMiddleware implementation uses an asyncio queue to store the individual chunks:

https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L30

This prevents any network backpressure handling -- if the client that is receiving the streaming response is on a slow connection, the queue will happily grow without bound and consume all memory, triggering kernel out-of-memory, when the ideal handling here would be for send to block (yield) when this happens.  I believe this would naturally happen if there were no queue here at all, so I am wondering why it needs to be here?

Would a PR to remove the queueing be accepted?

If not, what is the appropriate way to override this to not use a queue?  We can write our own, but the use of BaseHTTPMiddleware is hardcoded: https://github.com/encode/starlette/blob/519f5750b5e797bb3d4805fd29657674304ce397/starlette/applications.py#L197, leaving only some fairly hacky approaches to preventing this queueing.


Discussion:
I kind of remember it has been discussed already here, not sure it's
relevant to your case though

https://github.com/encode/starlette/issues/802

On Mon, Jul 27, 2020, 7:23 PM via <notifications@github.com> wrote:

> We've been running into memory issues when providing very large async
> generators to a streaming response. We have these generators producing
> large (larger than memory set) responses in a way that allows us to only
> keep small chunks in memory at a time. However, it looks like the
> BaseHTTPMiddleware implementation uses an asyncio queue to store the
> individual chunks:
>
>
> https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L30
>
> This prevents any network backpressure handling -- if the client that is
> receiving the streaming response is on a slow connection, the queue will
> happily grow without bound and consume all memory, triggering kernel
> out-of-memory, when the ideal handling here would be for send to block
> (yield) when this happens. I believe this would naturally happen if there
> were no queue here at all, so I am wondering why it needs to be here?
>
> Would a PR to remove the queueing be accepted?
>
> If not, what is the appropriate way to override this to not use a queue?
> We can write our own, but the use of BaseHTTPMiddleware is hardcoded:
> https://github.com/encode/starlette/blob/519f5750b5e797bb3d4805fd29657674304ce397/starlette/applications.py#L197,
> leaving only some fairly hacky approaches to preventing this queueing.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/encode/starlette/issues/1012>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAINSPT7UOTUWBXAWKM352TR5WZX5ANCNFSM4PJAVLSA>
> .
>

I did see that ticket, but I do think its important to distinguish between the high level backpressure logic that I think that ticket is mostly about (specifically managing concurrent request limits, etc), and network backpressure.  I do not believe that ticket is correct when it suggests that uvicorn/gnuicorn manages the low-level network socket backpressure, as the use of the asyncio.Queue there is completely in starlette, and will happily absorb the entire response payload until it dies.
I think queueing is fine, but the queue is unbounded which results in losing the backpressure properties provided by the server. (As you said, it’s allowed to grow forever even if the other side isn’t consuming chunks fast enough.)

In my mind setting a `maxsize` of eg 10 would resolve the memory issues to you’re seeing, because then if the client isn’t consuming chunks fast enough the app will block on `send()` until there’s room in the queue.

Edit: actually as discussed by @erewok in the chat, a `maxsize` of 1 should be just enough. This way we stick to the backpressure behavior of the server as close as we can, and I don't think there's a risk of deadlock. In any case, any fix would be worth validating on a real-life big upload too. :-)
For a bit of background, the `BaseHTTPMiddleware` offers a way for users to write middleware that takes a request and builds a response. This is a bit closer to the request-handling logic than other middleware, but in order to assemble a response, the `BaseHTTPMiddleware` needs some generic way to handle any potential response the handler would produce, which is the reason for the queue and streaming.

Also, as @florimondmanca mentioned above, @via If you have an easy opportunity to do so, you may consider experimentally running the branch from PR #1017  ([branch: `stream_response_bg`](https://github.com/encode/starlette/tree/stream_response_bg)) and seeing if your memory issues in the case of large responses are resolved. If you do find an opportunity to try it, please report back.

Resolving patch:
diff --git a/starlette/middleware/base.py b/starlette/middleware/base.py
--- a/starlette/middleware/base.py
+++ b/starlette/middleware/base.py
@@ -3,7 +3,7 @@
 
 from starlette.requests import Request
 from starlette.responses import Response, StreamingResponse
-from starlette.types import ASGIApp, Receive, Scope, Send
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
 
 RequestResponseEndpoint = typing.Callable[[Request], typing.Awaitable[Response]]
 DispatchFunction = typing.Callable[
@@ -27,7 +27,7 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
 
     async def call_next(self, request: Request) -> Response:
         loop = asyncio.get_event_loop()
-        queue = asyncio.Queue()  # type: asyncio.Queue
+        queue: "asyncio.Queue[typing.Optional[Message]]" = asyncio.Queue(maxsize=1)
 
         scope = request.scope
         receive = request.receive


Test patch: