Repository: encode/uvicorn

Issue:
HttpToolsProtocol: receive() hangs due to signalling mixup between request cycles
I have a FastAPI server running with uvicorn. Starting with starlette 0.13.7 I'm hitting a race condition on a specific resource limited setup (RPi), however if my analysis is correct the root cause is a bug in uvicorn's protocol.http.httptools_impl.

The race condition is causing starlette to keep receive()ing on a RequestResponseCycle for a bit after the response is fully sent, while the client sends a new request on the same connection:

1. Cycle 1 is blocking on `message_event` in RequestResponseCycle.receive() (called before `response_complete` is set)
2. HttpToolsProtocol reads the body of the new request into Cycle 2 and sets `message_event`
3. Cycle 1 clears `message_event` and returns
4. Receiving on Cycle 2 blocks because `message_event` is already cleared

If I'm not mistaken, the fix is pretty simple:
1. Create a separate event for each cycle.
2. In RequestResponseCycle.send, when setting `response_complete` also set `message_event` to kill any lingering receive()s

## Minimal test case

This raw ASGI server imitates the behavior in Starlette that triggers the bug.

```python
import asyncio

async def wait_for_disconnect(receive):
    while True:
        p = await receive()
        if p['type'] == 'http.disconnect':
            print('Disconnected!')
            break
           
async def app(scope, receive, send):
    await asyncio.sleep(0.2)
    m = await receive()

    if m['type'] == 'lifespan.startup':
        await send({'type': 'lifespan.startup.complete'})
    elif m['type'] == 'http.request':
        if scope['path'] == '/foo':
            asyncio.create_task(wait_for_disconnect(receive))
            await asyncio.sleep(0.2)

        await send({'type': 'http.response.start', 'status': 404})
        await send({'type': 'http.response.body', 'body': b'Not found!\n'})
```

Test with `curl localhost:8000/foo localhost:8000/bar`. You should see that it hangs at the request to /bar.

The /foo request calls receive() in the background, and this call is still running in the background when the request to /bar arrives. The `http.request` message for the /bar request incorrectly wakes up the background (/foo) receive, causing the receive() for /bar to hang forever, as there are no more events for this request.


Discussion:
hi @itayperl 
afaik FastAPI currently pins Starlette on 0.13.6, 1st thing to check would be if that what you described still happens respecting the pin.
@euri10, I'm sorry if my description was a bit unclear, but this is a bug in uvicorn that should be fixed regardless of Starlette. It just didn't manifest (in my usecase at least) with older versions of Starlette. Any code working with multiple httptools_impl.RequestResponseCycle objects in parallel can potentially hit this bug.
could you either confirm you experience this on 0.13.8 too (because can't remember exactly but I think 0.13.8 reverted something in middleware that makes me think it's related, can be totally off of course)

I was able to reproduce the bug on both 0.13.6 and 0.13.8 by adding the 100ms sleep in starlette as described above.

I can't say if it will actually reproduce "in the field" in either version, without the sleep, but I did see it happen on 0.13.7 in a real production setting after a day or so of operation.
ok I don't want to ditch this one, but the fact you have to kind of "hack"
the app to reproduce doesn't help.
To stay constructive, what would be fantastic would be a simple asgi app
that reproduces the issue, then we can advise on what to do next, seems ok
for you ?


On Mon, Oct 12, 2020 at 3:37 PM Itay Perl <notifications@github.com> wrote:

> I was able to reproduce the bug on both 0.13.6 and 0.13.8 by adding the
> 100ms sleep in starlette as described above.
>
> I can't say if it will actually reproduce "in the field" in either
> version, without the sleep, but I did see it happen on 0.13.7 in a real
> production setting after a day or so of operation.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/encode/uvicorn/issues/748#issuecomment-707125086>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAINSPT7ZLV7CV2R5CPKRKLSKMBCDANCNFSM4PZJ7CMQ>
> .
>


-- 
benoit barthelet
http://pgp.mit.edu/pks/lookup?op=get&search=0xF150E01A72F6D2EE

Edit: updated test case moved to issue description
@euri10, were you able to reproduce the bug using the new test case?

Resolving patch:
diff --git a/uvicorn/protocols/http/h11_impl.py b/uvicorn/protocols/http/h11_impl.py
--- a/uvicorn/protocols/http/h11_impl.py
+++ b/uvicorn/protocols/http/h11_impl.py
@@ -113,7 +113,6 @@ def __init__(self, config, server_state, _loop=None):
         self.scope = None
         self.headers = None
         self.cycle = None
-        self.message_event = asyncio.Event()
 
     # Protocol interface
     def connection_made(self, transport):
@@ -146,7 +145,8 @@ def connection_lost(self, exc):
                 # Premature client disconnect
                 pass
 
-        self.message_event.set()
+        if self.cycle is not None:
+            self.cycle.message_event.set()
         if self.flow is not None:
             self.flow.resume_writing()
 
@@ -234,7 +234,7 @@ def handle_events(self):
                     access_logger=self.access_logger,
                     access_log=self.access_log,
                     default_headers=self.default_headers,
-                    message_event=self.message_event,
+                    message_event=asyncio.Event(),
                     on_response=self.on_response_complete,
                 )
                 task = self.loop.create_task(self.cycle.run_asgi(app))
@@ -247,7 +247,7 @@ def handle_events(self):
                 self.cycle.body += event.data
                 if len(self.cycle.body) > HIGH_WATER_LIMIT:
                     self.flow.pause_reading()
-                self.message_event.set()
+                self.cycle.message_event.set()
 
             elif event_type is h11.EndOfMessage:
                 if self.conn.our_state is h11.DONE:
@@ -255,7 +255,7 @@ def handle_events(self):
                     self.conn.start_next_cycle()
                     continue
                 self.cycle.more_body = False
-                self.message_event.set()
+                self.cycle.message_event.set()
 
     def handle_upgrade(self, event):
         upgrade_value = None
@@ -491,6 +491,7 @@ async def send(self, message):
             # Handle response completion
             if not more_body:
                 self.response_complete = True
+                self.message_event.set()
                 event = h11.EndOfMessage()
                 output = self.conn.send(event)
                 self.transport.write(output)
diff --git a/uvicorn/protocols/http/httptools_impl.py b/uvicorn/protocols/http/httptools_impl.py
--- a/uvicorn/protocols/http/httptools_impl.py
+++ b/uvicorn/protocols/http/httptools_impl.py
@@ -121,7 +121,6 @@ def __init__(self, config, server_state, _loop=None):
         self.headers = None
         self.expect_100_continue = False
         self.cycle = None
-        self.message_event = asyncio.Event()
 
     # Protocol interface
     def connection_made(self, transport):
@@ -146,7 +145,8 @@ def connection_lost(self, exc):
 
         if self.cycle and not self.cycle.response_complete:
             self.cycle.disconnected = True
-        self.message_event.set()
+        if self.cycle is not None:
+            self.cycle.message_event.set()
         if self.flow is not None:
             self.flow.resume_writing()
 
@@ -267,7 +267,7 @@ def on_headers_complete(self):
             access_logger=self.access_logger,
             access_log=self.access_log,
             default_headers=self.default_headers,
-            message_event=self.message_event,
+            message_event=asyncio.Event(),
             expect_100_continue=self.expect_100_continue,
             keep_alive=http_version != "1.0",
             on_response=self.on_response_complete,
@@ -288,13 +288,13 @@ def on_body(self, body: bytes):
         self.cycle.body += body
         if len(self.cycle.body) > HIGH_WATER_LIMIT:
             self.flow.pause_reading()
-        self.message_event.set()
+        self.cycle.message_event.set()
 
     def on_message_complete(self):
         if self.parser.should_upgrade() or self.cycle.response_complete:
             return
         self.cycle.more_body = False
-        self.message_event.set()
+        self.cycle.message_event.set()
 
     def on_response_complete(self):
         # Callback for pipelined HTTP requests to be started.
@@ -530,6 +530,7 @@ async def send(self, message):
                 if self.expected_content_length != 0:
                     raise RuntimeError("Response content shorter than Content-Length")
                 self.response_complete = True
+                self.message_event.set()
                 if not self.keep_alive:
                     self.transport.close()
                 self.on_response()


Test patch: