Repository: django/channels

Issue:
Channels 3+ with Django 2.2 under gunicorn/uvicorn fails when hit with concurrent requests
The issue I'm seeing is that with a very barebones Django/Channels/Uvicorn/Gunicorn setup I'm seeing consistent failures when more than one request is made concurrently. Errors included at the bottom.

**Reproduction instructions**

I've made the simplest Django 2.2 LTS project I can at https://github.com/AlexHill/django_channels_test.

Just clone that, then `make env` to create the venv and `make asgi` to start the application server. Then run

```env/bin/python load_test.py 1 10```

to hit it with ten requests serially, which should work fine, and then

```env/bin/python load_test.py 10 10```

to hit it with ten requests in parallel, which should fail.

Downgrading to Channels 2.x makes the problem go away.

**Errors**

```[2020-11-19 06:05:19 +0000] [22144] [ERROR] ASGI callable returned without starting response.```

And sometimes:

```
[2020-11-19 06:05:19 +0000] [22144] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 391, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/channels/routing.py", line 71, in __call__
    return await application(scope, receive, send)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/channels/http.py", line 200, in __call__
    await self.handle(body_stream)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/asgiref/sync.py", line 304, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
    return (yield from fut)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/asgiref/sync.py", line 343, in thread_handler
    return func(*args, **kwargs)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/channels/http.py", line 258, in handle
    self.send(response_message)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/asgiref/sync.py", line 147, in __call__
    return call_result.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/asgiref/sync.py", line 212, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/alexh/signiq/uvicorn_test/env/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 535, in send
    raise RuntimeError(msg % message_type)
RuntimeError: Unexpected ASGI message 'http.response.start' sent, after response already completed.
```


Discussion:
we are also facing this issue with the provided versions.
but we are facing it with the django debug server and daphne server.
Hi @AlexHill — thanks for the report and the sample project!

Looks like an issue in `AsgiHandler`... — I don't have the bandwidth to dig-in this week — have you had any chance to look to a cause? 

Resolving patch:
diff --git a/channels/http.py b/channels/http.py
--- a/channels/http.py
+++ b/channels/http.py
@@ -188,8 +188,6 @@ async def __call__(self, scope, receive, send):
                 "The AsgiHandler can only handle HTTP connections, not %s"
                 % scope["type"]
             )
-        self.scope = scope
-        self.send = async_to_sync(send)
 
         # Receive the HTTP request body as a stream object.
         try:
@@ -197,7 +195,7 @@ async def __call__(self, scope, receive, send):
         except RequestAborted:
             return
         # Launch into body handling (and a synchronous subthread).
-        await self.handle(body_stream)
+        await self.handle(scope, async_to_sync(send), body_stream)
 
     async def read_body(self, receive):
         """Reads a HTTP body from an ASGI connection."""
@@ -220,19 +218,19 @@ async def read_body(self, receive):
         return body_file
 
     @sync_to_async
-    def handle(self, body):
+    def handle(self, scope, send, body):
         """
         Synchronous message processing.
         """
         # Set script prefix from message root_path, turning None into empty string
-        script_prefix = self.scope.get("root_path", "") or ""
+        script_prefix = scope.get("root_path", "") or ""
         if settings.FORCE_SCRIPT_NAME:
             script_prefix = settings.FORCE_SCRIPT_NAME
         set_script_prefix(script_prefix)
-        signals.request_started.send(sender=self.__class__, scope=self.scope)
+        signals.request_started.send(sender=self.__class__, scope=scope)
         # Run request through view system
         try:
-            request = self.request_class(self.scope, body)
+            request = self.request_class(scope, body)
         except UnicodeDecodeError:
             logger.warning(
                 "Bad Request (UnicodeDecodeError)",
@@ -255,7 +253,7 @@ def handle(self, body):
                 response.block_size = 1024 * 512
         # Transform response into messages, which we yield back to caller
         for response_message in self.encode_response(response):
-            self.send(response_message)
+            send(response_message)
         # Close the response now we're done with it
         response.close()
 
diff --git a/channels/staticfiles.py b/channels/staticfiles.py
--- a/channels/staticfiles.py
+++ b/channels/staticfiles.py
@@ -51,11 +51,15 @@ class StaticFilesHandler(AsgiHandler):
 
     # TODO: Review hierarchy here. Do we NEED to inherit BaseHandler, AsgiHandler?
 
+    async def __call__(self, scope, receive, send):
+        self.static_base_url = scope["static_base_url"][2]
+        return await super().__call__(scope, receive, send)
+
     def file_path(self, url):
         """
         Returns the relative path to the media file on disk for the given URL.
         """
-        relative_url = url[len(self.scope["static_base_url"][2]) :]
+        relative_url = url[len(self.static_base_url) :]
         return url2pathname(relative_url)
 
     def serve(self, request):


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -351,6 +351,35 @@ async def test_handler_body_ignore_extra():
     assert body_stream.read() == b"chunk one"
 
 
+@pytest.mark.django_db
+@pytest.mark.asyncio
+@pytest.mark.filterwarnings("ignore::DeprecationWarning")
+async def test_handler_concurrent_requests():
+    """
+    Tests request handling ignores anything after more_body: False
+    """
+    scope = {"type": "http", "http_version": "1.1", "method": "GET", "path": "/test/"}
+    handler = MockHandler()
+    comm_1 = ApplicationCommunicator(handler, {**scope})
+    comm_2 = ApplicationCommunicator(handler, {**scope})
+
+    request_1 = comm_1.send_input(
+        {"type": "http.request", "body": b"request 1", "more_body": False}
+    )
+    request_2 = comm_2.send_input(
+        {"type": "http.request", "body": b"request 2", "more_body": False}
+    )
+
+    await request_1
+    await request_2
+
+    await comm_1.receive_output(1)  # response start
+    await comm_1.receive_output(1)  # response body
+
+    await comm_2.receive_output(1)  # response start
+    await comm_2.receive_output(1)  # response body
+
+
 @pytest.mark.django_db(transaction=True)
 @pytest.mark.asyncio
 async def test_sessions():
diff --git a/tests/test_staticfiles.py b/tests/test_staticfiles.py
--- a/tests/test_staticfiles.py
+++ b/tests/test_staticfiles.py
@@ -1,6 +1,6 @@
 import pytest
 
-from channels.staticfiles import StaticFilesWrapper
+from channels.staticfiles import StaticFilesHandler, StaticFilesWrapper
 
 
 @pytest.fixture(autouse=True)
@@ -99,3 +99,27 @@ def test_is_single_callable():
         "StaticFilesWrapper should be recognized as a single callable by "
         "asgiref compatibility tools"
     )
+
+
+@pytest.mark.asyncio
+async def test_staticfiles_handler_can_generate_file_path():
+    """
+    StaticFilesHandler.file_path must not rely on scope being assigned to self.
+    """
+
+    class MockedHandler(StaticFilesHandler):
+        async def __call__(self, scope, receive, send):
+            # Equivalent setUp from real __call__.
+            request = self.request_class(scope, "")
+            self.static_base_url = scope["static_base_url"][2]
+            # Method under test.
+            return self.file_path(request.path)
+
+    wrapper = StaticFilesWrapper(
+        MockApplication("application"), staticfiles_handler=MockedHandler
+    )
+    scope = request_for_path("/static/image.png")
+    scope["method"] = "GET"
+    assert (
+        await wrapper(scope, None, None) == "/image.png"
+    ), "StaticFilesWrapper should serve paths under the STATIC_URL path"