Repository: encode/uvicorn

Issue:
Handshaking may not be completed yet at `shutdown` in wsproto impl
Traceback:
```
LocalProtocolError: Event CloseConnection(code=1012, reason=None) cannot be sent during the handshake
  File "gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "uvicorn/workers.py", line 57, in init_process
    super(UvicornWorker, self).init_process()
  File "gunicorn/workers/base.py", line 140, in init_process
    self.run()
  File "uvicorn/workers.py", line 66, in run
    loop.run_until_complete(server.serve(sockets=self.sockets))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "uvicorn/main.py", line 403, in serve
    await self.shutdown(sockets=sockets)
  File "uvicorn/main.py", line 539, in shutdown
    connection.shutdown()
  File "uvicorn/protocols/websockets/wsproto_impl.py", line 115, in shutdown
    output = self.conn.send(wsproto.events.CloseConnection(code=1012))
  File "__init__.py", line 61, in send
    data += self.handshake.send(event)
  File "wsproto/handshake.py", line 101, in send
    "Event {} cannot be sent during the handshake".format(event)
```


Discussion:
Could you please send the command you used?
Does it happen without gunicorn?
Do you have a minimal example of the code that triggers it?


Le sam. 21 mars 2020 à 7:28 AM, MisLink <notifications@github.com> a écrit :

> Traceback:
>
> LocalProtocolError: Event CloseConnection(code=1012, reason=None) cannot be sent during the handshake
>   File "gunicorn/arbiter.py", line 583, in spawn_worker
>     worker.init_process()
>   File "uvicorn/workers.py", line 57, in init_process
>     super(UvicornWorker, self).init_process()
>   File "gunicorn/workers/base.py", line 140, in init_process
>     self.run()
>   File "uvicorn/workers.py", line 66, in run
>     loop.run_until_complete(server.serve(sockets=self.sockets))
>   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
>   File "uvicorn/main.py", line 403, in serve
>     await self.shutdown(sockets=sockets)
>   File "uvicorn/main.py", line 539, in shutdown
>     connection.shutdown()
>   File "uvicorn/protocols/websockets/wsproto_impl.py", line 115, in shutdown
>     output = self.conn.send(wsproto.events.CloseConnection(code=1012))
>   File "__init__.py", line 61, in send
>     data += self.handshake.send(event)
>   File "wsproto/handshake.py", line 101, in send
>     "Event {} cannot be sent during the handshake".format(event)
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/encode/uvicorn/issues/596>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAINSPV2D3TVNJIGDAGBSE3RIRM7VANCNFSM4LQZ45BQ>
> .
>

@euri10 Yes, it happens every time we send TERM to gunicorn master process in our production environment. I checked the source code:
https://github.com/encode/uvicorn/blob/master/uvicorn/protocols/websockets/wsproto_impl.py#L113-L117
and I concluded by myself that there is no check for completion of the handshake.

Sorry for the lack of info, I'll make a minimal example without gunicorn.
The minimal app:
```python
import asyncio


async def app(scope, receive, send):
    assert scope["type"] == "websocket"
    event = await receive()
    if event["type"] == "websocket.connect":
        await asyncio.sleep(10)
        await send({"type": "websocket.accept"})
```

1. start it with uvicorn and wsproto: `uvicorn --ws wsproto app:app` and write down the process id;
2. connect it with a websocket client, I use [websocat](https://github.com/vi/websocat): `websocat ws://127.0.0.1:8000`;
3. kill the server before the sleep finished: `kill <pid>`, then you will see the same traceback.

sorry I cant reproduce, am I doing 
```
(venv) ➜  uvicorn git:(596_shake) ✗ uvicorn --ws wsproto toto:app --log-level=trace
INFO:     Started server process [18661]
INFO:     Waiting for application startup.
TRACE:    ASGI [1] Started scope={'type': 'lifespan'}
TRACE:    ASGI [1] Raised exception
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
TRACE:    127.0.0.1:45208 - Connection made
TRACE:    127.0.0.1:45208 - ASGI [2] Started scope={'type': 'websocket', 'http_version': '1.1', 'scheme': 'ws', 'server': ('127.0.0.1', 8000), 'client': ('127.0.0.1', 45208), 'root_path': '', 'path': '/', 'raw_path': '/', 'query_string': b'', 'headers': '<...>', 'subprotocols': []}
TRACE:    127.0.0.1:45208 - ASGI [2] Receive {'type': 'websocket.connect'}
[1]    18661 killed     uvicorn --ws wsproto toto:app --log-level=trace
```
then in another terminal
```
➜  websocat git:(websocat_2) ✗ target/release/websocat ws://127.0.0.1:8000
websocat: WebSocketError: WebSocket protocol error
websocat: error running
```
and before the 10s in a 3rd one:
```
➜  websocat git:(websocat_2) ✗ kill -9 18661
```
@euri10 please use `kill` (same as `kill -15`), it handled by uvicorn for graceful shutdown.
ok sorry I thought kill was sigkill, I reproduced with sigterm and tried a fix but not very happy with it
this doenst happen with the other ws implementation though
on a side note thanks for websocat, it rocks
Does this relate at all to https://github.com/encode/uvicorn/issues/244? That one is an error in Uvicorn that occurs when application-level code calls `send({'type': 'websocket.close'})` during auth.

(My project has logged this stack trace 365,250 times, so I'm making some noise :).)
This didn't get fixe by #704 — I was able to reproduce against this sample app:

```python
import asyncio


async def app(scope, receive, send):
    assert scope["type"] == "websocket"
    event = await receive()
    assert event["type"] == "websocket.connect"

    print("Go ahead, stop the server...")
    await asyncio.sleep(10)
    print("Too late!")

    await send({"type": "websocket.accept"})
```

```python
>>> import websockets
>>> await websockets.connect("ws://localhost:8000")
websockets.exceptions.InvalidMessage: did not receive a valid HTTP response
```

After connecting the client I go ahead and shut down the server with `Ctrl+C`, reproducing the error:

```console
$ uvicorn debug.app:app --ws wsproto
INFO:     Started server process [91950]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
Go ahead, stop the server...
^CINFO:     Shutting down
Traceback (most recent call last):
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/bin/uvicorn", line 33, in <module>
    sys.exit(load_entry_point('uvicorn', 'console_scripts', 'uvicorn')())
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/main.py", line 362, in main
    run(**kwargs)
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/main.py", line 386, in run
    server.run()
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/server.py", line 48, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/server.py", line 69, in serve
    await self.shutdown(sockets=sockets)
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/server.py", line 221, in shutdown
    connection.shutdown()
  File "/Users/florimond/Developer/python-projects/uvicorn/uvicorn/protocols/websockets/wsproto_impl.py", line 115, in shutdown
    output = self.conn.send(wsproto.events.CloseConnection(code=1012))
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/wsproto/__init__.py", line 61, in send
    data += self.handshake.send(event)
  File "/Users/florimond/Developer/python-projects/uvicorn/venv/lib/python3.9/site-packages/wsproto/handshake.py", line 100, in send
    raise LocalProtocolError(
wsproto.utilities.LocalProtocolError: Event CloseConnection(code=1012, reason=None) cannot be sent during the handshake
```

Resolving patch:
diff --git a/uvicorn/protocols/websockets/websockets_impl.py b/uvicorn/protocols/websockets/websockets_impl.py
--- a/uvicorn/protocols/websockets/websockets_impl.py
+++ b/uvicorn/protocols/websockets/websockets_impl.py
@@ -345,6 +345,8 @@ async def asgi_receive(
             data = await self.recv()
         except ConnectionClosed as exc:
             self.closed_event.set()
+            if self.ws_server.closing:
+                return {"type": "websocket.disconnect", "code": 1012}
             return {"type": "websocket.disconnect", "code": exc.code}
 
         msg: WebSocketReceiveEvent = {  # type: ignore[typeddict-item]
diff --git a/uvicorn/protocols/websockets/wsproto_impl.py b/uvicorn/protocols/websockets/wsproto_impl.py
--- a/uvicorn/protocols/websockets/wsproto_impl.py
+++ b/uvicorn/protocols/websockets/wsproto_impl.py
@@ -125,9 +125,12 @@ def resume_writing(self):
         self.writable.set()
 
     def shutdown(self):
-        self.queue.put_nowait({"type": "websocket.disconnect", "code": 1012})
-        output = self.conn.send(wsproto.events.CloseConnection(code=1012))
-        self.transport.write(output)
+        if self.handshake_complete:
+            self.queue.put_nowait({"type": "websocket.disconnect", "code": 1012})
+            output = self.conn.send(wsproto.events.CloseConnection(code=1012))
+            self.transport.write(output)
+        else:
+            self.send_500_response()
         self.transport.close()
 
     def on_task_complete(self, task):
@@ -222,9 +225,8 @@ def send_500_response(self):
     async def run_asgi(self):
         try:
             result = await self.app(self.scope, self.receive, self.send)
-        except BaseException as exc:
-            msg = "Exception in ASGI application\n"
-            self.logger.error(msg, exc_info=exc)
+        except BaseException:
+            self.logger.exception("Exception in ASGI application\n")
             if not self.handshake_complete:
                 self.send_500_response()
             self.transport.close()
@@ -257,14 +259,15 @@ async def send(self, message):
                 extensions = []
                 if self.config.ws_per_message_deflate:
                     extensions.append(PerMessageDeflate())
-                output = self.conn.send(
-                    wsproto.events.AcceptConnection(
-                        subprotocol=subprotocol,
-                        extensions=extensions,
-                        extra_headers=extra_headers,
+                if not self.transport.is_closing():
+                    output = self.conn.send(
+                        wsproto.events.AcceptConnection(
+                            subprotocol=subprotocol,
+                            extensions=extensions,
+                            extra_headers=extra_headers,
+                        )
                     )
-                )
-                self.transport.write(output)
+                    self.transport.write(output)
 
             elif message_type == "websocket.close":
                 self.queue.put_nowait({"type": "websocket.disconnect", "code": None})


Test patch:
diff --git a/tests/protocols/test_websocket.py b/tests/protocols/test_websocket.py
--- a/tests/protocols/test_websocket.py
+++ b/tests/protocols/test_websocket.py
@@ -528,7 +528,6 @@ async def app(scope, receive, send):
         while True:
             message = await receive()
             if message["type"] == "websocket.connect":
-                print("accepted")
                 await send({"type": "websocket.accept"})
             elif message["type"] == "websocket.disconnect":
                 break
@@ -551,6 +550,66 @@ async def app(scope, receive, send):
     assert got_disconnect_event_before_shutdown is True
 
 
+@pytest.mark.anyio
+@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
+@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
+async def test_not_accept_on_connection_lost(ws_protocol_cls, http_protocol_cls):
+    send_accept_task = asyncio.Event()
+
+    async def app(scope, receive, send):
+        while True:
+            message = await receive()
+            if message["type"] == "websocket.connect":
+                await send_accept_task.wait()
+                await send({"type": "websocket.accept"})
+            elif message["type"] == "websocket.disconnect":
+                break
+
+    async def websocket_session(uri):
+        async with websockets.client.connect(uri):
+            while True:
+                await asyncio.sleep(0.1)
+
+    config = Config(app=app, ws=ws_protocol_cls, http=http_protocol_cls, lifespan="off")
+    async with run_server(config):
+        task = asyncio.create_task(websocket_session("ws://127.0.0.1:8000"))
+        await asyncio.sleep(0.1)
+        task.cancel()
+        send_accept_task.set()
+
+
+@pytest.mark.anyio
+@pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
+@pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)
+async def test_send_close_on_server_shutdown(ws_protocol_cls, http_protocol_cls):
+    disconnect_message = {}
+
+    async def app(scope, receive, send):
+        nonlocal disconnect_message
+        while True:
+            message = await receive()
+            if message["type"] == "websocket.connect":
+                await send({"type": "websocket.accept"})
+            elif message["type"] == "websocket.disconnect":
+                disconnect_message = message
+                break
+
+    async def websocket_session(uri):
+        async with websockets.client.connect(uri):
+            while True:
+                await asyncio.sleep(0.1)
+
+    config = Config(app=app, ws=ws_protocol_cls, http=http_protocol_cls, lifespan="off")
+    async with run_server(config):
+        task = asyncio.create_task(websocket_session("ws://127.0.0.1:8000"))
+        await asyncio.sleep(0.1)
+        disconnect_message_before_shutdown = disconnect_message
+
+    assert disconnect_message_before_shutdown == {}
+    assert disconnect_message == {"type": "websocket.disconnect", "code": 1012}
+    task.cancel()
+
+
 @pytest.mark.anyio
 @pytest.mark.parametrize("ws_protocol_cls", WS_PROTOCOLS)
 @pytest.mark.parametrize("http_protocol_cls", HTTP_PROTOCOLS)