Repository: aio-libs/aiohttp

Issue:
web_ws - Can't close tcp socket when receiving a close message from client.
### Describe the bug

I'm working on a https://github.com/GNS3/gns3-server/issues/2320 for GNS3 that I think [PR7978](https://github.com/aio-libs/aiohttp/pull/7978) lines up with. The GNS3 issue is that when a web socket client sends a close message the server never closes the tcp socket. I tried the patch above but it doesn't address the issue.

High level GNS3 is calling [ws.receive()](https://github.com/GNS3/gns3-server/blob/master/gns3server/handlers/api/controller/notification_handler.py#L33C14-L33C27) at which point aiohttp receive() the close message.

I believe the problem is because ws.receive is setting [self._closing = True](https://github.com/aio-libs/aiohttp/blob/bf4edce5d6c967fa1d6a581b3aaab1bd2f5121cd/aiohttp/web_ws.py#L474) when it reiceves a close message. This then causes self.close() to [return here](https://github.com/aio-libs/aiohttp/blob/bf4edce5d6c967fa1d6a581b3aaab1bd2f5121cd/aiohttp/web_ws.py#L397) which prevents [if msg.type == WSMsgType.CLOSE:](https://github.com/aio-libs/aiohttp/blob/bf4edce5d6c967fa1d6a581b3aaab1bd2f5121cd/aiohttp/web_ws.py#L413) from being reached.

I'm almost thinking [this should be removed](https://github.com/aio-libs/aiohttp/blob/bf4edce5d6c967fa1d6a581b3aaab1bd2f5121cd/aiohttp/web_ws.py#L397) but i'm not sure what the intent of that is so i'm unsure if that is the proper fix.


### To Reproduce

I don't have a great way to reproduce this. I'm currently just using a fully installed GNS3 instance. I can work on making a reproduction now that I understand the issue.

### Expected behavior

aiohttp.web_ws should close the socket when it receives a close message. 

### Logs/tracebacks

```python-traceback
I'll add these later in the week. I wasn't prepared for this.
```


### Python Version

```console
$ python --version
Python 3.10.12
```


### aiohttp Version

```console
$ python -m pip show aiohttp
(gns3-server) root@ubuntu:~# source /usr/share/gns3/gns3-server/bin/activate
(gns3-server) root@ubuntu:~# python3 -m pip show aiohttp
Name: aiohttp
Version: 3.9.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: /usr/share/gns3/gns3-server/lib/python3.10/site-packages
Requires: aiosignal, async-timeout, attrs, frozenlist, multidict, yarl
Required-by: aiohttp-cors, gns3-server
(gns3-server) root@ubuntu:~#
```


### multidict Version

```console
$ python -m pip show multidict
(gns3-server) root@ubuntu:~# python3 -m pip show multidict
Name: multidict
Version: 6.0.5
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/share/gns3/gns3-server/lib/python3.10/site-packages
Requires:
Required-by: aiohttp, yarl
(gns3-server) root@ubuntu:~#
```


### yarl Version

```console
$ python -m pip show yarl
(gns3-server) root@ubuntu:~# python3 -m pip show yarl
Name: yarl
Version: 1.9.4
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /usr/share/gns3/gns3-server/lib/python3.10/site-packages
Requires: idna, multidict
Required-by: aiohttp
(gns3-server) root@ubuntu:~#
```

```
### OS

(gns3-server) root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
(gns3-server) root@ubuntu:~#
```

### Related component

Server

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

EDIT: Update PR request in description.


Discussion:
Not sure what I did to strike through the OS but that is all correct.
It does look a little odd, and seems a little obvious which makes me wonder why it hasn't been noticed before.

Do you think you could write a test in a PR to reproduce it? Then we could also try your suggestion of removing that check and seeing if any other tests break, which would be a good indication if it's the wrong change to make.
I think the code is expecting that the client will close the connection, but we can't always rely on that even if they say they're going to close it, they may not. 
yeah, I forked aiohttp last night. I'll work on a small replication. I did try removing the call. This results in the [next read request](https://github.com/aio-libs/aiohttp/blob/6c3122f4ab2978b67ab28bb0a943404efe8602e6/aiohttp/web_ws.py#L425) in close() failing with timeout error and then the socket does close but with a .. uh.. 1006 message I think.

I also started looking out how to pass message to close() so I could add a condition for the read statement. Didn't get very far but it was kind of late. 
https://datatracker.ietf.org/doc/html/rfc6455#section-7

> Once an endpoint has both sent and received a Close control frame, that endpoint SHOULD _Close the WebSocket Connection_

So, this is what it is trying to achieve when close is initiated by us.

> As such, when a server is instructed to _Close the WebSocket Connection_ it SHOULD initiate a TCP Close immediately, and when a client is instructed to do the same, it SHOULD wait for a TCP Close from the server.

> Except as indicated above or as specified by the application layer (e.g., a script using the WebSocket API), clients SHOULD NOT close the connection.

I think we just need to tweak that code so that the server does not wait to receive a close code (as it has already received one), but it should continue to close the transport. i.e. Rather than removing that check we want to add `self._set_code_close_transport(...)` in there (but, we might need to avoid calling it twice?).
so something like this instead of deleting? I looked through the code and it seems like _close_code is always set but i'm not sure if its safe to assume it is? Default is None.

```python
        if self._closing:
            self._set_code_close_transport(self._close_code)
            return True
```


> but i'm not sure if its safe to assume it is?

Not sure, I'd have to look more closely. If we get the test up first, then we can see how it works.
Boiling this down has been harder then I expected. So far I'm not recreating the issue. More research and foul language will be required. 

Resolving patch:
diff --git a/aiohttp/web_ws.py b/aiohttp/web_ws.py
--- a/aiohttp/web_ws.py
+++ b/aiohttp/web_ws.py
@@ -417,6 +417,7 @@ async def close(
             return True
 
         if self._closing:
+            self._close_transport()
             return True
 
         reader = self._reader
@@ -440,9 +441,18 @@ async def close(
         self._exception = asyncio.TimeoutError()
         return True
 
+    def _set_closing(self, code: WSCloseCode) -> None:
+        """Set the close code and mark the connection as closing."""
+        self._closing = True
+        self._close_code = code
+
     def _set_code_close_transport(self, code: WSCloseCode) -> None:
         """Set the close code and close the transport."""
         self._close_code = code
+        self._close_transport()
+
+    def _close_transport(self) -> None:
+        """Close the transport."""
         if self._req is not None and self._req.transport is not None:
             self._req.transport.close()
 
@@ -487,14 +497,12 @@ async def receive(self, timeout: Optional[float] = None) -> WSMessage:
                 return WSMessage(WSMsgType.ERROR, exc, None)
             except Exception as exc:
                 self._exception = exc
-                self._closing = True
-                self._close_code = WSCloseCode.ABNORMAL_CLOSURE
+                self._set_closing(WSCloseCode.ABNORMAL_CLOSURE)
                 await self.close()
                 return WSMessage(WSMsgType.ERROR, exc, None)
 
             if msg.type == WSMsgType.CLOSE:
-                self._closing = True
-                self._close_code = msg.data
+                self._set_closing(msg.data)
                 # Could be closed while awaiting reader.
                 if not self._closed and self._autoclose:  # type: ignore[redundant-expr]
                     # The client is likely going to close the
@@ -503,7 +511,7 @@ async def receive(self, timeout: Optional[float] = None) -> WSMessage:
                     # likely result writing to a broken pipe.
                     await self.close(drain=False)
             elif msg.type == WSMsgType.CLOSING:
-                self._closing = True
+                self._set_closing(WSCloseCode.OK)
             elif msg.type == WSMsgType.PING and self._autoping:
                 await self.pong(msg.data)
                 continue
@@ -547,5 +555,8 @@ async def __anext__(self) -> WSMessage:
         return msg
 
     def _cancel(self, exc: BaseException) -> None:
+        # web_protocol calls this from connection_lost
+        # or when the server is shutting down.
+        self._closing = True
         if self._reader is not None:
             set_exception(self._reader, exc)


Test patch:
diff --git a/tests/test_web_websocket.py b/tests/test_web_websocket.py
--- a/tests/test_web_websocket.py
+++ b/tests/test_web_websocket.py
@@ -12,7 +12,7 @@
 from aiohttp.streams import EofStream
 from aiohttp.test_utils import make_mocked_coro, make_mocked_request
 from aiohttp.web import HTTPBadRequest, WebSocketResponse
-from aiohttp.web_ws import WS_CLOSED_MESSAGE, WebSocketReady
+from aiohttp.web_ws import WS_CLOSED_MESSAGE, WebSocketReady, WSMessage
 
 
 @pytest.fixture
@@ -344,6 +344,93 @@ async def test_receive_eofstream_in_reader(make_request: Any, loop: Any) -> None
     assert ws.closed
 
 
+async def test_receive_exception_in_reader(make_request: Any, loop: Any) -> None:
+    req = make_request("GET", "/")
+    ws = WebSocketResponse()
+    await ws.prepare(req)
+
+    ws._reader = mock.Mock()
+    exc = Exception()
+    res = loop.create_future()
+    res.set_exception(exc)
+    ws._reader.read = make_mocked_coro(res)
+    ws._payload_writer.drain = mock.Mock()
+    ws._payload_writer.drain.return_value = loop.create_future()
+    ws._payload_writer.drain.return_value.set_result(True)
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.ERROR
+    assert ws.closed
+    assert len(ws._req.transport.close.mock_calls) == 1
+
+
+async def test_receive_close_but_left_open(make_request: Any, loop: Any) -> None:
+    req = make_request("GET", "/")
+    ws = WebSocketResponse()
+    await ws.prepare(req)
+    close_message = WSMessage(WSMsgType.CLOSE, 1000, "close")
+
+    ws._reader = mock.Mock()
+    ws._reader.read = mock.AsyncMock(return_value=close_message)
+    ws._payload_writer.drain = mock.Mock()
+    ws._payload_writer.drain.return_value = loop.create_future()
+    ws._payload_writer.drain.return_value.set_result(True)
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSE
+    assert ws.closed
+    assert len(ws._req.transport.close.mock_calls) == 1
+
+
+async def test_receive_closing(make_request: Any, loop: Any) -> None:
+    req = make_request("GET", "/")
+    ws = WebSocketResponse()
+    await ws.prepare(req)
+    closing_message = WSMessage(WSMsgType.CLOSING, 1000, "closing")
+
+    ws._reader = mock.Mock()
+    read_mock = mock.AsyncMock(return_value=closing_message)
+    ws._reader.read = read_mock
+    ws._payload_writer.drain = mock.Mock()
+    ws._payload_writer.drain.return_value = loop.create_future()
+    ws._payload_writer.drain.return_value.set_result(True)
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSING
+    assert not ws.closed
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSING
+    assert not ws.closed
+
+    ws._cancel(ConnectionResetError("Connection lost"))
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSING
+
+
+async def test_close_after_closing(make_request: Any, loop: Any) -> None:
+    req = make_request("GET", "/")
+    ws = WebSocketResponse()
+    await ws.prepare(req)
+    closing_message = WSMessage(WSMsgType.CLOSING, 1000, "closing")
+
+    ws._reader = mock.Mock()
+    ws._reader.read = mock.AsyncMock(return_value=closing_message)
+    ws._payload_writer.drain = mock.Mock()
+    ws._payload_writer.drain.return_value = loop.create_future()
+    ws._payload_writer.drain.return_value.set_result(True)
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSING
+    assert not ws.closed
+    assert len(ws._req.transport.close.mock_calls) == 0
+
+    await ws.close()
+    assert ws.closed
+    assert len(ws._req.transport.close.mock_calls) == 1
+
+
 async def test_receive_timeouterror(make_request: Any, loop: Any) -> None:
     req = make_request("GET", "/")
     ws = WebSocketResponse()
diff --git a/tests/test_web_websocket_functional.py b/tests/test_web_websocket_functional.py
--- a/tests/test_web_websocket_functional.py
+++ b/tests/test_web_websocket_functional.py
@@ -2,7 +2,7 @@
 # HTTP websocket server functional tests
 
 import asyncio
-from typing import Any
+from typing import Any, Optional
 
 import pytest
 
@@ -258,7 +258,7 @@ async def handler(request):
     assert "reply" == (await ws.receive_str())
 
     # The server closes here.  Then the client sends bogus messages with an
-    # internval shorter than server-side close timeout, to make the server
+    # interval shorter than server-side close timeout, to make the server
     # hanging indefinitely.
     await asyncio.sleep(0.08)
     msg = await ws._reader.read()
@@ -309,6 +309,35 @@ async def handler(request):
     assert msg.type == WSMsgType.CLOSED
 
 
+async def test_close_op_code_from_client(loop: Any, aiohttp_client: Any) -> None:
+    srv_ws: Optional[web.WebSocketResponse] = None
+
+    async def handler(request):
+        nonlocal srv_ws
+        ws = srv_ws = web.WebSocketResponse(protocols=("foo", "bar"))
+        await ws.prepare(request)
+
+        msg = await ws.receive()
+        assert msg.type == WSMsgType.CLOSE
+        await asyncio.sleep(0)
+        return ws
+
+    app = web.Application()
+    app.router.add_get("/", handler)
+    client = await aiohttp_client(app)
+
+    ws: web.WebSocketResponse = await client.ws_connect("/", protocols=("eggs", "bar"))
+
+    await ws._writer._send_frame(b"", WSMsgType.CLOSE)
+
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSE
+
+    await asyncio.sleep(0)
+    msg = await ws.receive()
+    assert msg.type == WSMsgType.CLOSED
+
+
 async def test_auto_pong_with_closing_by_peer(loop: Any, aiohttp_client: Any) -> None:
     closed = loop.create_future()