Repository: mitmproxy/mitmproxy

Issue:
'socket.send() raised exception.' on stderr
#### Problem Description
In a long-running session of mitmdump that captures traffic of all surfing on an iPad, standard error logs a bunch of
```
socket.send() raised exception.
```
Nothing else is logged. Subjectively every now and then this is logged a browser tab stalls on the iPad. Looking at where the message comes from (`asyncio/*_events.py`) I believe this is the result of a socket already being dead when the HTTP/2 PING is sent periodically. 
It looks to me like asyncio is catching this exception but mitmproxy is not made aware that the server connection is actually dead. Trying to use the dead connection often enough results in this log message:

```
            if self._conn_lost >= constants.LOG_THRESHOLD_FOR_CONNLOST_WRITES:
                logger.warning('socket.send() raised exception.')
```
I do not see any method to query the status of `self._conn_lost` which would help fixing this.

#### System Information
Mitmproxy: 9.0.0.dev (+151, commit 46f01ad)
Python:    3.9.7
OpenSSL:   OpenSSL 1.1.1n  15 Mar 2022
Platform:  Linux-4.18.0-383.el8.x86_64-x86_64-with-glibc2.28



Discussion:
I did some googling and issues like this show up in the bug tracker of various projects. I found this fix for the issue in one project:
https://github.com/ska-sa/aiokatcp/commit/200189687148ab63b79081d8cfd6c2c92277895c
I did a small modification to `mitmproxy/proxy/server.py` to see if this approach would be viable:
```
                 elif isinstance(command, commands.SendData):
                     writer = self.transports[command.connection].writer
                     assert writer
+                    if writer.transport.is_closing():
+                        self.log(f"XXX Write to closed connection {command.connection}")
                     writer.write(command.data)
                 elif isinstance(command, commands.CloseConnection):
                     self.close_connection(command.connection, command.half_close)
...
             try:
                 writer = self.transports[connection].writer
                 assert writer
+                if writer.transport.is_closing():
+                    self.log(f"XXX Eof for closed connection {connection}")
                 writer.write_eof()
             except OSError:
```
I did get the above log entry for `writer.write` pretty quickly. The log entries for one connection with the above log entry looks like this:
```
192.168.0.10:49775: client connect
192.168.0.10:49775: server connect x.x.x.x:443
192.168.0.10:49775: half-closing Server(x.x.x.x:443, state=open, src_port=49486)
192.168.0.10:49775: XXX Write to closed connection Client(192.168.0.10:49775, state=can_write)
192.168.0.10:49775: client disconnect
192.168.0.10:49775: closing transports...
192.168.0.10:49775: server disconnect x.x.x.x:443
192.168.0.10:49775: transports closed!
```
Any suggestions if this test before the write is a good idea and whether to just not write if it is true or if this should trigger a flow kill?

Thanks again for the very nice report even though information is understandably limited at this time. :) Can you verify that this does not occur when disabling HTTP/2 keepalives?

Assuming it's that, we can do multiple things here:

1. Add the check you suggested. That seems reasonable to me.
2. Check if the connection state is still open when receiving a wakeup, and only then sending a ping. This seems equally reasonable, maybe even more clean.
3. Add a new `CancelWakeup` command to cancel the wakup requests when receiving a ConnectionClosed. Mentioned here for completeness, but this approach seems mostly overengineered.
I let this run for more than a day now and here is what I got:
- 150 x write() to closed client connection
- 3 x write() to closed server connection
- 1 x write_eol() to closed client connection

**All connections** that logged the messages were pass-through connections. This is probably also the reason why few other will see the original error message. I have quite a list of hosts and domains for which I just want pass-through, because I don't care (ads, tracking) or because of certificate pinning or TLS fingerprinting. Anybody who wants to reproduce the error may want to do so with a script like this:
```
def tls_clienthello(data: tls.ClientHelloData):
    data.ignore_connection = True
```
So given that no higher protocol is involved it may be the best to just add the checks in the two locations in `server.py`, optionally only if the connection is pass-through, but I think adding it for all cases is safer. What I am not sure is that if `is_closing()` is true then the proxy is guaranteed to otherwise notice the closed connection and terminate the flow and both ends. It looks to me that way and just not doing the `write()` or `write_eol()` should be enough.
I didn't find any other occurrences of `write()` or `write_eol()` but maybe I wasn't thorough enough.
Followup: I got one 'socket.send() raised exception.' in the error log and there was one matching client connection that logged five times `XXX Write ...`. `LOG_THRESHOLD_FOR_CONNLOST_WRITES` is defined to be 5 in `asyncio/constants.py` so I am pretty sure that connection produced the error log entry.

Resolving patch:
diff --git a/mitmproxy/net/udp.py b/mitmproxy/net/udp.py
--- a/mitmproxy/net/udp.py
+++ b/mitmproxy/net/udp.py
@@ -175,8 +175,12 @@ def __init__(
         """
         self._transport = transport
         self._remote_addr = remote_addr
-        self._reader = reader
-        self._closed = asyncio.Event() if reader is not None else None
+        if reader is not None:
+            self._reader = reader
+            self._closed = asyncio.Event()
+        else:
+            self._reader = None
+            self._closed = None
 
     @property
     def _protocol(self) -> DrainableDatagramProtocol | udp_wireguard.WireGuardDatagramTransport:
@@ -199,9 +203,15 @@ def close(self) -> None:
             self._transport.close()
         else:
             self._closed.set()
-        if self._reader is not None:
+            assert self._reader
             self._reader.feed_eof()
 
+    def is_closing(self) -> bool:
+        if self._closed is None:
+            return self._transport.is_closing()
+        else:
+            return self._closed.is_set()
+
     async def wait_closed(self) -> None:
         if self._closed is None:
             await self._protocol.wait_closed()
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -367,7 +367,8 @@ def server_event(self, event: events.Event) -> None:
                 elif isinstance(command, commands.SendData):
                     writer = self.transports[command.connection].writer
                     assert writer
-                    writer.write(command.data)
+                    if not writer.is_closing():
+                        writer.write(command.data)
                 elif isinstance(command, commands.CloseConnection):
                     self.close_connection(command.connection, command.half_close)
                 elif isinstance(command, commands.StartHook):
@@ -393,7 +394,8 @@ def close_connection(
             try:
                 writer = self.transports[connection].writer
                 assert writer
-                writer.write_eof()
+                if not writer.is_closing():
+                    writer.write_eof()
             except OSError:
                 # if we can't write to the socket anymore we presume it completely dead.
                 connection.state = ConnectionState.CLOSED


Test patch:
diff --git a/test/mitmproxy/net/test_udp.py b/test/mitmproxy/net/test_udp.py
--- a/test/mitmproxy/net/test_udp.py
+++ b/test/mitmproxy/net/test_udp.py
@@ -45,11 +45,16 @@ def handle_datagram(
     server.resume_writing()
     await server.drain()
 
+    assert not client_writer.is_closing()
+    assert not server_writer.is_closing()
+
     assert await client_reader.read(MAX_DATAGRAM_SIZE) == b"msg4"
     client_writer.close()
+    assert client_writer.is_closing()
     await client_writer.wait_closed()
 
     server_writer.close()
+    assert server_writer.is_closing()
     await server_writer.wait_closed()
 
     server.close()