Repository: mitmproxy/mitmproxy

Issue:
Client sometimes using dead connection
#### Problem Description
I have noticed that web browsers sometimes stall loading images or pages. I get it both on an iPad with Safari and a Fedora with Firefox. It especially happens with pass-through connections but also without.

I think I have tracked down the bug to _mitmproxy/proxy/server.py_ where in ConnectionHandler.handle_connection() there is the following loop:

```
        while True:
            try:
                data = await reader.read(65535)
                if not data:
                    raise OSError("Connection closed by peer.")
            except OSError:
                break
            except asyncio.CancelledError as e:
                cancelled = e
                break
            else:
                self.server_event(events.DataReceived(connection, data))
                for transport in self.transports.values():
                    if transport.writer is not None:
                        await transport.writer.drain()
```

The final drain() can actually throw and if it does all the connection management following the loop is not properly done, leaving the connection in a strange state. My suggested fix that made it work without further hick-ups for me is this:

```
        reader = self.transports[connection].reader
        writer = self.transports[connection].writer
        assert reader
        assert writer
        while True:
...
            else:
                self.server_event(events.DataReceived(connection, data))
                for transport in self.transports.values():
                    if transport.writer is not None:
                        try:
                            await transport.writer.drain()
                        except OSError as e:
                            if transport.writer == writer:
                                cancelled = e
                                break
                else:
                    continue
                break
```

#### Steps to reproduce the behavior:
1. Start mitmdump as transparent proxy with '--ignore-hosts .com'
2. Go to apple.com
3. Quickly click on links, attempt to abort a page load with a re-load of the same page

#### System Information
Mitmproxy: 8.0.0.dev (+200, commit ef8c88d)
Python:    3.9.7
OpenSSL:   OpenSSL 1.1.1m  14 Dec 2021
Platform:  Linux-4.18.0-348.2.1.el8_5.x86_64-x86_64-with-glibc2.28



Discussion:
Honestly I was dreading to even open this issue after reading "Client sometimes ..." in the title, but turns out there is a positive surprise in here. 😁 This is super super interesting, thank you very much! 🍰

1. I can't trigger `.drain()` to raise on my system, but that might be OS-specific. Out of interest, would you mind posting the full OSError `repr()` here?
2. How did you approach finding this? Any tips/tricks you can share?
3. It may be useful to just cancel any handler (not just the current one) if we cannot write. Could you verify that the following patch also works for you?

```diff
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -227,7 +227,11 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
                 self.server_event(events.DataReceived(connection, data))
                 for transport in self.transports.values():
                     if transport.writer is not None:
-                        await transport.writer.drain()
+                        try:
+                            await transport.writer.drain()
+                        except OSError as e:
+                            if transport.handler is not None:
+                                asyncio_utils.cancel_task(transport.handler, f"cannot drain writer: {e}")
```

Thanks for looking into this! Apart from this issue the new sansio design rocks! I can even scroll around in Google Maps and the CPU usage of the proxy machine remains in the one digit range. With V4 this was laggy with >100% load. Cool.

>     1. I can't trigger `.drain()` to raise on my system, but that might be OS-specific. Out of interest, would you mind posting the full OSError `repr()` here?

When I log a `repr(e)` when drain() raises an exception then the log entry reads:

```ConnectionResetError('Connection lost')```

Out of curiosity I also added an `except asyncio.CancelledError as e:` after `drain()` and that also sometimes catches an exception (meaning my fix wasn't complete...):

```CancelledError('client disconnected')```

I wonder what it takes to reproduce this on your side. I run mitmdump in transparent mode (easy for me to add and remove my machines), from the git repo directly on a CentOS 8 Stream on a laptop. Basically it provides a gateway from my WIFI network to the ADSL router.

>     2. How did you approach finding this? Any tips/tricks you can share?

The hard way, with tcpdump/wireshark, `--set proxy_debug=true` and putting `self.log()` all over the place in _server.py_. With tcpdump I noticed that when a page load stalled that I could still tap on links and that the browser then still sent data to the proxy after that transport has been closed (HTTP/2 does that) so it had to be the stream management at a low level. I also noticed that for the connection that stalled, the log entry `client disconnect` in `ConnectionHandler()` after `watch.cancel()` was missing for that connection. After narrowing down this far I started inserting `self.log()` until I found that the one after `drain()` sometimes was not logging.

What also helped me on the way was to add this to my script:

```
def error(flow: http.HTTPFlow):
    if flow.error is not None:
        ctx.log("FLOWERROR " + str(flow.error))
```

>     3. It may be useful to just cancel any handler (not just the current one) if we cannot write. Could you verify that the following patch also works for you?
> 
> 
> ```diff
> --- a/mitmproxy/proxy/server.py
> +++ b/mitmproxy/proxy/server.py
> @@ -227,7 +227,11 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
>                  self.server_event(events.DataReceived(connection, data))
>                  for transport in self.transports.values():
>                      if transport.writer is not None:
> -                        await transport.writer.drain()
> +                        try:
> +                            await transport.writer.drain()
> +                        except OSError as e:
> +                            if transport.handler is not None:
> +                                asyncio_utils.cancel_task(transport.handler, f"cannot drain writer: {e}")
> ```

I just tried this variant a bit. It made it worse for pass-through connections (apple.com on my iPad). After the first or second click already the connection stalls consistently. I guess the approach is too aggressive.

For normal HTTP/2 connections it worked OK but I did get a _502 Bad Gateway_ when reading heise.de after a few pages. The `error()` in my script logged:

```
FLOWERROR HTTP/2 protocol error: Invalid input ConnectionInputs.RECV_HEADERS in state ConnectionState.CLOSED
```
I do not fully understand the code in _server.py_, for instance, why there is a need to drain all writers, not just the one from this connection. I am not strongly advocating my solution, so far it is just the most solid for me, even if I apparently missed the `CancelError`s. If you have more suggestions then I will try them out, this is dead easy for me to reproduce. Which still worries me, apparenly nobody else had this issue so far... :)

> Thanks for looking into this! Apart from this issue the new sansio design rocks! I can even scroll around in Google Maps and the CPU usage of the proxy machine remains in the one digit range. With V4 this was laggy with >100% load. Cool.

Excellent. Very happy to hear that. 😃 

> I wonder what it takes to reproduce this on your side. I run mitmdump in transparent mode (easy for me to add and remove my machines), from the git repo directly on a CentOS 8 Stream on a laptop.

That's helpful, thanks. My main driver is a Windows machine in regular mode, so that's quite a different stack.

> The hard way, with tcpdump/wireshark, `--set proxy_debug=true` and putting `self.log()` all over the place in _server.py_. With tcpdump I noticed that when a page load stalled that I could still tap on links and that the browser then still sent data to the proxy after that transport has been closed (HTTP/2 does that) so it had to be the stream management at a low level. I also noticed that for the connection that stalled, the log entry `client disconnect` in `ConnectionHandler()` after `watch.cancel()` was missing for that connection. After narrowing down this far I started inserting `self.log()` until I found that the one after `drain()` sometimes was not logging.

Very impressive! I'm actually somewhat puzzled which part silently swallows the OSError here, I definitely need to figure out why exceptions are ignored and correct that. But let's focus on the initial issue here.

> I just tried this variant a bit. It made it worse for pass-through connections (apple.com on my iPad). After the first or second click already the connection stalls consistently. I guess the approach is too aggressive.

This confused me for a bit, but I think I have a working theory now: When a coroutine/task is cancelled, it continues to run normally until the next `await` statement, which throws a `CancelledError`. What's may be happening here is that we first cancel our very own connection handler task we're currently in, and after that then `drain()` another transport. As we have already cancelled our own coroutine this `await drain()` immediately throws a `CancelledError`, which as you correctly pointed out both of us did not catch before! The problem just happens to be much more visible in my patch.

> I do not fully understand the code in server.py, for instance, why there is a need to drain all writers

We do that to create backpressure. Assume you have a very fast connection to your laptop, a slower connection upstream, and you want to upload a large file. mitmproxy would very quickly read everything from your local device and then have an immense send buffer for the upstream connection. So while mitmproxy is slowly shoveling bits upstream, your client is done uploading for minutes already, hits a timeout, and aborts the connection[^1]. 
Draining the upstream connection aftere each read means we wait for space in the upstream send buffer to be available before we continue reading from the downstream socket again. In other words, while we drain the upstream send buffer, our own downstream receive buffer fills up and the OS' TCP implementation signals to downstream that they need to slow down. Does that make sense?

> If you have more suggestions then I will try them out, this is dead easy for me to reproduce. Which still worries me, apparenly nobody else had this issue so far... :)

Here's a revised patch that does catch the CancelledError. Could you test that and see how it works?

```diff
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -223,11 +223,21 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
             except asyncio.CancelledError as e:
                 cancelled = e
                 break
-            else:
-                self.server_event(events.DataReceived(connection, data))
-                for transport in self.transports.values():
-                    if transport.writer is not None:
+
+            self.server_event(events.DataReceived(connection, data))
+
+            # Drain all writers to create some backpressure. We won't continue reading until there's space available in
+            # our write buffers, so our own read buffers run full and the TCP recv stream is throttled.
+            for transport in self.transports.values():
+                if transport.writer is not None:
+                    try:
                         await transport.writer.drain()
+                    except OSError as e:
+                        if transport.handler is not None:
+                            asyncio_utils.cancel_task(transport.handler, f"Error sending data: {e}")
+                    except asyncio.CancelledError as e:
+                        cancelled = e
+                        break
```

[^1]: I'm assuming a streamed request body (or plain TCP) here. If mitmproxy is told not to stream this issue still exists.
I'll reply in full and run extensive tests in the evening. One question though on the new patch that you may be able to answer before that:

> Here's a revised patch that does catch the CancelledError. Could you test that and see how it works?
> 
> ```diff
> --- a/mitmproxy/proxy/server.py
> +++ b/mitmproxy/proxy/server.py
> @@ -223,11 +223,21 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
>              except asyncio.CancelledError as e:
>                  cancelled = e
>                  break
> -            else:
> -                self.server_event(events.DataReceived(connection, data))
> -                for transport in self.transports.values():
> -                    if transport.writer is not None:
> +
> +            self.server_event(events.DataReceived(connection, data))
> +
> +            # Drain all writers to create some backpressure. We won't continue reading until there's space available in
> +            # our write buffers, so our own read buffers run full and the TCP recv stream is throttled.
> +            for transport in self.transports.values():
> +                if transport.writer is not None:
> +                    try:
>                          await transport.writer.drain()
> +                    except OSError as e:
> +                        if transport.handler is not None:
> +                            asyncio_utils.cancel_task(transport.handler, f"Error sending data: {e}")
> +                    except asyncio.CancelledError as e:
> +                        cancelled = e
> +                        break
> ```

That last break, it will only exit the inner loop which does not make sense to me. I would either ignore the exception and continue draining the rest of the writers as well or exit the outer loop with an 'else - continue - break' construct similar to what I used in my patch. Since you set cancelled I guess you want the later.

Yes, good catch. Let's break out entirely. :)
Here's a revised revised patch that should break out:
```diff
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
index 4b26b23c6..7af7f09d6 100644
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -223,11 +223,14 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
             except asyncio.CancelledError as e:
                 cancelled = e
                 break
-            else:
-                self.server_event(events.DataReceived(connection, data))
-                for transport in self.transports.values():
-                    if transport.writer is not None:
-                        await transport.writer.drain()
+
+            self.server_event(events.DataReceived(connection, data))
+
+            try:
+                await self.drain_writers()
+            except asyncio.CancelledError as e:
+                cancelled = e
+                break

         if cancelled is None:
             connection.state &= ~ConnectionState.CAN_READ
@@ -253,6 +256,19 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
         if cancelled:
             raise cancelled

+    async def drain_writers(self):
+        """
+        Drain all writers to create some backpressure. We won't continue reading until there's space available in our
+        write buffers, so if we cannot write fast enough our own read buffers run full and the TCP recv stream is throttled.
+        """
+        for transport in self.transports.values():
+            if transport.writer is not None:
+                try:
+                    await transport.writer.drain()
+                except OSError as e:
+                    if transport.handler is not None:
+                        asyncio_utils.cancel_task(transport.handler, f"Error sending data: {e}")
+
     async def on_timeout(self) -> None:
         self.log(f"Closing connection due to inactivity: {self.client}")
         handler = self.transports[self.client].handler
```
> Here's a revised revised patch that should break out:
> 
> ```diff
> diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
> index 4b26b23c6..7af7f09d6 100644
> --- a/mitmproxy/proxy/server.py
> +++ b/mitmproxy/proxy/server.py
> @@ -223,11 +223,14 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
>              except asyncio.CancelledError as e:
>                  cancelled = e
>                  break
> -            else:
> -                self.server_event(events.DataReceived(connection, data))
> -                for transport in self.transports.values():
> -                    if transport.writer is not None:
> -                        await transport.writer.drain()
> +
> +            self.server_event(events.DataReceived(connection, data))
> +
> +            try:
> +                await self.drain_writers()
> +            except asyncio.CancelledError as e:
> +                cancelled = e
> +                break
> 
>          if cancelled is None:
>              connection.state &= ~ConnectionState.CAN_READ
> @@ -253,6 +256,19 @@ class ConnectionHandler(metaclass=abc.ABCMeta):
>          if cancelled:
>              raise cancelled
> 
> +    async def drain_writers(self):
> +        """
> +        Drain all writers to create some backpressure. We won't continue reading until there's space available in our
> +        write buffers, so if we cannot write fast enough our own read buffers run full and the TCP recv stream is throttled.
> +        """
> +        for transport in self.transports.values():
> +            if transport.writer is not None:
> +                try:
> +                    await transport.writer.drain()
> +                except OSError as e:
> +                    if transport.handler is not None:
> +                        asyncio_utils.cancel_task(transport.handler, f"Error sending data: {e}")
> +
>      async def on_timeout(self) -> None:
>          self.log(f"Closing connection due to inactivity: {self.client}")
>          handler = self.transports[self.client].handler
> ```

I have applied this patch and tried various websites, Youtube, Google Maps, etc. and there was not a single delayed load or missing image during my hours of surfing. In the logs I did not see any suspicious server-side errors. So I think the above fix works great.

Thanks for the solution! I keep the patch in my checkout until it is also committed to the main branch. But this issue can be closed I think.

Resolving patch:
diff --git a/mitmproxy/proxy/server.py b/mitmproxy/proxy/server.py
--- a/mitmproxy/proxy/server.py
+++ b/mitmproxy/proxy/server.py
@@ -223,11 +223,14 @@ async def handle_connection(self, connection: Connection) -> None:
             except asyncio.CancelledError as e:
                 cancelled = e
                 break
-            else:
-                self.server_event(events.DataReceived(connection, data))
-                for transport in self.transports.values():
-                    if transport.writer is not None:
-                        await transport.writer.drain()
+
+            self.server_event(events.DataReceived(connection, data))
+
+            try:
+                await self.drain_writers()
+            except asyncio.CancelledError as e:
+                cancelled = e
+                break
 
         if cancelled is None:
             connection.state &= ~ConnectionState.CAN_READ
@@ -253,6 +256,19 @@ async def handle_connection(self, connection: Connection) -> None:
         if cancelled:
             raise cancelled
 
+    async def drain_writers(self):
+        """
+        Drain all writers to create some backpressure. We won't continue reading until there's space available in our
+        write buffers, so if we cannot write fast enough our own read buffers run full and the TCP recv stream is throttled.
+        """
+        for transport in self.transports.values():
+            if transport.writer is not None:
+                try:
+                    await transport.writer.drain()
+                except OSError as e:
+                    if transport.handler is not None:
+                        asyncio_utils.cancel_task(transport.handler, f"Error sending data: {e}")
+
     async def on_timeout(self) -> None:
         self.log(f"Closing connection due to inactivity: {self.client}")
         handler = self.transports[self.client].handler


Test patch: