Repository: mitmproxy/mitmproxy

Issue:
Proxyauth doesn't work on HTTPS on IOS
#### Problem Description
I'm trying to make my proxy have basic authentication. I'm running the proxy with the command `mitmdump --proxyauth ac:dc`, It starts on port `8080`. This all works well for cURL `curl -x ac:dc@127.0.0.1:8080  https://httpbin.org`, but when I go on IOS 
![IMG_6834](https://github.com/mitmproxy/mitmproxy/assets/93884113/59e69bce-e419-47eb-b2c3-cf94965c3cf3) (The password is there IOS just hides it in screenshots) 

In my proxy terminal I get:
```
[17:39:17.187][192.168.1.28:59059] client connect
192.168.1.28:59059: CONNECT httpbin.org:443
                 << 407 Proxy Authentication Required 129b
[17:39:17.221][192.168.1.28:59056] client disconnect
```
If I go to http://httpbin.org/ this is what I get:
```
[17:38:11.208][192.168.1.28:59047] server connect httpbin.org:80 (107.23.135.58:80)
192.168.1.28:59047: GET http://httpbin.org/flasgger_static/lib/jquery.min.js
                 << 200 OK 83.6k
192.168.1.28:59045: GET http://httpbin.org/flasgger_static/swagger-ui-bundle.js
                 << 200 OK 1.4m
192.168.1.28:59046: GET http://httpbin.org/flasgger_static/swagger-ui-standalone-preset.js
                 << 200 OK 430k
[17:38:24.734][192.168.1.28:59044] client disconnect
192.168.1.28:59043: GET http://httpbin.org/
                 << 200 OK 9.4k
[17:38:24.772][192.168.1.28:59049] client connect
192.168.1.28:59049: CONNECT fonts.googleapis.com:443
                 << 407 Proxy Authentication Required 129b
```
Which works up until it gets a HTTPS request.

I found issue https://github.com/mitmproxy/mitmproxy/issues/5519 but it went stale and cannot find any answers / other issues related to this.

#### Steps to reproduce the behavior:
1. Setup your proxy with a username and password
2. Go to an IOS device (My version is 16.5) and connect to the proxy with the username/password in the authentication section
3. Examine terminal output

#### System Information
Mitmproxy: 10.1.1 binary
Python:    3.11.5
OpenSSL:   OpenSSL 3.1.3 19 Sep 2023
Platform:  Windows-10-10.0.19044-SP0

On my IPhone:
Model: IPhone 14 Pro
IOS Version: 16.5


Discussion:
Can you run `mitmdump -v --proxyauth ac:dc` please? That should display the headers that are being sent with the CONNECT requests.
> Can you run `mitmdump -v --proxyauth ac:dc` please? That should display the headers that are being sent with the CONNECT requests.

HTTPS:
```
[07:00:09.099][192.168.1.28:57310] client connect
192.168.1.28:57310: CONNECT httpbin.org:443
    Host: httpbin.org
    Proxy-Connection: keep-alive
    Connection: keep-alive
 << 407 Proxy Authentication Required 129b
    Proxy-Authenticate: Basic realm="mitmproxy"
    content-length: 129
[07:00:21.503][192.168.1.28:57306] client disconnect
```
And on an HTTP request:
```
[07:11:44.804][192.168.1.28:57409] server connect httpbin.org:80 (54.83.187.171:80)
192.168.1.28:57409: GET http://httpbin.org/
    Host: httpbin.org
    Proxy-Connection: keep-alive
    Upgrade-Insecure-Requests: 1
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Mobile/15E148 Safari/604.1
    Accept-Language: en-US,en;q=0.9
    Accept-Encoding: gzip, deflate
    Connection: keep-alive
 << 200 OK 9.4k
    Date: Fri, 27 Oct 2023 11:11:44 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 9593
    Connection: keep-alive
    Server: gunicorn/19.9.0
    Access-Control-Allow-Origin: *
    Access-Control-Allow-Credentials: true
```
I'm running into this too.

In Wireshark, I see the following:

```
CONNECT bag.itunes.apple.com:443 HTTP/1.1
Host: bag.itunes.apple.com
Proxy-Connection: keep-alive
Connection: keep-alive

HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="mitmproxy"
content-length: 129

<html><head><title>407 Proxy Authentication Required</title></head><body><h1>407 Proxy Authentication Required</h1></body></html>CONNECT bag.itunes.apple.com:443 HTTP/1.1
Host: bag.itunes.apple.com
Connection: keep-alive
Proxy-Authorization: Basic dGVzdDp0ZXN0
Proxy-Connection: keep-alive

```

It looks like the 407 is sent correctly from mitmproxy, and then the iOS device retries with the credentials attached. But, for some reason, mitmproxy does not appear to open a connection afterwards (there's no outgoing connection from the proxy, and the incoming connection to the proxy is just held open until Safari on iOS just gives up 30 seconds later).

There's nothing in the log. The same behaviour occurs even if `allow_hosts` is used to ignore hosts like this. Ordinary HTTP requests, which look substantially similar, succeed without issue.

This is *a regression*: mitmproxy 8.0.0 works fine, mitmproxy 9.0.0 does not.
...regression is from 8.0.0 (working) to 8.1.0 (not working). I'll run a bisect to see which commit broke things.
Bad commit is 035b3bf37d9785fef45e81eb9c0c47fc53ab24d2: "drop HTTP streams that are completed, fix #4456".

Maybe streams should only be dropped in mitmdump mode, and not while there's still a live TCP connection? However, I have not looked carefully at this commit to understand exactly how the drop logic works.
A minimal patch which works around this issue:

```
diff --git a/mitmproxy/proxy/layers/http/__init__.py b/mitmproxy/proxy/layers/http/__init__.py
index 7a2d0a305..b03b92e79 100644
--- a/mitmproxy/proxy/layers/http/__init__.py
+++ b/mitmproxy/proxy/layers/http/__init__.py
@@ -404,8 +404,7 @@ class HttpStream(layer.Layer):
         if self.flow.response.trailers:
             yield SendHttp(ResponseTrailers(self.stream_id, self.flow.response.trailers), self.context.client)
 
-        if self.client_state == self.state_done:
-            yield from self.flow_done()
+        yield from self.flow_done()
 
     def flow_done(self):
         if not self.flow.websocket:
```

This works on both 8.1.0 and on 10.1.1 (after compensating for whitespace changes) and restores HTTPS functionality. I'm not totally sure what the implication of this patch is, but it restores the pre-8.1.0 logic of calling the `flow_done` code unconditionally at the end of `send_response`.
Thank you two! That's clearly a protocol-level bug that needs a fix + a test case to make sure we don't regress again. Happy to look into this, but it likely won't be this week. Fortunately @nneonneo has found a workaround in the meantime. :)
> I'm not totally sure what the implication of this patch is

If mitmproxy is streaming request bodies, the entire response processing may be completed before the request has been completed. For example, a server may send a *413 Content Too Large* while the client is still uploading. In that case we want to avoid prematurely dropping the stream. The logic bug here is easy to fix (setting `self.client_state = self.state_done` right before https://github.com/mitmproxy/mitmproxy/blob/10.1.1/mitmproxy/proxy/layers/http/__init__.py#L759), but I actually want to go in and see if there are similar issues lurking somewhere. 

Thanks again for the super helpful bisect! 😃 

Resolving patch:
diff --git a/mitmproxy/proxy/layers/http/__init__.py b/mitmproxy/proxy/layers/http/__init__.py
--- a/mitmproxy/proxy/layers/http/__init__.py
+++ b/mitmproxy/proxy/layers/http/__init__.py
@@ -700,6 +700,7 @@ def make_server_connection(self) -> layer.CommandGenerator[bool]:
             return True
 
     def handle_connect(self) -> layer.CommandGenerator[None]:
+        self.client_state = self.state_done
         yield HttpConnectHook(self.flow)
         if (yield from self.check_killed(False)):
             return


Test patch:
diff --git a/test/mitmproxy/proxy/layers/http/test_http.py b/test/mitmproxy/proxy/layers/http/test_http.py
--- a/test/mitmproxy/proxy/layers/http/test_http.py
+++ b/test/mitmproxy/proxy/layers/http/test_http.py
@@ -678,6 +678,7 @@ def test_server_unreachable(tctx, connect):
         # Our API isn't ideal here, there is no error hook for CONNECT requests currently.
         # We could fix this either by having CONNECT request go through all our regular hooks,
         # or by adding dedicated ok/error hooks.
+        # See also: test_connect_unauthorized
         playbook << http.HttpErrorHook(flow)
         playbook >> reply()
     playbook << SendData(
@@ -1632,6 +1633,42 @@ def test_connect_more_newlines(tctx):
     assert nl().data_client() == b"\x16\x03\x03\x00\xb3\x01\x00\x00\xaf\x03\x03"
 
 
+def test_connect_unauthorized(tctx):
+    """Continue a connection after proxyauth returns a 407, https://github.com/mitmproxy/mitmproxy/issues/6420"""
+    playbook = Playbook(http.HttpLayer(tctx, HTTPMode.regular))
+    flow = Placeholder(HTTPFlow)
+
+    def require_auth(f: HTTPFlow):
+        f.response = Response.make(
+            status_code=407, headers={"Proxy-Authenticate": f'Basic realm="mitmproxy"'}
+        )
+
+    assert (
+        playbook
+        >> DataReceived(tctx.client, b"CONNECT example.com:80 HTTP/1.1\r\n\r\n")
+        << http.HttpConnectHook(flow)
+        >> reply(side_effect=require_auth)
+        # This isn't ideal - we should probably have a custom CONNECT error hook here.
+        # See also: test_server_unreachable
+        << http.HttpResponseHook(flow)
+        >> reply()
+        << SendData(
+            tctx.client,
+            b"HTTP/1.1 407 Proxy Authentication Required\r\n"
+            b'Proxy-Authenticate: Basic realm="mitmproxy"\r\n'
+            b"content-length: 0\r\n\r\n",
+        )
+        >> DataReceived(
+            tctx.client,
+            b"CONNECT example.com:80 HTTP/1.1\r\n"
+            b"Proxy-Authorization: Basic dGVzdDp0ZXN0\r\n\r\n",
+        )
+        << http.HttpConnectHook(Placeholder(HTTPFlow))
+        >> reply()
+        << OpenConnection(Placeholder(Server))
+    )
+
+
 def flows_tracked() -> int:
     return sum(isinstance(x, HTTPFlow) for x in gc.get_objects())