Repository: mitmproxy/mitmproxy

Issue:
Instagram TCP connections broken with HTTP2 enabled
#### Problem Description
Using the latest binary (v7) with HTTP2 enabled and in SOCKS mode, Instagram on Android can't use MQTT through TCP anymore.
Other Facebook apps might also be affected as they use a similar connection.

In an affected version, you can see a lot of TCP connections to two different hosts with the same port (443). If you look at the tcp messages, the last message is always `missing connection preface` (then the connection is closed).

This doesn't happen if mitmproxy has HTTP2 disabled (instagram uses http2, so you only see tcp connections but the MQTT connections mentioned at the start work fine; there are two persistent connections).

I beleive this was introduced in #4529 (commit `f94a9a3`) as the build for #4532 (commit `0e8dd73`) doesn't have this issue.

#### Steps to reproduce the behavior:
0. Have an Instagram account with [WhiteHat mode](https://www.facebook.com/whitehat) enabled (and enable no TLS 1.3 and allow user certificates in the app).
1. Fully close Instagram
2. Start mitmproxy in SOCKS mode
3. Open Instagram

#### System Information
```
Mitmproxy: 7.0.0.dev binary
Python:    3.9.2
OpenSSL:   OpenSSL 1.1.1k  25 Mar 2021
Platform:  Windows-10-10.0.19041-SP0
```
Instagram TCP connections broken with HTTP2 enabled
#### Problem Description
Using the latest binary (v7) with HTTP2 enabled and in SOCKS mode, Instagram on Android can't use MQTT through TCP anymore.
Other Facebook apps might also be affected as they use a similar connection.

In an affected version, you can see a lot of TCP connections to two different hosts with the same port (443). If you look at the tcp messages, the last message is always `missing connection preface` (then the connection is closed).

This doesn't happen if mitmproxy has HTTP2 disabled (instagram uses http2, so you only see tcp connections but the MQTT connections mentioned at the start work fine; there are two persistent connections).

I beleive this was introduced in #4529 (commit `f94a9a3`) as the build for #4532 (commit `0e8dd73`) doesn't have this issue.

#### Steps to reproduce the behavior:
0. Have an Instagram account with [WhiteHat mode](https://www.facebook.com/whitehat) enabled (and enable no TLS 1.3 and allow user certificates in the app).
1. Fully close Instagram
2. Start mitmproxy in SOCKS mode
3. Open Instagram

#### System Information
```
Mitmproxy: 7.0.0.dev binary
Python:    3.9.2
OpenSSL:   OpenSSL 1.1.1k  25 Mar 2021
Platform:  Windows-10-10.0.19041-SP0
```


Discussion:
Thanks! A few questions:

- Can you reproduce this with `--set connection_strategy=lazy`?
- Which version are you on exactly? Downloaded today/in the last few days?

Also, a `mitmdump --set proxy_debug -vvv` log would be fabulous. If you are unsure whether it contains private data feel free to shoot me an email. :)
> Can you reproduce this with `--set connection_strategy=lazy`?

I can't. With this option (?) set, it works fine.


> Which version are you on exactly? Downloaded today/in the last few days?

The mitmdump logs I'll send you are on the latest build. The first occurence of this issue was after #4529 landed, so basically [this build](https://github.com/mitmproxy/mitmproxy/commit/f94a9a3c9dc6cc1ebfda7038676aa64560bd4f56). The previous build was fine.

I sent you the log via email.
Repro:
```
mitmdump -vvv --set proxy_debug --mode socks5
curl -x socks5h://localhost:8080 -k https://facebook.com -v --no-alpn
```

I'll have a fix out momentarily. 😃 
Thanks! A few questions:

- Can you reproduce this with `--set connection_strategy=lazy`?
- Which version are you on exactly? Downloaded today/in the last few days?

Also, a `mitmdump --set proxy_debug -vvv` log would be fabulous. If you are unsure whether it contains private data feel free to shoot me an email. :)
> Can you reproduce this with `--set connection_strategy=lazy`?

I can't. With this option (?) set, it works fine.


> Which version are you on exactly? Downloaded today/in the last few days?

The mitmdump logs I'll send you are on the latest build. The first occurence of this issue was after #4529 landed, so basically [this build](https://github.com/mitmproxy/mitmproxy/commit/f94a9a3c9dc6cc1ebfda7038676aa64560bd4f56). The previous build was fine.

I sent you the log via email.
Repro:
```
mitmdump -vvv --set proxy_debug --mode socks5
curl -x socks5h://localhost:8080 -k https://facebook.com -v --no-alpn
```

I'll have a fix out momentarily. 😃 

Resolving patch:
diff --git a/mitmproxy/addons/tlsconfig.py b/mitmproxy/addons/tlsconfig.py
--- a/mitmproxy/addons/tlsconfig.py
+++ b/mitmproxy/addons/tlsconfig.py
@@ -169,18 +169,19 @@ def create_proxy_server_ssl_conn(self, tls_start: tls.TlsStartData) -> None:
         if not server.alpn_offers:
             if client.alpn_offers:
                 if ctx.options.http2:
+                    # We would perfectly support HTTP/1 -> HTTP/2, but we want to keep things on the same protocol
+                    # version. There are some edge cases where we want to mirror the regular server's behavior
+                    # accurately, for example header capitalization.
                     server.alpn_offers = tuple(client.alpn_offers)
                 else:
                     server.alpn_offers = tuple(x for x in client.alpn_offers if x != b"h2")
-            elif client.tls_established:
-                # We would perfectly support HTTP/1 -> HTTP/2, but we want to keep things on the same protocol version.
-                # There are some edge cases where we want to mirror the regular server's behavior accurately,
-                # for example header capitalization.
-                server.alpn_offers = []
-            elif ctx.options.http2:
-                server.alpn_offers = tls.HTTP_ALPNS
             else:
-                server.alpn_offers = tls.HTTP1_ALPNS
+                # We either have no client TLS or a client without ALPN.
+                # - If the client does use TLS but did not send an ALPN extension, we want to mirror that upstream.
+                # - If the client does not use TLS, there's no clear-cut answer. As a pragmatic approach, we also do
+                #   not send any ALPN extension in this case, which defaults to whatever protocol we are speaking
+                #   or falls back to HTTP.
+                server.alpn_offers = []
 
         if not server.cipher_list and ctx.options.ciphers_server:
             server.cipher_list = ctx.options.ciphers_server.split(":")


Test patch:
diff --git a/test/helper_tools/loggrep.py b/test/helper_tools/loggrep.py
--- a/test/helper_tools/loggrep.py
+++ b/test/helper_tools/loggrep.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 import fileinput
 import sys
+import re
 
 if __name__ == "__main__":
     if len(sys.argv) < 3:
@@ -10,7 +11,7 @@
     port = sys.argv[1]
     matches = False
     for line in fileinput.input(sys.argv[2:]):
-        if line.startswith("["):
+        if re.match(r"^\[|(\d+\.){3}", line):
             matches = port in line
         if matches:
             print(line, end="")
diff --git a/test/mitmproxy/addons/test_tlsconfig.py b/test/mitmproxy/addons/test_tlsconfig.py
--- a/test/mitmproxy/addons/test_tlsconfig.py
+++ b/test/mitmproxy/addons/test_tlsconfig.py
@@ -190,8 +190,8 @@ def assert_alpn(http2, client_offers, expected):
 
             assert_alpn(True, tls.HTTP_ALPNS + (b"foo",), tls.HTTP_ALPNS + (b"foo",))
             assert_alpn(False, tls.HTTP_ALPNS + (b"foo",), tls.HTTP1_ALPNS + (b"foo",))
-            assert_alpn(True, [], tls.HTTP_ALPNS)
-            assert_alpn(False, [], tls.HTTP1_ALPNS)
+            assert_alpn(True, [], [])
+            assert_alpn(False, [], [])
             ctx.client.timestamp_tls_setup = time.time()
             # make sure that we don't upgrade h1 to h2,
             # see comment in tlsconfig.py