Repository: mitmproxy/mitmproxy

Issue:
MITMproxy crashes when using upstream proxy with -k
#### Problem Description

As the title says, I'm using mitmproxy with an upstream proxy (burp) and the
`-k` flag, and it crashes


#### Steps to reproduce the behavior:
1. Setup burp listening on 8081
2. Open mitmproxy with `mitmproxy -k --mode "upstream:http://127.0.0.1:8081"`
3. Try and send a request, e.g. `curl https://jonathanh.co.uk -k` through the proxies

#### System Information
Paste the output of "mitmproxy --version" here.


```
Mitmproxy: 7.0.0
Python:    3.9.6
OpenSSL:   OpenSSL 1.1.1k  25 Mar 2021
Platform:  Linux-5.12.15-zen1-1-zen-x86_64-with-glibc2.33
```
### Event Log

The event log contains the following:

```
info: Proxy server listening at http://*:8080
info: 127.0.0.1:55706: client connect
info: 127.0.0.1:55706: server connect 127.0.0.1:8081
error: Addon error: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/mitmproxy/addons/tlsconfig.py", line 122, in tls_start_client
    self.create_client_proxy_ssl_conn(tls_start)
  File "/usr/lib/python3.9/site-packages/mitmproxy/addons/tlsconfig.py", line 131, in create_client_proxy_ssl_conn
    entry = self.get_cert(tls_start.context)
  File "/usr/lib/python3.9/site-packages/mitmproxy/addons/tlsconfig.py", line 287, in get_cert
    if upstream_cert.cn:
  File "/usr/lib/python3.9/site-packages/mitmproxy/certs.py", line 117, in cn
    attrs = self._cert.subject.get_attributes_for_oid(x509.NameOID.COMMON_NAME)
  File "/usr/lib/python3.9/site-packages/cryptography/hazmat/backends/openssl/x509.py", line 110, in subject
    return _decode_x509_name(self._backend, subject)
  File "/usr/lib/python3.9/site-packages/cryptography/hazmat/backends/openssl/decode_asn1.py", line 63, in _decode_x509_name
    attribute = _decode_x509_name_entry(backend, entry)
  File "/usr/lib/python3.9/site-packages/cryptography/hazmat/backends/openssl/decode_asn1.py", line 54, in _decode_x509_name_entry
    return x509.NameAttribute(x509.ObjectIdentifier(oid), value, type)
  File "/usr/lib/python3.9/site-packages/cryptography/x509/name.py", line 91, in __init__
    raise ValueError(
ValueError: Country name must be a 2 character country code

error: 127.0.0.1:55706: No TLS context was provided, failing connection.
error: 127.0.0.1:55706: mitmproxy has crashed!
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/server.py", line 279, in server_event
    for command in layer_commands:
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 168, in handle_event
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 255, in handle_event
    yield from self._handle(event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 168, in handle_event
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/http/__init__.py", line 690, in _handle_event
    yield from self.event_to_child(stream, event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/http/__init__.py", line 740, in event_to_child
    for command in child.handle_event(event):
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 168, in handle_event
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/http/__init__.py", line 608, in passthrough
    for command in self.child_layer.handle_event(event):
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 168, in handle_event
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/tunnel.py", line 85, in _handle_event
    yield from self.event_to_child(event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/tunnel.py", line 104, in event_to_child
    for command in self.child_layer.handle_event(event):
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 168, in handle_event
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/tunnel.py", line 85, in _handle_event
    yield from self.event_to_child(event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/tls.py", line 320, in event_to_child
    yield from super().event_to_child(event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/tunnel.py", line 104, in event_to_child
    for command in self.child_layer.handle_event(event):
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 129, in handle_event
    yield from self.__continue(event)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 220, in __continue
    yield from self.__process(command_generator, event.reply)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layer.py", line 207, in __process
    command = next(command_generator)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/tunnel.py", line 61, in _handle_event
    done, err = yield from self.receive_handshake_data(event.data)
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/tls.py", line 381, in receive_handshake_data
    yield from self.start_tls()
  File "/usr/lib/python3.9/site-packages/mitmproxy/proxy/layers/tls.py", line 173, in start_tls
    assert tls_start.ssl_conn
AssertionError

info: 127.0.0.1:55706: Client TLS handshake failed. The client disconnected during the handshake. If this happens consistently for jonathanh.co.uk, this may
indicate that the client does not trust the proxy's certificate.
info: 127.0.0.1:55706: client disconnect
info: 127.0.0.1:55706: server disconnect 127.0.0.1:8081
```


Discussion:
I added the `-k` flag in the hope that it would make mitmproxy ignore the fact
that it doesn't trust burp's certificate.
Thanks for the report! The problem here isn't `-k` in particular, but the fact that mitmproxy does not like the certificate generated by Burp. Do you happen to have a copy of the cert presented to the client?
In either case we'll get this fixed for the next patch release, a cert would just be nice to add a test. 😃 
Here is the output of `openssl s_client -proxy 127.0.0.1:8081 -connect jonathanh.co.uk:443 | openssl x509 -noout -text`

```
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 128888301 (0x7aeaded)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PortSwigger, ST = PortSwigger, L = PortSwigger, O = PortSwigger, OU = PortSwigger CA, CN = PortSwigger CA
        Validity
            Not Before: Jul 16 16:49:03 2021 GMT
            Not After : Jul 16 16:49:03 2022 GMT
        Subject: C = PortSwigger, O = PortSwigger, OU = PortSwigger CA, CN = jonathanh.co.uk
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:9a:29:92:4b:68:9f:93:d3:27:d7:7b:7d:99:4e:
                    18:b1:c2:67:99:f3:8c:a1:45:c5:b1:09:1e:36:e2:
                    9c:78:b6:79:36:7f:f0:73:33:5b:17:6d:26:c5:ee:
                    4c:eb:4e:7e:7b:44:d6:69:49:9e:d9:27:3b:a5:b6:
                    4d:02:6d:af:7b:69:7d:b1:f5:63:72:3c:b6:ac:f2:
                    a0:73:73:4c:13:b1:0d:e2:d8:18:78:ae:40:96:a6:
                    9b:7c:65:16:9a:7e:68:88:22:c1:91:b5:ef:a4:86:
                    db:94:95:cf:5f:90:a7:c4:3e:83:d3:cf:66:2c:a7:
                    79:1d:0f:c1:86:06:8d:33:5b:f8:49:ea:a0:94:a7:
                    64:d4:af:a5:cf:80:ab:42:b7:6b:8e:e7:23:03:39:
                    dd:5c:bd:c9:76:d8:2d:c2:5d:15:ff:cb:c5:f9:5a:
                    ac:ef:87:1f:cb:7f:f3:c0:a1:ea:34:9f:ae:1a:8f:
                    f6:d4:f8:2b:27:14:e4:95:66:0a:eb:c0:d4:bf:a1:
                    ee:5c:5a:e3:4b:43:08:c3:81:b3:f2:1c:82:46:bf:
                    a1:00:72:b8:e6:b4:a9:49:81:1b:a9:e1:bc:0b:57:
                    70:b0:30:cc:14:8b:44:c3:93:4e:1a:f0:70:02:8f:
                    4e:81:44:06:9e:4c:f4:6e:c5:da:82:95:59:5d:5b:
                    33:35
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:jonathanh.co.uk
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
    Signature Algorithm: sha256WithRSAEncryption
         34:84:d1:64:1e:ba:1a:8c:9c:ce:0d:0d:28:81:d0:fd:05:0c:
         cb:88:2a:0e:bc:e6:09:fa:73:f6:a4:96:09:78:f9:e1:9b:05:
         b5:28:f1:19:22:5e:8e:4e:a7:7f:ab:6d:11:7f:c2:94:f1:20:
         61:75:a7:b7:08:16:a7:c3:80:45:96:c0:58:b6:95:d6:ba:88:
         bf:21:45:b2:74:8d:8f:c2:69:bd:a5:d3:f6:00:5c:6a:1c:fa:
         0c:fb:b8:d7:34:31:6e:1d:c7:5b:39:27:e3:a3:46:30:f5:f0:
         29:e6:83:9b:30:c7:26:dd:67:2b:83:45:02:26:52:4e:12:de:
         d2:69:01:80:09:ee:c9:23:94:8d:39:e7:5c:80:5c:48:f0:ba:
         2c:e9:bf:27:d1:6d:3d:19:af:8f:f8:06:e5:be:79:5b:ca:65:
         94:6b:a3:63:9c:7f:af:95:a4:c2:ce:0c:74:d8:c8:cf:dd:1a:
         58:7f:1f:39:18:93:b9:bf:40:f4:b5:bc:43:1b:cd:d5:a8:71:
         be:49:89:cc:e3:f3:dc:3d:66:3d:2d:ad:f2:62:65:fe:a9:3d:
         60:2f:0d:20:cd:80:f6:b9:9d:0e:6d:f7:69:14:8d:f9:4b:70:
         1e:e6:12:b4:96:41:da:74:64:16:a0:e8:8f:f2:b2:ed:c8:f6:
         99:38:8c:d6
```

Is there anything else you need?

Also, I used the `-k` flag because I thought it would ignore the fact that
certificates are invalid / untrusted. Could you point me towards documentation on how I
should instruct mitmproxy to trust burp's CA?
So, it would appear that this is a result of the crypto library being used. 

https://github.com/pyca/cryptography/blob/37103b3b65c5a41a7d4dd8cc00b09684f136c2d0/src/cryptography/x509/name.py#L92-L95

Is it worth opening an issue there?

The certificate that burp uses doesn't have a 2 letter country code. I am not
sure if that is a violtation of spec or just unusual but the crypto library
assumes that the country code will be two letters
https://github.com/pyca/cryptography/issues/3857
I've marked this as an upstream issue, not sure if we can workaround this easily.

For Burp Suite in particular it might be worth asking PortSwigger to fix their certificate.
I have created a bug report on the port swigger forums. Hopefully that will get sorted:

https://forum.portswigger.net/thread/valid-certificate-generated-8064875d

Resolving patch:
diff --git a/mitmproxy/addons/tlsconfig.py b/mitmproxy/addons/tlsconfig.py
--- a/mitmproxy/addons/tlsconfig.py
+++ b/mitmproxy/addons/tlsconfig.py
@@ -290,11 +290,18 @@ def get_cert(self, conn_context: context.Context) -> certs.CertStoreEntry:
         # Use upstream certificate if available.
         if conn_context.server.certificate_list:
             upstream_cert = conn_context.server.certificate_list[0]
-            if upstream_cert.cn:
-                altnames.append(upstream_cert.cn)
+            try:
+                # a bit clunky: access to .cn can fail, see https://github.com/mitmproxy/mitmproxy/issues/4713
+                if upstream_cert.cn:
+                    altnames.append(upstream_cert.cn)
+            except ValueError:
+                pass
             altnames.extend(upstream_cert.altnames)
-            if upstream_cert.organization:
-                organization = upstream_cert.organization
+            try:
+                if upstream_cert.organization:
+                    organization = upstream_cert.organization
+            except ValueError:
+                pass
 
         # Add SNI. If not available, try the server address as well.
         if conn_context.client.sni:


Test patch:
diff --git a/test/mitmproxy/addons/test_tlsconfig.py b/test/mitmproxy/addons/test_tlsconfig.py
--- a/test/mitmproxy/addons/test_tlsconfig.py
+++ b/test/mitmproxy/addons/test_tlsconfig.py
@@ -84,6 +84,10 @@ def test_get_cert(self, tdata):
             entry = ta.get_cert(ctx)
             assert entry.cert.altnames == ["example.mitmproxy.org", "sni.example"]
 
+            with open(tdata.path("mitmproxy/data/invalid-subject.pem"), "rb") as f:
+                ctx.server.certificate_list = [certs.Cert.from_pem(f.read())]
+            assert ta.get_cert(ctx)  # does not raise
+
     def test_tls_clienthello(self):
         # only really testing for coverage here, there's no point in mirroring the individual conditions
         ta = tlsconfig.TlsConfig()
diff --git a/test/mitmproxy/data/invalid-subject.pem b/test/mitmproxy/data/invalid-subject.pem
new file mode 100644
--- /dev/null
+++ b/test/mitmproxy/data/invalid-subject.pem
@@ -0,0 +1,17 @@
+-----BEGIN CERTIFICATE-----
+MIICzDCCAomgAwIBAgIEZu2lSDALBgcqhkjOOAQDBQAwNjEQMA4GA1UEBhMHc2lh
+dmFzaDEQMA4GA1UEChMHc2lhdmFzaDEQMA4GA1UEAxMHc2lhdmFzaDAgFw0xNjAx
+MjUwNzI1MzBaGA8yMDU0MDUyNTA3MjUzMFowNjEQMA4GA1UEBhMHc2lhdmFzaDEQ
+MA4GA1UEChMHc2lhdmFzaDEQMA4GA1UEAxMHc2lhdmFzaDCCAbgwggEsBgcqhkjO
+OAQBMIIBHwKBgQD9f1OBHXUSKVLfSpwu7OTn9hG3UjzvRADDHj+AtlEmaUVdQCJR
++1k9jVj6v8X1ujD2y5tVbNeBO4AdNG/yZmC3a5lQpaSfn+gEexAiwk+7qdf+t8Yb
++DtX58aophUPBPuD9tPFHsMCNVQTWhaRMvZ1864rYdcq7/IiAxmd0UgBxwIVAJdg
+UI8VIwvMspK5gqLrhAvwWBz1AoGBAPfhoIXWmz3ey7yrXDa4V7l5lK+7+jrqgvlX
+TAs9B4JnUVlXjrrUWU/mcQcQgYC0SRZxI+hMKBYTt88JMozIpuE8FnqLVHyNKOCj
+rh4rs6Z1kW6jfwv6ITVi8ftiegEkO8yk8b6oUZCJqIPf4VrlnwaSi2ZegHtVJWQB
+TDv+z0kqA4GFAAKBgQDptGpQCqvLt7Quk0d9ufisfIGerdOS9Pot18EJc8OZ0A1F
+eWBSv7N2VnpRjP4UMqD4ReS+dap2RiM1v3+EtsY0u2yIS2ri57bxzHjDvm+YtlmA
+gyxA2do9KwR8upiyTId7Ks55CLCvVv1iNn3zVeARoEgsm4dRw3Oo0TEHbNAUIaMh
+MB8wHQYDVR0OBBYEFO6sMuXwh4G7by85KBSVGzKIaK5MMAsGByqGSM44BAMFAAMw
+ADAtAhUAkgnrEulmqvrigWIOfal0N3TXhGMCFEvzapY0lnykdjm7wQcSVxsDbhPV
+-----END CERTIFICATE-----