Repository: mitmproxy/mitmproxy

Issue:
SSLKEYLOGFILE is not containing TLSv1.3 secrets
#### When using mitmproxy with the SSLKEYLOGFILE environment variable TLSv1.3 keys are not exported or correctly labeled. 

I want to analyze and decrypt TLSv1.3 traffic of an application with mitmproxy and Wireshark.
I configured a gateway running mitmproxy in transparent mode and inside mitmproxy the traffic gets decrypted but Wireshark can not decrypt the captured data using the keylogfile provided by mitmproxy. 
After some research I found [this presentation](https://lekensteyn.nl/files/wireshark-tls-debugging-sharkfest19us.pdf) regarding the decryption of TLSv1.3 traffic with Wireshark. On Slide 9 there is a keylogfile example for decrypting TLSv1.3. 
In the  keylogfile provided by mitmproxy I can't find any lines starting with CLIENT_HANDSHAKE_TRAFFIC_SECRET, CLIENT_TRAFFIC_SECRET_0 nor EXPORTER_SECRET but only ones starting with CLIENT_RANDOM.


#### Steps to reproduce the behavior:
1. Export the SSLKEYLOGFILE environment variable
2. Setup mitmproxy in transparent monde
3. Open a website using TLSv1.3
4. Check the keylogfile

#### System Information
Mitmproxy: 5.1.1 binary
Python:    3.7.6
OpenSSL:   OpenSSL 1.1.1f  31 Mar 2020
Platform:  Linux-5.5.0-kali2-amd64-x86_64-with-debian-kali-rolling



Discussion:
Thanks! We're currently waiting for upstream to expose the new OpenSSL hooks. For now the workaround is to disable 1.3 for key logging, eventually we'll support it properly. :)

refs https://github.com/pyca/cryptography/pull/5187
refs https://github.com/pyca/pyopenssl/pull/910
I'm sorry for disturbing, could you please advise when you plan to fix it?
https://github.com/pyca/pyopenssl/pull/910 has been merged just now, so that should not take that long…™
Looking forward to it 🤞
I'm not here to rush you (we are 1 week later now), just do you have any estimate for this fix?
No estimate. I don't think I will look into that before the next pyOpenSSL release with my changes ships.

I'd be more than happy to merge a PR that implements it directly on top of cryptography (the pyOpenSSL wrapper is minimal) if somebody wants to give it a stab.
So 'cryptography' has had the required change since version 3.0, but we're waiting for either:
* a post-19.1.0 release of pyOpenSSL
* someone to step up and build the feature directly on top of cryptography, side-stepping the missing pyOpenSSL release

Resolving patch:
diff --git a/mitmproxy/net/tls.py b/mitmproxy/net/tls.py
--- a/mitmproxy/net/tls.py
+++ b/mitmproxy/net/tls.py
@@ -79,43 +79,24 @@ def client_arguments_from_options(options: "mitmproxy.options.Options") -> dict:
 
 class MasterSecretLogger:
     def __init__(self, filename):
-        self.filename = filename
+        self.filename = os.path.expanduser(filename)
         self.f = None
         self.lock = threading.Lock()
 
     # required for functools.wraps, which pyOpenSSL uses.
     __name__ = "MasterSecretLogger"
 
-    def __call__(self, connection, where, ret):
-        done_now = (
-            where == SSL.SSL_CB_HANDSHAKE_DONE and ret == 1
-        )
-        # this is a horrendous workaround for https://github.com/mitmproxy/mitmproxy/pull/3692#issuecomment-608454530:
-        # OpenSSL 1.1.1f decided to not make connection.master_key() fail in the SSL_CB_HANDSHAKE_DONE callback.
-        # To support various OpenSSL versions and still log master secrets, we now mark connections where this has
-        # happened and then try again on the next event. This is ugly and shouldn't be done, but eventually we
-        # replace this with context.set_keylog_callback anyways.
-        done_previously_but_not_logged_yet = (
-            hasattr(connection, "_still_needs_masterkey")
-        )
-        if done_now or done_previously_but_not_logged_yet:
-            with self.lock:
-                if not self.f:
-                    d = os.path.dirname(self.filename)
-                    if not os.path.isdir(d):
-                        os.makedirs(d)
-                    self.f = open(self.filename, "ab")
-                    self.f.write(b"\r\n")
-                try:
-                    client_random = binascii.hexlify(connection.client_random())
-                    masterkey = binascii.hexlify(connection.master_key())
-                except (AssertionError, SSL.Error):  # careful: exception type changes between pyOpenSSL versions
-                    connection._still_needs_masterkey = True
-                else:
-                    self.f.write(b"CLIENT_RANDOM %s %s\r\n" % (client_random, masterkey))
-                    self.f.flush()
-                    if hasattr(connection, "_still_needs_masterkey"):
-                        delattr(connection, "_still_needs_masterkey")
+    def __call__(self, connection, keymaterial):
+        with self.lock:
+            if not self.f:
+                d = os.path.dirname(self.filename)
+                if not os.path.isdir(d):
+                    os.makedirs(d)
+                self.f = open(self.filename, "ab")
+                self.f.write(b"\n")
+            self.f.write(keymaterial)
+            self.f.write(b"\n")
+            self.f.flush()
 
     def close(self):
         with self.lock:
@@ -203,7 +184,7 @@ def _create_ssl_context(
 
     # SSLKEYLOGFILE
     if log_master_secret:
-        context.set_info_callback(log_master_secret)
+        context.set_keylog_callback(log_master_secret)
 
     if alpn_protos is not None:
         # advertise application layer protocols
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -83,7 +83,7 @@
         "passlib>=1.6.5, <1.8",
         "protobuf>=3.6.0, <3.14",
         "pyasn1>=0.3.1,<0.5",
-        "pyOpenSSL>=19.1.0,<19.2",
+        "pyOpenSSL>=20.0,<20.1",
         "pyparsing>=2.4.2,<2.5",
         "pyperclip>=1.6.0,<1.9",
         "ruamel.yaml>=0.16,<0.17",


Test patch:
diff --git a/test/mitmproxy/net/test_tls.py b/test/mitmproxy/net/test_tls.py
--- a/test/mitmproxy/net/test_tls.py
+++ b/test/mitmproxy/net/test_tls.py
@@ -43,7 +43,7 @@ def test_log(self, tmpdir):
 
             tls.log_master_secret.close()
             with open(logfile, "rb") as f:
-                assert f.read().count(b"CLIENT_RANDOM") >= 2
+                assert f.read().count(b"SERVER_HANDSHAKE_TRAFFIC_SECRET") >= 2
 
         tls.log_master_secret = _logfun