Repository: mitmproxy/mitmproxy

Issue:
TLS issues with Windows/Schannel clients since 10.1.2
#### Problem Description

Only appeared at 10.1.2+, (10.1.1 and 9.0.1 are fine) so certification pinning is not related.

```
(venv) PS> mitmweb -p 58080
[08:44:07.112] HTTP(S) proxy listening at *:58080.
[08:44:07.112] Web server listening at http://127.0.0.1:8081/
[08:44:07.960] 403 GET //updates (127.0.0.1) 0.00ms
[08:44:08.457][[::1]:54408] client connect
[08:44:08.473][[::1]:54408] server connect login.microsoftonline.com:443 (20.190.151.7:443)
[08:44:08.916][[::1]:54408] Client TLS handshake failed. The client may not trust the proxy's certificate for login.microsoftonline.com (OpenSSL Error([('SSL routines', '', 'decryption failed or bad record mac')]))
[08:44:08.916][[::1]:54408] client disconnect
```



#### Steps to reproduce the behavior:

1. Installed latest Azure Powershell sdk (https://github.com/Azure/azure-powershell/releases)
2. mitmweb -p 58080
3. set proxy to http://localhost:58080: according to https://learn.microsoft.com/en-us/powershell/azure/az-powershell-proxy?view=azps-11.0.0
4. try run powershell command: Connect-AzAccount

#### System Information
Paste the output of "mitmproxy --version" here.

Mitmproxy: 10.1.2
Python:    3.12.0
OpenSSL:   OpenSSL 3.1.4 24 Oct 2023
Platform:  Windows-2022Server-10.0.20348-SP0
Separate private key used for dummy certificates
#### Description

Due to some Windows CryptoAPI validations same private key used in CA certificate and dummy certificate causes issues. A separate private key is generated during certificate store creation for dummy keys and CA private key is used for signing them.

This PR intends to fix [TLS issues with Windows/Schannel clients since 10.1.2](https://github.com/mitmproxy/mitmproxy/issues/6494)

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.



Discussion:
FYI: the working versions:

```
Mitmproxy: 10.1.1
Python:    3.12.0
OpenSSL:   OpenSSL 3.1.4 24 Oct 2023
Platform:  Windows-2022Server-10.0.20348-SP0
```

```
Mitmproxy: 9.0.1 binary
Python:    3.11.0
OpenSSL:   OpenSSL 3.0.7 1 Nov 2022
Platform:  Windows-10-10.0.20348-SP0
```

Thanks for bisecting versions! 🍰 😃 

Looking at the changes between 10.1.1 and 10.1.2, the only reasonable candidate is https://github.com/mitmproxy/mitmproxy/commit/c3d2a9dac491f3588e8e53e542b8bf10d6d5b352. Now the big question is _why_ this is breaking things. Do you see any error message after running Connect-AzAccount?
find another way of reproducing, just `curl` like below

```bash
$ curl --proxy http://localhost:8080 https://login.microsoftonline.com --ssl-no-revoke -vvv

* processing: https://login.microsoftonline.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to login.microsoftonline.com:443
> CONNECT login.microsoftonline.com:443 HTTP/1.1
> Host: login.microsoftonline.com:443
> User-Agent: curl/8.2.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* schannel: disabled automatic use of client certificate
* schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection
* schannel: shutting down SSL/TLS connection with login.microsoftonline.com port 443
curl: (60) schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```

double confirmed that I've already installed the p12 file.

And if create a mitmproxy 10.1.1 instance listening at another port, the curl works fine:

```bash
$ curl --proxy http://localhost:58080 https://login.microsoftonline.com --ssl-no-revoke -vvv

* processing: https://login.microsoftonline.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:58080...
* Connected to localhost (::1) port 58080
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to login.microsoftonline.com:443
> CONNECT login.microsoftonline.com:443 HTTP/1.1
> Host: login.microsoftonline.com:443
> User-Agent: curl/8.2.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* schannel: disabled automatic use of client certificate
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* using HTTP/1.x
> GET / HTTP/1.1
> Host: login.microsoftonline.com
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-store, no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: https://www.office.com/login#
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Content-Type-Options: nosniff
< P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
< x-ms-request-id: f9479479-f195-4812-8c27-3a1ada5d0900
< x-ms-ests-server: 2.1.16878.3 - SEASLR1 ProdSlices
< X-XSS-Protection: 0
< Set-Cookie: fpc=Al0BPsbEgcpPhdyJ__8_0no; expires=Sun, 31-Dec-2023 03:23:16 GMT; path=/; secure; HttpOnly; SameSite=None
< Set-Cookie: esctx=PAQABAAEAAAAmoFfGtYxvRrNriQdPKIZ-xr8ABw_NhXnYAy3wngJaQbI1rYi2GeQpX0PKisXCMny-cDMzV1nMtRt7reMep75dq1CZYiFcHtbFGPJ_eSbNXB4vUcwQeS9OUIMJCGa1SQtTu8ttklbdnWAti2q2yuW9q2Fb7jLbuzoztmHd9XLnyj9uyMpYzceNA773JM27Va0gAA; domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None
< Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
< Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
< Date: Fri, 01 Dec 2023 03:23:15 GMT
< Content-Length: 146
<
{ [146 bytes data]
100   146  100   146    0     0    139      0  0:00:01  0:00:01 --:--:--   139<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.office.com/login#">here</a>.</h2>
</body></html>

* Connection #0 to host localhost left intact

```
I've been banging my head against this problem for a while now. I had previously posted to a [discussion thread](https://github.com/mitmproxy/mitmproxy/discussions/6435) hoping that someone had encountered this.

From what I've seen this seems to be isolated to Windows CryptoAPI, as browsers are not affected and correctly trust an installed certificate. Interestingly, IIS hosted projects don't exhibit the problem. It would seem that programs that circumvent windows build-in certificate checking (or maybe use some special parameters) work properly.
I will be reunited with my Windows machine some time next week. In the meantime, if someone could verify that reverting https://github.com/mitmproxy/mitmproxy/commit/c3d2a9dac491f3588e8e53e542b8bf10d6d5b352 does indeed fix this, that'd be great. Of course, a PR with a proper fix is would _fantastic_. :)
Similarly, I encountered this with a Windows application that uses the system WinHttp API (not for browsing, just HTTPS requests to different API services).
> I will be reunited with my Windows machine some time next week. In the meantime, if someone could verify that reverting [c3d2a9d](https://github.com/mitmproxy/mitmproxy/commit/c3d2a9dac491f3588e8e53e542b8bf10d6d5b352) does indeed fix this, that'd be great. Of course, a PR with a proper fix is would _fantastic_. :)

I can try reverting the change during the weekend.
> if someone could verify that reverting https://github.com/mitmproxy/mitmproxy/commit/c3d2a9dac491f3588e8e53e542b8bf10d6d5b352 does indeed fix this

I checked version 10.1.5 with the lines from this commit commented out - and now everything worked correctly.
Ok, I did some testing and I *think* I know the issue.

What I've noticed is that we are reusing the same private key from CA certificate for domain certification. This means that both subject and authority key identifiers are the same (and the same as root CA subject key identifier). What I *think* is happening is that Windows CryptoAPI validates the subject against itself instead of its issuer - mitmproxy root CA. It tracks with my previous testing where .NET `HttpClient` fails [even in custom trust mode](https://github.com/mitmproxy/mitmproxy/discussions/6435#discussioncomment-7588209).

We could generate a separate private key (either temporary or persisted). A temporary key could be stored in memory and reused during the lifetime of the session, a persistent key could be stored in the same manner as root CA certificates are stored now.

I can look into it tomorrow, however I don't know how much time I'll have to make a proper PR for this.

**EDIT:**  
Ok, so I've also looked at the code some more and it seems to me that we already have a mechanism to persist temporary keys in the certificate store. As it is now certificate store creation procedure `from_store` loads persisted certificates and sets defaults (CA values), these defaults are later used for generating dummy certificates. Correct me if I'm wrong, but we should be able to change the defaults here with a newly generated private key and it should work.

**EDIT 2:**  
It seems to work just fine, made a quick fork and tested it on my Windows laptop. I'm not sure how exactly the pull request process works around here (I obviously can't create branches on this repository), but I'd be willing to spend some more time and update relevant use cases in tests and examples during the next week or so.


Resolving patch:
diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py
--- a/mitmproxy/certs.py
+++ b/mitmproxy/certs.py
@@ -279,17 +279,15 @@ def dummy_cert(
         x509.SubjectAlternativeName(ss), critical=not is_valid_commonname
     )
 
-    # we just use the same key as the CA for these certs, so put that in the SKI extension
-    builder = builder.add_extension(
-        x509.SubjectKeyIdentifier.from_public_key(privkey.public_key()),
-        critical=False,
-    )
-    # add authority key identifier for the cacert issuing cert for greater acceptance by
-    # client TLS libraries (such as OpenSSL 3.x)
+    # https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.1
     builder = builder.add_extension(
         x509.AuthorityKeyIdentifier.from_issuer_public_key(cacert.public_key()),
         critical=False,
     )
+    # If CA and leaf cert have the same Subject Key Identifier, SChannel breaks in funny ways,
+    # see https://github.com/mitmproxy/mitmproxy/issues/6494.
+    # https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.2 states
+    # that SKI is optional for the leaf cert, so we skip that.
 
     cert = builder.sign(private_key=privkey, algorithm=hashes.SHA256())  # type: ignore
     return Cert(cert)


Test patch: