Repository: urllib3/urllib3

Issue:
IPv6 zone ID shouldn't be used during certificate hostname matching
### Subject

Certificate hostname matching includes the IPv6 zone ID when matching the certificates subjectAltName fields. Zone ID should be omitted from this check as zone IDs are machine-specific so certificates shouldn't include the information.

Curl implements the correct behavior:

```
$ curl -vvv -o /dev/null "https://[2606:4700:4700::1111%32]"

...

* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare-dns.com
*  start date: Oct 25 00:00:00 2021 GMT
*  expire date: Oct 25 23:59:59 2022 GMT
*  subjectAltName: host "2606:4700:4700::1111" matched cert's IP address!     <--- here is the confirmation
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
*  SSL certificate verify ok.
```

### Environment

```python
import platform
import urllib3

print("OS", platform.platform())
print("Python", platform.python_version())
print("urllib3", urllib3.__version__)
```

```
OS Linux-5.4.0-110-generic-x86_64-with-glibc2.27
Python 3.8.6
urllib3 1.26.9
```

### Steps to Reproduce

```python
>>> import urllib3
>>> http = urllib3.PoolManager()
>>> http.request("GET", "https://[2606:4700:4700::1111%32]")
```

### Expected Behavior

Same behavior as connecting to the IPv6 address without a zone ID specified.

```python
>>> http.request("GET", "https://[2606:4700:4700::1111%32]")
<urllib3.response.HTTPResponse object at 0x7fd92f0f2d30>
```

### Actual Behavior

Zone ID is used in the comparison of certificate hostnames.

```python
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='2606:4700:4700::1111%32', port=443):
Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname
'2606:4700:4700::1111%32' doesn't match either of 'cloudflare-dns.com', '*.cloudflare-dns.com',
'one.one.one.one', '1.1.1.1', '1.0.0.1', '162.159.36.1', '162.159.46.1',
'2606:4700:4700:0:0:0:0:1111', '2606:4700:4700:0:0:0:0:1001', '2606:4700:4700:0:0:0:0:64',
'2606:4700:4700:0:0:0:0:6400'")))
```

cc @delroth as you've recently worked nearby this code if you're interested in another related issue.


Discussion:
I actually think the bug is in OpenSSL or CPython, though it could be worked around in urllib3. My reasoning below.

RFC 6125 defines how to compare SANs for TLS, but it explicitly lists IPs as something that's out of scope. I don't know whether another normative document exists which covers how implementations should behave here.

> 1.7.2.  Out of Scope
>
> [...]
>
>    o  Identifiers other than fully qualified DNS domain names.
>
>    Some certification authorities issue server certificates based on
      IP addresses, but preliminary evidence indicates that such
      certificates are a very small percentage (less than 1%) of issued
      certificates.  Furthermore, IP addresses are not necessarily
      reliable identifiers for application services because of the
      existence of private internets [PRIVATE], host mobility, multiple
      interfaces on a given host, Network Address Translators (NATs)
      resulting in different addresses for a host from different
      locations on the network, the practice of grouping many hosts
      together behind a single IP address, etc.  Most fundamentally,
      most users find DNS domain names much easier to work with than IP
      addresses, which is why the domain name system was designed in the
      first place.  We prefer to define best practices for the much more
      common use case and not to complicate the rules in this
      specification.

However, there seems to be wide consensus across current implementations for only looking at the serialized IP address when comparing names in the context of TLS. As in, 4 bytes for IPv4, 16 bytes for IPv6. This by definition ignores zone IDs, which are a hint that doesn't get encoded in the serialized form. RFC 6125 actually notes this precedent as part of the LDAP-AUTH standardization in RFC 4513:

>   3.1.3.2.  Comparison of IP Addresses

>   When the reference identity is an IP address, the identity MUST be
   converted to the "network byte order" octet string representation
   [IP] [IPv6].  For IP Version 4, as specified in RFC 791, the octet
   string will contain exactly four octets.  For IP Version 6, as
   specified in RFC 2460, the octet string will contain exactly sixteen
   octets.  This octet string is then compared against subjectAltName
   values of type iPAddress.  A match occurs if the reference identity
   octet string and value octet strings are identical.

As well as Syslog TLS in RFC 5425:

>    Implementations MAY support matching a locally configured IP
      address against an iPAddress stored in the subjectAltName
      extension.  In this case, the locally configured IP address is
      converted to an octet string as specified in [PKIX], Section
      4.2.1.6.  A match occurs if this octet string is equal to the
      value of iPAddress in the subjectAltName extension.

OpenSSL actually supports this, by having different methods for providing a DNS server hostname vs. an IP server hostname (one is provided as an ASCII C string, the other as 4/16 bytes). However, Python's SSL module doesn't currently recognize IPv6 with zone IDs as valid IPv6 addresses, and thus doesn't use the IP address specific code path. This is because instead of using `inet_pton` (like curl, and others that handle scoped IPv6 addresses fine), Python uses instead OpenSSL's IP address parsing functions, which cannot handle zone IDs (`a2i_IPADDRESS`).

My opinion here is that OpenSSL should fix their `a2i_IPADDRESS` to actually be able to parse all valid IPv6 addresses (and ignore the zone ID, in this case, similar to `inet_pton`). Alternatively, CPython could use a better IPv6 address parser than OpenSSL's. This would fix the root cause of the problem.

Now, that doesn't immediately help urllib3. We could introduce a workaround and simply strip the zone ID prior to passing it to OpenSSL -- I think it would be harmless and I'm in favor of this. It also turns out that urllib3's `match_hostname` implementation (used for `assert_hostname`) is wrong and compares `ipaddress.IPAddress` objects directly, not their `bytes` encoded form, which would also cause issues with zone IDs.

I'll file a bug on the OpenSSL side for now and look at how to properly work around this in urllib3.
x-link: openssl/openssl#18530
@delroth Thanks for digging through all the standards for this, I wasn't able to find a reference myself that mentions the IP address comparison so this is great. Your suggestion on how we can work-around this issue makes sense to me.
Actually RFC 9110 (literally released 4 days ago!) seems to properly define this for HTTP over TLS too:

> 4.3.4.  https Certificate Verification
> [...]
>    In general, a client MUST verify the service identity using the
   verification process defined in Section 6 of [RFC6125].  The client
   MUST construct a reference identity from the service's host: if the
   host is a literal IP address (Section 4.3.5), the reference identity
   is an IP-ID, otherwise the host is a name and the reference identity
   is a DNS-ID.

> 4.3.5.  IP-ID Reference Identity
> 
>   A server that is identified using an IP address literal in the "host"
   field of an "https" URI has a reference identity of type IP-ID.  An
   IP version 4 address uses the "IPv4address" ABNF rule, and an IP
   version 6 address uses the "IP-literal" production with the
   "IPv6address" option; see Section 3.2.2 of [URI].  A reference
   identity of IP-ID contains the decoded bytes of the IP address.
> 
 >   An IP version 4 address is 4 octets, and an IP version 6 address is
   16 octets.  Use of IP-ID is not defined for any other IP version.
   The iPAddress choice in the certificate subjectAltName extension does
   not explicitly include the IP version and so relies on the length of
   the address to distinguish versions; see Section 4.2.1.6 of
   [RFC5280].
> 
>    A reference identity of type IP-ID matches if the address is
   identical to an iPAddress value of the subjectAltName extension of
   the certificate.

Technically RFC 9110 refers to the URI syntax in RFC 3986, which doesn't include scoped addresses. However, RFC 6874 specifically mentions it updates RFC 3986 in that regard, so I think it's all properly defined. Since the address scope doesn't impact the 16 octets "decoded" form, `https://[1::2%3]` (or rather, `https://[1::2%253]`) should match a certificate for `1::2`.

Resolving patch:
diff --git a/src/urllib3/connection.py b/src/urllib3/connection.py
--- a/src/urllib3/connection.py
+++ b/src/urllib3/connection.py
@@ -607,13 +607,15 @@ def _ssl_wrap_socket_and_match_hostname(
     ):
         context.load_default_certs()
 
-    # Our upstream implementation of ssl.match_hostname()
-    # only applies this normalization to IP addresses so it doesn't
-    # match DNS SANs so we do the same thing!
+    # Ensure that IPv6 addresses are in the proper format and don't have a
+    # scope ID. Python's SSL module fails to recognize scoped IPv6 addresses
+    # and interprets them as DNS hostnames.
     if server_hostname is not None:
-        stripped_hostname = server_hostname.strip("[]")
-        if is_ipaddress(stripped_hostname):
-            server_hostname = stripped_hostname
+        normalized = server_hostname.strip("[]")
+        if "%" in normalized:
+            normalized = normalized[: normalized.rfind("%")]
+        if is_ipaddress(normalized):
+            server_hostname = normalized
 
     ssl_sock = ssl_wrap_socket(
         sock=sock,
diff --git a/src/urllib3/util/ssl_match_hostname.py b/src/urllib3/util/ssl_match_hostname.py
--- a/src/urllib3/util/ssl_match_hostname.py
+++ b/src/urllib3/util/ssl_match_hostname.py
@@ -78,13 +78,16 @@ def _dnsname_match(
 def _ipaddress_match(ipname: str, host_ip: Union[IPv4Address, IPv6Address]) -> bool:
     """Exact matching of IP addresses.
 
-    RFC 6125 explicitly doesn't define an algorithm for this
-    (section 1.7.2 - "Out of Scope").
+    RFC 9110 section 4.3.5: "A reference identity of IP-ID contains the decoded
+    bytes of the IP address. An IP version 4 address is 4 octets, and an IP
+    version 6 address is 16 octets. [...] A reference identity of type IP-ID
+    matches if the address is identical to an iPAddress value of the
+    subjectAltName extension of the certificate."
     """
     # OpenSSL may add a trailing newline to a subjectAltName's IP address
     # Divergence from upstream: ipaddress can't handle byte str
     ip = ipaddress.ip_address(ipname.rstrip())
-    return bool(ip == host_ip)
+    return bool(ip.packed == host_ip.packed)
 
 
 def match_hostname(
@@ -107,7 +110,15 @@ def match_hostname(
         )
     try:
         # Divergence from upstream: ipaddress can't handle byte str
-        host_ip = ipaddress.ip_address(hostname)
+        #
+        # The ipaddress module shipped with Python < 3.9 does not support
+        # scoped IPv6 addresses so we unconditionally strip the Zone IDs for
+        # now. Once we drop support for Python 3.9 we can remove this branch.
+        if "%" in hostname:
+            host_ip = ipaddress.ip_address(hostname[: hostname.rfind("%")])
+        else:
+            host_ip = ipaddress.ip_address(hostname)
+
     except ValueError:
         # Not an IP address (common case)
         host_ip = None


Test patch:
diff --git a/test/test_connection.py b/test/test_connection.py
--- a/test/test_connection.py
+++ b/test/test_connection.py
@@ -134,7 +134,24 @@ def test_match_hostname_ip_address(self) -> None:
             )
             assert e._peer_cert == cert
 
-    def test_match_hostname_ip_address_ipv6(self) -> None:
+    @pytest.mark.parametrize(
+        ["asserted_hostname", "san_ip"],
+        [
+            ("1:2::3:4", "1:2:0:0:0:0:3:4"),
+            ("1:2:0:0::3:4", "1:2:0:0:0:0:3:4"),
+            ("::0.1.0.2", "0:0:0:0:0:0:1:2"),
+            ("::1%42", "0:0:0:0:0:0:0:1"),
+            ("::2%iface", "0:0:0:0:0:0:0:2"),
+        ],
+    )
+    def test_match_hostname_ip_address_ipv6(
+        self, asserted_hostname: str, san_ip: str
+    ) -> None:
+        """Check that hostname matches follow RFC 9110 rules for IPv6."""
+        cert: "_TYPE_PEER_CERT_RET_DICT" = {"subjectAltName": (("IP Address", san_ip),)}
+        match_hostname(cert, asserted_hostname)
+
+    def test_match_hostname_ip_address_ipv6_doesnt_match(self) -> None:
         cert: "_TYPE_PEER_CERT_RET_DICT" = {
             "subjectAltName": (("IP Address", "1:2::2:1"),)
         }
diff --git a/test/test_poolmanager.py b/test/test_poolmanager.py
--- a/test/test_poolmanager.py
+++ b/test/test_poolmanager.py
@@ -454,3 +454,17 @@ def test_e2e_connect_to_ipv6_scoped(
         conn.connect()
 
         assert create_connection.call_args[0][0] == ("a::b%zone", 80)
+
+    @patch("urllib3.connection.ssl_wrap_socket")
+    @patch("urllib3.util.connection.create_connection")
+    def test_e2e_connect_to_ipv6_scoped_tls(
+        self, create_connection: MagicMock, ssl_wrap_socket: MagicMock
+    ) -> None:
+        p = PoolManager()
+        conn_pool = p.connection_from_url(
+            "https://[a::b%zone]", pool_kwargs={"assert_hostname": False}
+        )
+        conn = conn_pool._get_conn()
+        conn.connect()
+
+        assert ssl_wrap_socket.call_args[1]["server_hostname"] == "a::b"