Repository: urllib3/urllib3

Issue:
when connecting to https by IP address, client hello message has a server_name SNI block with literal ip address
I originally opened this in [requests](https://github.com/requests/requests/issues/4178), but was asked to raise it here.

When connecting to a machine over https using an IP address, the SSL client hello includes a server_name extension with the ip address listed as the server name. This is not allowed in the [SNI spec](https://tools.ietf.org/html/rfc3546#section-3.1), and it should not include the server_name extension. 

Reproduce:
1. Set up packet logging
2. ```python
    import urllib3
    http = urllib3.PoolManager()
    http.request('GET', 'https://8.8.8.8')
    ````
3. Inspect packet logs for client hello. It'll look a bit like this:
```
Frame 31: 571 bytes on wire (4568 bits), 571 bytes captured (4568 bits) on interface 0
Ethernet II, Src: BizlinkK_XX:XX:XX (9c:eb:e8:XX:XX:XX), Dst: JuniperN_XX:XX:XX (08:81:f4:XX:XX:XX)
Internet Protocol Version 4, Src: 192.168.89.18, Dst: 8.8.8.8
Transmission Control Protocol, Src Port: 1156, Dst Port: 443, Seq: 1, Ack: 1, Len: 517
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 512
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 508
            Version: TLS 1.2 (0x0303)
            Random
            Session ID Length: 0
            Cipher Suites Length: 148
            Cipher Suites (74 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 319
            Extension: server_name
                Type: server_name (0x0000)
                Length: 12
                Server Name Indication extension
                    Server Name list length: 10
                    Server Name Type: host_name (0)
                    Server Name length: 7
                    Server Name: 8.8.8.8
            Extension: ec_point_formats
            Extension: elliptic_curves
            Extension: SessionTicket TLS
            Extension: signature_algorithms
            Extension: Heartbeat
            Extension: Padding
```

The particular server I'm connecting to (azure windows 2016 Datacenter edition) immediately closes the connection when it sees a client hello like this.


Discussion:
Ok, so this is a thing that's bothered me for a while. SNI absolutely should not have IP addresses in it, but spotting an IP address is a tricky thing to do. I think our best bet is to optionally attempt to import the `ipaddress` module to do IP address detection.

Failing that, if we want an internal utility for this we can write one: it just turns out to be moderately tricky.

Anyone got opinions on this?
We're hitting this in pywinrm/Ansible as well if SNI is enabled. I'd be +1 for a soft dep on `ipaddress`- we can make it a hard dep for pywinrm just to make sure it's always there for our fairly common use case of someone hitting a WinRM endpoint by IP (in the off chance SNI is enabled on that host)...
This is a known bug in Python's ssl module. I'll fix it for 3.7 as soon as I can drop support for OpenSSL < 1.0.2.

Experimental patch: https://github.com/tiran/cpython/commit/a3fa2cf76dfc5a1976a416627131d4e4c6070b9e#diff-e1cc5bf74055e388cda128367a814c8fR690
@nitzmahone In #1287 you wrote that SNI IP address is causing connection abort with Windows' http.sys. I didn't know that the issue can cause connection problems. Can you please open an issue on bugs.python.org?
@tiran done: https://bugs.python.org/issue32185 - I see it's already been default-assigned to you :D
@nitzmahone Yeah, I'm the poor soul that has to fix all SSL/TLS issues. :)
As a user, I can confirm that this SNI IP address issue seems to be a problem when using pywinrm with Windows Server 2012.

I was using IP addresses for two Windows Server 2012 boxes. One was connecting and one was failing with a SSL handshake error. I look at the packets, and when my client sent the IP address, the server rejected the packet. (Same thing worked on the other box though.)

When I hard-coded random hostnames into /etc/hosts with those IP addresses, it worked a treat. Winrm is listening on all addresses and doesn't seem to care what hostname it has so long as it matches the SSL certificate. 

I'm curious now how one determines if SNI is enabled or not... with WinRM or Windows in general. All I see is info about IIS which I assume isn't relevant int he case of WinRM...


While using IP addresses, client would send a SYN, server would send a SYN-ACK, client would send an ACK, and then the client would send a push packet containing the IP addresses and the server would send a reset packet. 

As soon as I switched to using hostnames, instead of a reset packet, I would get a packet that appeared to contain the SSL certificate or at least some details about it. 

Note that this latter behaviour also happened on one of the Windows machines even when using IP addresses.

Trying to figure out how SNI relates to WinRM...
Ahh... I take it that both IIS and WinRM use http.sys...
ayup
Still no idea what the difference between those machines was as I think http.sys has SNI enabled by default? The client was exactly the same so I don't see how that could have impacted it. 

I'm guessing maybe it had to do with some IIS sites and differences in configuration there between the servers?
It doesn't, but it's global- if one listener enables the SNI extension, http.sys will accept/process it on all of them- otherwise it just ignores it.
  
Hmm, that doesn't seem to be the case here. It looks like there is one site on the 2nd server with SNI enabled, but I wasn't having any problems with WinRM just using an IP address on that server :/.
@Lukasa, given that this appears to be something that'll be fixed in Python 3.7, do you think we should proceed with a fix for this in our own code? I agree that if so, a soft dependency on `ipaddress` makes sense.
@haikuginger I'm planning to backport the fix to 2.7 and 3.6, too. It will work with most common setups, that is either OpenSSL >= 1.0.2 or LibreSSL >= 2.5.3 or or platforms with ``inet_pton()`` in libc.

You don't need the ``ipaddress`` module. You can simply use the socket module to detect if a string is an IPv4 or IPv6 address. It's faster, too. I proposed a solution on another PR or issue.

```
import socket
import sys

def check_ipaddr(hostname):
    if sys.version_info >= (3, 0) and isinstance(hostname, bytes):
        # IDN A-label bytes are ASCII compatible
        hostname = hostname.decode('ascii')
    families = [socket.AF_INET]
    if hasattr(socket, 'AF_INET6'):
        families.append(socket.AF_INET6)
    for af in families:
        try:
            socket.inet_pton(af, hostname)
        except socket.error:
            pass
        else:
            return True
    return False
```

```
>>> check_ipaddr('::1')
True
>>> check_ipaddr('127.0.0.1')
True
>>> check_ipaddr(b'127.0.0.1')
True
>>> check_ipaddr('www.python.org')
False
>>> check_ipaddr(b'www.python.org')
False
```
https://bugs.python.org/issue32185 has landed in CPython branches 2.7, 3.6, 3.7, and master.

Resolving patch:
diff --git a/urllib3/util/ssl_.py b/urllib3/util/ssl_.py
--- a/urllib3/util/ssl_.py
+++ b/urllib3/util/ssl_.py
@@ -2,11 +2,13 @@
 import errno
 import warnings
 import hmac
+import socket
 
 from binascii import hexlify, unhexlify
 from hashlib import md5, sha1, sha256
 
 from ..exceptions import SSLError, InsecurePlatformWarning, SNIMissingWarning
+from ..packages import six
 
 
 SSLContext = None
@@ -53,6 +55,27 @@ def _const_compare_digest_backport(a, b):
     OP_NO_SSLv2, OP_NO_SSLv3 = 0x1000000, 0x2000000
     OP_NO_COMPRESSION = 0x20000
 
+
+# Python 2.7 and earlier didn't have inet_pton on non-Linux
+# so we fallback on inet_aton in those cases. This means that
+# we can only detect IPv4 addresses in this case.
+if hasattr(socket, 'inet_pton'):
+    inet_pton = socket.inet_pton
+else:
+    # Maybe we can use ipaddress if the user has urllib3[secure]?
+    try:
+        import ipaddress
+
+        def inet_pton(_, host):
+            if isinstance(host, six.binary_type):
+                host = host.decode('ascii')
+            return ipaddress.ip_address(host)
+
+    except ImportError:  # Platform-specific: Non-Linux
+        def inet_pton(_, host):
+            return socket.inet_aton(host)
+
+
 # A secure default.
 # Sources for more information on TLS ciphers:
 #
@@ -325,17 +348,49 @@ def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
 
     if certfile:
         context.load_cert_chain(certfile, keyfile)
-    if HAS_SNI:  # Platform-specific: OpenSSL with enabled SNI
-        return context.wrap_socket(sock, server_hostname=server_hostname)
-
-    warnings.warn(
-        'An HTTPS request has been made, but the SNI (Server Name '
-        'Indication) extension to TLS is not available on this platform. '
-        'This may cause the server to present an incorrect TLS '
-        'certificate, which can cause validation failures. You can upgrade to '
-        'a newer version of Python to solve this. For more information, see '
-        'https://urllib3.readthedocs.io/en/latest/advanced-usage.html'
-        '#ssl-warnings',
-        SNIMissingWarning
-    )
+
+    # If we detect server_hostname is an IP address then the SNI
+    # extension should not be used according to RFC3546 Section 3.1
+    # We shouldn't warn the user if SNI isn't available but we would
+    # not be using SNI anyways due to IP address for server_hostname.
+    if ((server_hostname is not None and not is_ipaddress(server_hostname))
+            or IS_SECURETRANSPORT):
+        if HAS_SNI and server_hostname is not None:
+            return context.wrap_socket(sock, server_hostname=server_hostname)
+
+        warnings.warn(
+            'An HTTPS request has been made, but the SNI (Server Name '
+            'Indication) extension to TLS is not available on this platform. '
+            'This may cause the server to present an incorrect TLS '
+            'certificate, which can cause validation failures. You can upgrade to '
+            'a newer version of Python to solve this. For more information, see '
+            'https://urllib3.readthedocs.io/en/latest/advanced-usage.html'
+            '#ssl-warnings',
+            SNIMissingWarning
+        )
+
     return context.wrap_socket(sock)
+
+
+def is_ipaddress(hostname):
+    """Detects whether the hostname given is an IP address.
+
+    :param str hostname: Hostname to examine.
+    :return: True if the hostname is an IP address, False otherwise.
+    """
+    if six.PY3 and isinstance(hostname, six.binary_type):
+        # IDN A-label bytes are ASCII compatible.
+        hostname = hostname.decode('ascii')
+
+    families = [socket.AF_INET]
+    if hasattr(socket, 'AF_INET6'):
+        families.append(socket.AF_INET6)
+
+    for af in families:
+        try:
+            inet_pton(af, hostname)
+        except (socket.error, ValueError, OSError):
+            pass
+        else:
+            return True
+    return False


Test patch:
diff --git a/test/test_ssl.py b/test/test_ssl.py
new file mode 100644
--- /dev/null
+++ b/test/test_ssl.py
@@ -0,0 +1,73 @@
+import mock
+import pytest
+from six import b
+from urllib3.util import ssl_
+from urllib3.exceptions import SNIMissingWarning
+
+
+@pytest.mark.parametrize('addr', [
+    '::1',
+    '::',
+    '127.0.0.1',
+    '8.8.8.8',
+    b('127.0.0.1')
+])
+def test_is_ipaddress_true(addr):
+    assert ssl_.is_ipaddress(addr)
+
+
+@pytest.mark.parametrize('addr', [
+    'www.python.org',
+    b('www.python.org')
+])
+def test_is_ipaddress_false(addr):
+    assert not ssl_.is_ipaddress(addr)
+
+
+@pytest.mark.parametrize(
+    ['has_sni', 'server_hostname', 'uses_sni'],
+    [(True, '127.0.0.1', False),
+     (False, 'www.python.org', False),
+     (False, '0.0.0.0', False),
+     (True, 'www.google.com', True),
+     (True, None, False),
+     (False, None, False)]
+)
+def test_context_sni_with_ip_address(monkeypatch, has_sni, server_hostname, uses_sni):
+    monkeypatch.setattr(ssl_, 'HAS_SNI', has_sni)
+
+    sock = mock.Mock()
+    context = mock.create_autospec(ssl_.SSLContext)
+
+    ssl_.ssl_wrap_socket(sock, server_hostname=server_hostname, ssl_context=context)
+
+    if uses_sni:
+        context.wrap_socket.assert_called_with(sock, server_hostname=server_hostname)
+    else:
+        context.wrap_socket.assert_called_with(sock)
+
+
+@pytest.mark.parametrize(
+    ['has_sni', 'server_hostname', 'should_warn'],
+    [(True, 'www.google.com', False),
+     (True, '127.0.0.1', False),
+     (False, '127.0.0.1', False),
+     (False, 'www.google.com', True),
+     (True, None, False),
+     (False, None, False)]
+)
+def test_sni_missing_warning_with_ip_addresses(monkeypatch, has_sni, server_hostname, should_warn):
+    monkeypatch.setattr(ssl_, 'HAS_SNI', has_sni)
+
+    sock = mock.Mock()
+    context = mock.create_autospec(ssl_.SSLContext)
+
+    with mock.patch('warnings.warn') as warn:
+        ssl_.ssl_wrap_socket(sock, server_hostname=server_hostname, ssl_context=context)
+
+    if should_warn:
+        assert warn.call_count >= 1
+        warnings = [call[0][1] for call in warn.call_args_list]
+        assert SNIMissingWarning in warnings
+    else:
+        assert warn.call_count == 0
diff --git a/test/test_util.py b/test/test_util.py
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -500,7 +500,8 @@ def test_ssl_wrap_socket_with_no_sni_warns(self):
         ssl_.HAS_SNI = False
         try:
             with patch('warnings.warn') as warn:
-                ssl_wrap_socket(ssl_context=mock_context, sock=socket)
+                ssl_wrap_socket(ssl_context=mock_context, sock=socket,
+                                server_hostname='www.google.com')
             mock_context.wrap_socket.assert_called_once_with(socket)
             assert warn.call_count >= 1
             warnings = [call[0][1] for call in warn.call_args_list]
diff --git a/test/with_dummyserver/test_https.py b/test/with_dummyserver/test_https.py
--- a/test/with_dummyserver/test_https.py
+++ b/test/with_dummyserver/test_https.py
@@ -279,14 +279,7 @@ def test_unverified_ssl(self):
             # the unverified warning. Older systems may also emit other
             # warnings, which we want to ignore here.
             calls = warn.call_args_list
-            if sys.version_info >= (2, 7, 9) or util.IS_PYOPENSSL \
-                    or util.IS_SECURETRANSPORT:
-                category = calls[0][0][1]
-            elif util.HAS_SNI:
-                category = calls[1][0][1]
-            else:
-                category = calls[2][0][1]
-            self.assertEqual(category, InsecureRequestWarning)
+            self.assertIn(InsecureRequestWarning, [x[0][1] for x in calls])
 
     def test_ssl_unverified_with_ca_certs(self):
         pool = HTTPSConnectionPool(self.host, self.port,