Repository: urllib3/urllib3

Issue:
Match_hostname: hostname 'github.com.' doesn't match 'github.com' or 'www.github.com'
I'm getting hostname mismatch error when I try to access a domain with a trailing dot.

Example:

Version of urllib3 I'm using

```
(venv) ➜  ~ pip list --format=columns | grep urllib3
urllib3                      1.22    
```

```
(venv) ➜  ~ python
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib3
>>> http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED')
>>> r = http.request('GET', 'https://github.com/')
>>> print r.status
200
>>> r = http.request('GET', 'https://github.com./')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/karteek/venv/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='github.com.', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'github.com.' doesn't match either of 'github.com', 'www.github.com'",),))
```

Similar request using curl seems to go fine.

```
curl -v https://github.com./ 
*   Trying 192.30.255.112...
* Connected to github.com (192.30.255.112) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: github.com
* Server certificate: DigiCert SHA2 Extended Validation Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET / HTTP/1.1
> Host: github.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Thu, 31 Aug 2017 21:06:01 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Server: GitHub.com
< Status: 200 OK
< Cache-Control: no-cache
```

Also, going to similar URL https://github.com./robots.txt on browsers Firefox and Chrome doesn't show any SSL errors (They get redirected to https://github.com/robots.txt)


Discussion:
This is an interesting issue! Looking at how cURL does it, it seems like it's internally setting the Host header to remove the trailing dot, as well as removing the trailing dot when doing hostname validation on the cert.

I did some quick checking on `getaddrinfo` (which we use for DNS resolution), and it seems like it does both check my system's search domain list when the dot isn't present, and respect the trailing dot (indicating an FQDN) to indicate an absolute domain when it is present, so we definitely want to make any transformations post-resolution.

It's unclear to me whether we should just need to make the modification needed for SSL certification [somewhere around here](https://github.com/shazow/urllib3/blob/4fc0be786662dc5b79bf7e939eedd653c78d4616/urllib3/connection.py#L325) (that'll pass a hostname with trailing dot removed to whatever SSL backend is in use for the purposes of certificate verification) or if we should perform the dot-ectomy change in the Host header we send as well (which would need more careful consideration due to the DNS resolution stuff mentioned above).

As for algorithm, I believe simply doing `host.rstrip('.')` is the minimally-invasive operation we need to remove a trailing dot that indicates an FQDN; as far as I can tell, neither the IPv4 nor IPv6 syntax allows for trailing dots.

@Lukasa, thoughts? I'd be happy to work on a fix for this.
Probably we should almost unconditionally do the dot-ectomy. It is likely to confuse most people to see it and also to cause latent bugs down the line, so probably after DNS resolution it should just be stripped. 
@Lukasa, sounds good. I'll get to work on that; the unfortunate part is that we'll probably need to store the domain twice (once for DNS purposes; once for everything else). 
Nah, let's just create a property that does the transformation for us when we need it. 😁
LAZY EVALUATION FOR THE LAZY DEVELOPMENT GOD

Resolving patch:
diff --git a/dummyserver/testcase.py b/dummyserver/testcase.py
--- a/dummyserver/testcase.py
+++ b/dummyserver/testcase.py
@@ -76,6 +76,23 @@ def tearDownClass(cls):
         if hasattr(cls, 'server_thread'):
             cls.server_thread.join(0.1)
 
+    def assert_header_received(
+        self,
+        received_headers,
+        header_name,
+        expected_value=None
+    ):
+        header_name = header_name.encode('ascii')
+        if expected_value is not None:
+            expected_value = expected_value.encode('ascii')
+        header_titles = []
+        for header in received_headers:
+            key, value = header.split(b': ')
+            header_titles.append(key)
+            if key == header_name and expected_value is not None:
+                self.assertEqual(value, expected_value)
+        self.assertIn(header_name, header_titles)
+
 
 class IPV4SocketDummyServerTestCase(SocketDummyServerTestCase):
     @classmethod
diff --git a/urllib3/connection.py b/urllib3/connection.py
--- a/urllib3/connection.py
+++ b/urllib3/connection.py
@@ -124,6 +124,35 @@ def __init__(self, *args, **kw):
         # Superclass also sets self.source_address in Python 2.7+.
         _HTTPConnection.__init__(self, *args, **kw)
 
+    @property
+    def host(self):
+        """
+        Getter method to remove any trailing dots that indicate the hostname is an FQDN.
+
+        In general, SSL certificates don't include the trailing dot indicating a
+        fully-qualified domain name, and thus, they don't validate properly when
+        checked against a domain name that includes the dot. In addition, some
+        servers may not expect to receive the trailing dot when provided.
+
+        However, the hostname with trailing dot is critical to DNS resolution; doing a
+        lookup with the trailing dot will properly only resolve the appropriate FQDN,
+        whereas a lookup without a trailing dot will search the system's search domain
+        list. Thus, it's important to keep the original host around for use only in
+        those cases where it's appropriate (i.e., when doing DNS lookup to establish the
+        actual TCP connection across which we're going to send HTTP requests).
+        """
+        return self._dns_host.rstrip('.')
+
+    @host.setter
+    def host(self, value):
+        """
+        Setter for the `host` property.
+
+        We assume that only urllib3 uses the _dns_host attribute; httplib itself
+        only uses `host`, and it seems reasonable that other libraries follow suit.
+        """
+        self._dns_host = value
+
     def _new_conn(self):
         """ Establish a socket connection and set nodelay settings on it.
 
@@ -138,7 +167,7 @@ def _new_conn(self):
 
         try:
             conn = connection.create_connection(
-                (self.host, self.port), self.timeout, **extra_kw)
+                (self._dns_host, self.port), self.timeout, **extra_kw)
 
         except SocketTimeout as e:
             raise ConnectTimeoutError(


Test patch:
diff --git a/test/test_connectionpool.py b/test/test_connectionpool.py
--- a/test/test_connectionpool.py
+++ b/test/test_connectionpool.py
@@ -118,6 +118,7 @@ def test_same_host_no_port_https(self, a, b):
         ('google.com', 'https://google.com/'),
         ('yahoo.com', 'http://google.com/'),
         ('google.com', 'https://google.net/'),
+        ('google.com', 'http://google.com./'),
     ])
     def test_not_same_host_no_port_http(self, a, b):
         with HTTPConnectionPool(a) as c:
@@ -130,6 +131,7 @@ def test_not_same_host_no_port_http(self, a, b):
         ('google.com', 'http://google.com/'),
         ('yahoo.com', 'https://google.com/'),
         ('google.com', 'https://google.net/'),
+        ('google.com', 'https://google.com./'),
     ])
     def test_not_same_host_no_port_https(self, a, b):
         with HTTPSConnectionPool(a) as c:
diff --git a/test/test_poolmanager.py b/test/test_poolmanager.py
--- a/test/test_poolmanager.py
+++ b/test/test_poolmanager.py
@@ -31,6 +31,14 @@ def test_same_url(self):
 
         assert conn1 == conn2
 
+        # Ensure that FQDNs are handled separately from relative domains
+        p = PoolManager(2)
+
+        conn1 = p.connection_from_url('http://localhost.:8081/foo')
+        conn2 = p.connection_from_url('http://localhost:8081/bar')
+
+        assert conn1 != conn2
+
     def test_many_urls(self):
         urls = [
             "http://localhost:8081/foo",
diff --git a/test/with_dummyserver/test_https.py b/test/with_dummyserver/test_https.py
--- a/test/with_dummyserver/test_https.py
+++ b/test/with_dummyserver/test_https.py
@@ -65,6 +65,11 @@ def test_simple(self):
         r = self._pool.request('GET', '/')
         self.assertEqual(r.status, 200, r.data)
 
+    def test_dotted_fqdn(self):
+        pool = HTTPSConnectionPool(self.host + '.', self.port)
+        r = pool.request('GET', '/')
+        self.assertEqual(r.status, 200, r.data)
+
     def test_set_ssl_version_to_tlsv1(self):
         self._pool.ssl_version = ssl.PROTOCOL_TLSv1
         r = self._pool.request('GET', '/')
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -1225,6 +1225,37 @@ def socket_handler(listener):
         pool.request('GET', '/', headers=OrderedDict(expected_request_headers))
         self.assertEqual(expected_request_headers, actual_request_headers)
 
+    def test_request_host_header_ignores_fqdn_dot(self):
+
+        received_headers = []
+
+        def socket_handler(listener):
+            sock = listener.accept()[0]
+
+            buf = b''
+            while not buf.endswith(b'\r\n\r\n'):
+                buf += sock.recv(65536)
+
+            for header in buf.split(b'\r\n')[1:]:
+                if header:
+                    received_headers.append(header)
+
+            sock.send((
+                u'HTTP/1.1 204 No Content\r\n'
+                u'Content-Length: 0\r\n'
+                u'\r\n').encode('ascii'))
+
+            sock.close()
+
+        self._start_server(socket_handler)
+
+        pool = HTTPConnectionPool(self.host + '.', self.port, retries=False)
+        self.addCleanup(pool.close)
+        pool.request('GET', '/')
+        self.assert_header_received(
+            received_headers, 'Host', '%s:%s' % (self.host, self.port)
+        )
+
     def test_response_headers_are_returned_in_the_original_order(self):
         # NOTE: Probability this test gives a false negative is 1/(K!)
         K = 16