Repository: urllib3/urllib3

Issue:
urllib3 keeps connection open after failed fingerprint validation
1.  Launch a single-threaded HTTPS server, like the one from Werkzeug/Flask, with ad-hoc certificate generation.
2.  Paste this into a file called lol.py:
   
   ```
   from urllib3.poolmanager import PoolManager
   
   def x():
       # This is GitHub's certificate fingerprint. This function should not exit successfully.
       fingerprint = 'A0:C4:A7:46:00:ED:A7:2D:C0:BE:CB:9A:8C:B6:07:CA:58:EE:74:5E'  
       p = PoolManager(assert_fingerprint=fingerprint)
       r = p.request('GET', 'https://127.0.0.1:5000')
       p.clear()  # doesn't seem to matter
   ```
3.  Launch `python -i lol.py`, in a Python 3 virtualenv with the latest PyPI-release of urllib3 installed.
4.  Execute `x()` once, an exception for the mismatching fingerprint is thrown.
5.  Execute `x()` again, the function doesn't terminate. Werkzeug is not responding to it because it is busy with another connection, but that one should've been closed.

Notes:
- If one kills the Python process where urllib3 is used, the connections are terminated and the server is usable again.
- This is Python-3-specific.
- This was found during usage with requests, which has the exact same problem. See https://github.com/untitaker/vdirsyncer/pull/161



Discussion:
Also, here's a simple Flask app which will suffice for this test:

```
import flask
app = flask.Flask(__name__)

@app.route('/')
def index():
    return "haha!"

app.run(ssl_context='adhoc')
```

+1 to closing connections after a failed validation. Should be reasonably easy to port this test to our test_socketlevel.py suite.

@untitaker Would you like to work on this PR to fix it?

Yes, but please don't expect results until next week.

@untitaker This is open source, we expect nothing but love. :)

Not sure where to start. The code at https://github.com/shazow/urllib3/blob/master/urllib3/connection.py#L241 raises the exception which hangs the connection, but I am not sure why it would cause any problems... `match_hostname` also raises exceptions of the same type, and they don't cause problems. Or do they?

Hm not sure why an exception raised would hang a connection.

@Lukasa @sigmavirus24 Any thoughts?

Raised exceptions can cause the interpreter to fail to exit in a threaded case. If we're not talking threading, the only thing I could think of is that the connection gets re-used after having been left in a bad way. Otherwise...

The example above is not threaded though.

In fact this works, in the sense that certificate verification fails but no connections are leaked:

```
from urllib3.poolmanager import PoolManager
import certifi

def x():
    p = PoolManager(cert_reqs='CERT_REQUIRED', ca_certs=certifi.where())

    r = p.request('GET', 'https://127.0.0.1:5000')
    p.clear()  # doesn't seem to matter
```

It seems though that the traceback for this test is originating from somewhere completely different.

@untitaker maybe you are hitting #524?
(sorry I am on my phone)

Yeah, I also just noticed in a local test, but the example above doesn't use it.

What completely irritates me that CA verification is already done when calling `ssl_wrap_socket` https://github.com/shazow/urllib3/blob/master/urllib3/connection.py#L234, while a few lines below, the same thing is done.

For this kind of TB to obtain, I had to apply [this patch](https://gist.github.com/untitaker/c0df8493db7fba23799c).

Have you seen #529?

Lol that's the issue you're posting in.

Hmpf, indeed. I meant #526 

It seems that raising any exceptions in `Connection.connect` after calling `self.sock.getpeercert` causes this connection leak.

Applying #526 doesn't help with both examples.

This fixes the issue for me:

``` diff
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
index 8bdf228..98b75f8 100644
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -549,7 +549,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
             # Timed out by queue.
             raise EmptyPoolError(self, "No pool connections are available.")

-        except (BaseSSLError, CertificateError) as e:
+        except (BaseSSLError, CertificateError, SSLError) as e:
             # Close the connection. If a connection is reused on which there
             # was a Certificate error, the next request will certainly raise
             # another Certificate error.
```

The problem is, that `BaseSSLError` comes from the stdlib `ssl` module and
`CertificateError` from `backports.ssl`. So we don't catch our own stuff.
Maybe we should clean this up.
And get a test.

Neat. Working on PR...


Resolving patch:
diff --git a/dummyserver/server.py b/dummyserver/server.py
--- a/dummyserver/server.py
+++ b/dummyserver/server.py
@@ -50,6 +50,7 @@ class SocketServerThread(threading.Thread):
     def __init__(self, socket_handler, host='localhost', port=8081,
                  ready_event=None):
         threading.Thread.__init__(self)
+        self.daemon = True
 
         self.socket_handler = socket_handler
         self.host = host
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -558,6 +558,14 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
                 conn = None
             raise SSLError(e)
 
+        except SSLError:
+            # Treat SSLError separately from BaseSSLError to preserve
+            # traceback.
+            if conn:
+                conn.close()
+                conn = None
+            raise
+
         except (TimeoutError, HTTPException, SocketError, ConnectionError) as e:
             if conn:
                 # Discard the connection for these exceptions. It will be


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -521,6 +521,43 @@ def socket_handler(listener):
         finally:
             timed_out.set()
 
+    def test_ssl_failed_fingerprint_verification(self):
+        def socket_handler(listener):
+            for i in range(2):
+                sock = listener.accept()[0]
+                ssl_sock = ssl.wrap_socket(sock,
+                                           server_side=True,
+                                           keyfile=DEFAULT_CERTS['keyfile'],
+                                           certfile=DEFAULT_CERTS['certfile'],
+                                           ca_certs=DEFAULT_CA)
+
+                ssl_sock.send(b'HTTP/1.1 200 OK\r\n'
+                              b'Content-Type: text/plain\r\n'
+                              b'Content-Length: 5\r\n\r\n'
+                              b'Hello')
+
+                ssl_sock.close()
+                sock.close()
+
+        self._start_server(socket_handler)
+        # GitHub's fingerprint. Valid, but not matching.
+        fingerprint = ('A0:C4:A7:46:00:ED:A7:2D:C0:BE:CB'
+                       ':9A:8C:B6:07:CA:58:EE:74:5E')
+
+        def request():
+            try:
+                pool = HTTPSConnectionPool(self.host, self.port,
+                                           assert_fingerprint=fingerprint)
+                response = pool.urlopen('GET', '/', preload_content=False,
+                                        timeout=Timeout(connect=1, read=0.001))
+                response.read()
+            finally:
+                pool.close()
+
+        self.assertRaises(SSLError, request)
+        # Should not hang, see https://github.com/shazow/urllib3/issues/529
+        self.assertRaises(SSLError, request)
+
 
 def consume_socket(sock, chunks=65536):
     while not sock.recv(chunks).endswith(b'\r\n\r\n'):