Repository: urllib3/urllib3

Issue:
conn.close() can raise OpenSSL exception after idle timeout
When returning a connection to the pool, urllib3 calls `conn.close()`:

```
        if conn and is_connection_dropped(conn):
            log.info("Resetting dropped connection: %s" % self.host)
            conn.close()
```

This can raise an exception under the following conditions:
- urllib3 is using pyOpenSSL and https
- Remote webserver is running Windows IIS (7.5 and 8 tested)
- KeepAlives are active (default for IIS)
- Connection drops after idle timeout

tcpdump shows that the remote IIS host sends a RST on idle timeout. This appears to be IIS-specific behaviour (Apache2 and haproxy both close the connection normally)

At that point nothing bad happens immediately. However the connection remains in the pool. The next time you send a request to this host, it is reused; some data is sent down it; this then raises an `OpenSSL.SSL.SysCallError` exception. I am guessing that it it is trying to send an SSL shutdown message down the already-closed socket.

More detail at https://github.com/zatosource/zato/issues/465

Proposed patch:

```
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -239,11 +239,15 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
         # If this is a persistent connection, check if it got disconnected
         if conn and is_connection_dropped(conn):
             log.info("Resetting dropped connection: %s" % self.host)
-            conn.close()
-            if getattr(conn, 'auto_open', 1) == 0:
-                # This is a proxied connection that has been mutated by
-                # httplib._tunnel() and cannot be reused (since it would
-                # attempt to bypass the proxy)
+            try:
+                conn.close()
+                if getattr(conn, 'auto_open', 1) == 0:
+                    # This is a proxied connection that has been mutated by
+                    # httplib._tunnel() and cannot be reused (since it would
+                    # attempt to bypass the proxy)
+                    conn = None
+            except Exception as e:
+                log.info("Closing connection failed: %s, force new connection" % str(e))
                 conn = None

         return conn or self._new_conn()
```



Discussion:
I have been working on a standalone reproducer for this problem. So far I have only been able to reproduce using [zato](https://github.com/zatosource/zato), which loads about 160 libraries including things like `gevent` and `twisted`

Preparation:
- Build a Windows 2012R2 Server in EC2. Enable IIS and HTTPS with self-signed cert. Take a note of its IP (x.x.x.x).
- You may also wish to reduce the [keep-alive timeout](http://serverfault.com/questions/136306/what-is-the-iis7-default-keepalive-time)
  from 120 down to 10 seconds (`IIS Manager > Sites > Default Web Site > Advanced Settings > Limits > Connection Timeout`)
- Setup client test environment: ubuntu 14.04, python 2.7.6. I did this inside a second EC2 VM so as to ensure no intervening NAT
- In a session on client test environment: `tcpdump -i eth0 -nn host x.x.x.x`

Reproducing under zato is straightforward: install the zato Ubuntu package, create the followings script:

```
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()

import urllib3, time
conn = urllib3.connection_from_url('https://x.x.x.x/')
print time.asctime()
r = conn.request('GET', '/')
r.close()
time.sleep(15)  # or 140, if you didn't reduce the timeout
print time.asctime()
r = conn.request('GET', '/')
r.close()
print "Problem did not reproduce :-("
```

And run:

```
zato@prov:~$ /opt/zato/2.0.5/bin/py ~bcandler/test-https.py
Wed Oct 28 11:22:52 2015
/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request ...
  InsecureRequestWarning)
Wed Oct 28 11:23:07 2015
Traceback (most recent call last):
  File "/opt/zato/2.0.5/bin/py", line 197, in <module>
    execfile(__file__)
  File "/home/bcandler/test-https.py", line 11, in <module>
    r = conn.request('GET', '/')
  File "/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/connectionpool.py", line 533, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/connectionpool.py", line 239, in _get_conn
    conn.close()
  File "/usr/lib/python2.7/httplib.py", line 786, in close
    self.sock.close()   # close it manually... there may be other refs
  File "/opt/zato/2.0.5/eggs/urllib3-1.10.4-py2.7.egg/urllib3/contrib/pyopenssl.py", line 213, in close
    return self.connection.shutdown()
  File "/opt/zato/2.0.5/eggs/pyOpenSSL-0.15.1-py2.7.egg/OpenSSL/SSL.py", line 1522, in shutdown
    self._raise_ssl_error(self._ssl, result)
  File "/opt/zato/2.0.5/eggs/pyOpenSSL-0.15.1-py2.7.egg/OpenSSL/SSL.py", line 1178, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')
```

Watch the tcpdump output. Check you see the the RST when the session times out:

```
09:58:59.421390 IP x.x.x.x.443 > c.c.c.c.51092: Flags [R.], seq 998, ack 133, win 0, length 0
```

NOTE: do not run the test program inside a VirtualBox VM behind VirtualBox's NAT, because this will convert the external RST into F (FIN) and this won't reproduce the problem:

```
09:14:42.050392 IP x.x.x.x.443 > 10.0.2.15.51957: Flags [F.], seq 2280, ack 603, win 65535, length 0
09:14:42.087059 IP 10.0.2.15.51957 > x.x.x.x.443: Flags [.], ack 2281, win 34075, length 0
```

I'm still working on reproducing this inside a virtualenv without zato.


Resolving patch:
diff --git a/urllib3/contrib/pyopenssl.py b/urllib3/contrib/pyopenssl.py
--- a/urllib3/contrib/pyopenssl.py
+++ b/urllib3/contrib/pyopenssl.py
@@ -217,7 +217,10 @@ def shutdown(self):
 
     def close(self):
         if self._makefile_refs < 1:
-            return self.connection.close()
+            try:
+                return self.connection.close()
+            except OpenSSL.SSL.Error:
+                return
         else:
             self._makefile_refs -= 1
 


Test patch: