Repository: urllib3/urllib3

Issue:
Need to update error handling for pyOpenSSL
If an `SSLError` comes from `pyOpenSSL`, we attempt to determine if it's an error due to request timeouts on python 2.6 being raised as `BaseSSLErrors` [here](https://github.com/shazow/urllib3/blob/29aa09bde9c42cc9a8d79aac47ee3d362b438cca/urllib3/connectionpool.py#L314). If the handshake ends unexpectedly under pyOpenSSL, then we get an `SSLError` that has a captured `SysCallError` which when you call `str(SSLError(msg, SysCallError(code, msg)))` you get

``` py
TypeError: __str__ returned non-string (type SysCallError)
```

So we might want to update how we handle SSLErrors to accomodate pyOpenSSL. To be clear, I observed this on: python 2.7.9 with openssl 1.0.1l (from brew) and I was using requests to debug an issue with `https://apissl.cloudfactory.com`, e.g., `requests.get('https://apissl.cloudfactory.com')`



Discussion:
Yea that whole process is voodoo. :/

Open to basically any kind of change. What do you have in mind?

I don't know yet. =P

Somedays I wonder if we should just fork out to golang. :P

A note that pyOpenSSL is no longer required for SNI in Python 2.7.9. =) Obviously I don't think that removes it from the support matrix, but it's worth being aware of.

Right. I was just trying to see if that would help with the problem the user was encountering with apissl.cloudfactory.com. Works just fine otherwise but it blows up here.

Blows up to other api's as well but I'm not at liberty to divulge a uri, would be very interested in a fix for this

I had to fix this on a production server by editing `requests/packages/urllib3/connectionpool.py`. At line 315, before `if 'timed out' in str(err) or 'did not complete (read)' in str(err):` I inserted:

```
if not isinstance(err, str) and hasattr(err, 'strerror'):
        raise ProtocolError(self, url, "{}".format(err.strerror))
```

At least I get some inkling on what's going on - in my case this bubbles up to my application from `requests` as:

> requests.exceptions.ConnectionError: (<requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0x7f3999ba18d0>, '/', "[('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]")

@Lukasa , it looks like https://github.com/shazow/urllib3/commit/e2de9f67f276f5a9c2ca3689ce50eac2c7361f4f fixes this, or maybe it only fixes one aspect of this problem.

The cause is 2.7.9 introduced a new `SSLError.__str__` implementation that returns `strerror` , which means it must be a string.
https://hg.python.org/cpython/rev/92163#l23.297

The last release of urllib3, and requests, still has the problematic code
https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/contrib/pyopenssl.py#L290

I did a sync of pyopenssl.py only into requests, and it passed all tests.  But perhaps urllib3 is planning to release another version soon...?

Requests has a policy about updating urllib3 releases, and part of it is that we do not force urllib3 to issue new releases. urllib3 will release when it's ready, and requests will release sometime after that. =)

Appreciate that.  Is there any policy on urllib3 point releases, or any clues as to when urllib3 might be ready to release this fix?  Are there other issues that are 'must haves' for the next point release of urllib3?

urllib3 is busted on Python 2.7.9-10 if pyopenssl is installed - it fails whenever an invalid certificate is encountered, or other ssl errors.  fwiw this was introduced in the very first version (953b5a41) of pyopenssl support by @kirkeby, so there is no available version of urllib3 which can happily use pyopenssl on Python 2.7.9-10.

I think urllib3 is likely to issue a point release fairly soon, if only because it's been a little while. That said, @shazow is the real authority here, I'm just the help. ;)

Opened a milestone here, aiming for early next week for v1.11 if we can get those last few issues in (code reviews will help speed it up, please take a look!): https://github.com/shazow/urllib3/milestones/v1.11

Thank you guys for your hard work so far on this problem :smile: . Sorry for the original issue - the problems are related to `requests` and not `urllib3` - I tested with version 1.11 and I'm able to successfully complete the requests. 

@Lukasa sorry to tag you over here about the issue, but do you have any info for if/when this fix will make its way into requests? I still get this error when making my requests with `requests`.

Requests 2.8.0 is imminent, but sadly right now I'm on holiday which means I have limited time available to make software releases. We _may_ be able to get one out today, otherwise it'll hopefully be early next week: I'll co-ordinate with @sigmavirus24 to see if we can make that happen.

@Lukasa and @sigmavirus24  don't worry / rush! I was just checking in on the beloved `requests` since I ran into this today while doing some https requests. Enjoy your holiday and relax please! Thank you guys very much.

Is this still happening with requests 2.8.0? 

Still happening on requests[security] 2.9.1. Is there an ETA for this fix? Thanks!

@borbamartin I do not believe this problem is still occurring on requests 2.9.1, because requests 2.9.1 contains urllib3 v1.13.1: this was fixed in urllib3 1.11. If you think you're suffering from a bug here, please open an issue on requests and we'll discuss further.

Oh, I'm sorry, I don't know what I'm talking about: it's unclear that this has been fixed at all. Let me dig into it for a second.

Though please note that this should have been at least partially fixed in urllib3 1.11. Can you please provide more details about what problem you're encountering, what version of requests you're using, where you got it from, and ideally some repro code?

Lol, sure. I had to install requests[security] a couple of days ago to fix a handshake issue on a project, and now on another project which uses the same libraries I'm getting the error mentioned on this issue, and after uninstalling the security package and re-installing the regular requests package, it goes away.

I'm getting this when sending a simple GET request to a TLS 1.0 connection using TLS_RSA_WITH_RC4_128_MD5:
File ".../lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)

This are my packages:
requests (2.9.1)
pyOpenSSL (0.15.1)

Unfortunately I cannot provide a repro code because it's a private server :(

Thanks in advance!

You can get the same libraries (I assume are the latest ones) by 'pip install requests[security]'

Hmm. Can you find line 185 of `requests/packages/urllib3/contrib/pyopenssl.py` and change it to read `raise SocketError(str(e))`?

Changing that line returns a:
File ".../lib/python2.7/site-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: SysCallError(54, 'ECONNRESET')",)

Ok, that looks like the correct behaviour to me. Are you open to providing a patch for this (which will include writing a test, which may be tricky)? Or would you like to hold off and see if I can find time to do it.

I'll have to hold off, this issue is still not solved for me, now I have to deal with the other error :( Thanks for your help @Lukasa !


Resolving patch:
diff --git a/urllib3/contrib/pyopenssl.py b/urllib3/contrib/pyopenssl.py
--- a/urllib3/contrib/pyopenssl.py
+++ b/urllib3/contrib/pyopenssl.py
@@ -182,7 +182,7 @@ def recv(self, *args, **kwargs):
             if self.suppress_ragged_eofs and e.args == (-1, 'Unexpected EOF'):
                 return b''
             else:
-                raise SocketError(e)
+                raise SocketError(str(e))
         except OpenSSL.SSL.ZeroReturnError as e:
             if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
                 return b''


Test patch: