Latest version broken?  (pyopenssl 0.13)

Seems like the release was done against pyopenssl 0.11 whereas the latest version is pyopenssl 0.13. With pyopenssl 0.13, pip install requests broke on debian 7.2. I could fix it by just moving to pyopenssl 0.11. (Not sure it works, it did install)

http://i.imgur.com/O0llHOB.png


Thanks for saving us again @shazow and @t-8ch.

Also, cross repo automatic issue closing? Very cool.


Thanks guys. I manually patched urllib3 and it works from requests. But requests : both the pip and git version is still broken with the same error. 


Here is another problem:

> > > requests.get("https://register.blib.us", verify=False)
> > > <Response [200]>
> > > requests.get("https://register123.blib.us", verify=False)
> > > ... hangs

This command hangs with or without timeout. This is the patched version with a fresh urllib3. The following two tracebacks might help (Fresh debian 7.2 install with pip installed pyopenssl/requests/ndg-...):

> > > requests.get("https://register123.blib.us", verify=False)
> > > c♥Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/api.py", line 55, i
> > > n get
> > >     return request('get', url, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/api.py", line 44, i
> > > n request
> > >     return session.request(method=method, url=url, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/sessions.py", line
> > > 361, in request
> > >     resp = self.send(prep, *_send_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/sessions.py", line
> > > 464, in send
> > >     r = adapter.send(request, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/adapters.py", line
> > > 321, in send
> > >     timeout=timeout
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnectionpool.py", line 471, in urlopen
> > >     body=body, headers=headers)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnectionpool.py", line 285, in _make_request
> > >     conn.request(method, url, **httplib_request_kw)
> > >   File "/usr/lib/python2.7/httplib.py", line 962, in request
> > >     self._send_request(method, url, body, headers)
> > >   File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
> > >     self.endheaders(body)
> > >   File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
> > >     self._send_output(message_body)
> > >   File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
> > >     self.send(msg)
> > >   File "/usr/lib/python2.7/httplib.py", line 780, in send
> > >     self.connect()
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnection.py", line 95, in connect
> > >     ssl_version=resolved_ssl_version)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > ntrib/pyopenssl.py", line 338, in ssl_wrap_socket
> > >     cnx.do_handshake()
> > > KeyboardInterrupt
> > > 
> > > requests.get("https://register123.blib.us", verify=False, timeout=5.0)
> > > ♥Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/api.py", line 55, i
> > > n get
> > >     return request('get', url, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/api.py", line 44, i
> > > n request
> > >     return session.request(method=method, url=url, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/sessions.py", line
> > > 361, in request
> > >     resp = self.send(prep, *_send_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/sessions.py", line
> > > 464, in send
> > >     r = adapter.send(request, *_kwargs)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/adapters.py", line
> > > 321, in send
> > >     timeout=timeout
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnectionpool.py", line 471, in urlopen
> > >     body=body, headers=headers)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnectionpool.py", line 285, in _make_request
> > >     conn.request(method, url, **httplib_request_kw)
> > >   File "/usr/lib/python2.7/httplib.py", line 962, in request
> > >     self._send_request(method, url, body, headers)
> > >   File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
> > >     self.endheaders(body)
> > >   File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
> > >     self._send_output(message_body)
> > >   File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
> > >     self.send(msg)
> > >   File "/usr/lib/python2.7/httplib.py", line 780, in send
> > >     self.connect()
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > nnection.py", line 95, in connect
> > >     ssl_version=resolved_ssl_version)
> > >   File "/usr/local/lib/python2.7/dist-packages/requests-2.0.1-py2.7.egg/requests/packages/urllib3/co
> > > ntrib/pyopenssl.py", line 340, in ssl_wrap_socket
> > >     select.select([sock], [], [])
> > > KeyboardInterrupt


@pikumar I can't reproduce this at all on a new Debian 7.2 container.
`register123.blip.us` resolves for me to `176.74.176.178`.
Is this the whole testscript?
Are you using some form of async environment (gevent etc)?
Maybe you are behind a firewall/proxy which drops packets with the `TCP` `RST` flag set?


@t-8ch Thanks for the quick reply. Perhaps there was a typo in your request? Please try : requests.get("https://register123.blib.us", verify=False) - you seem to be trying blip.us? register123.blib.us should resolve to  216.230.230.76.


@pikumar You are right, I had the wrong domain in the container (while testing on my main box I got it right)
But on the other hand the behaviour with the right domain is the same as with the wrong one.
`TCP RST` followed by a an exception from requests.
You could use wireshark to sniff the traffic and analyse or upload the sniffed data.


@t-8ch Indeed wget/curl/requests all hang on this new box. Looking into it. 


@t-8ch This is what is happening on wireshark: http://cloudshark.org/captures/7e86057afd34 (Its a debian Virtualbox VM under Win7 = Box 1). openssl version 1.0.1e. With openssl version 1.0.0 on another box (Box 2), things look fine (A non-existent https+sni domain access gives --> curl: (35) Unknown SSL protocol error in connection to register123.blib.us:443) . For Box 1, this is what a correct url + successful connection looks like : http://cloudshark.org/captures/687634100c81. (Server = https+SNI).

On Box 1, requests/wget/curl all hang with : https://non-existant-domain.blib.us - but this fixes the hang for wget:
wget ----secure-protocol=SSLv3 https://non-existant-domain.blib.us

It might not be a firewall issue? It probably is an openssl/tls issue? It might be my server not handling TLS 1.2 properly - even in that case, the clients should not hang I would think. (Remember, for existing domains, everything seems to be fine (both box 1/2), its only the non-existent domains that hang things)


@pikumar I don't see SNI in your requests at all. There should be a `Extension: server_name` in your `Client hello`.
My Debian does this.
The trace from your hanging request looks like mine from the working one.
Where did you take the trace? Could it be the `RST` packet gets dropped after that?
(Like you traced on the Windows System and the Debian box itself does some firewalling)

To ask again: You are not using gevent or anything like that?

You can also force requests to use `SSLv3`.
Like always theres a blog post by @Lukasa for that: http://lukasa.co.uk/2013/01/Choosing_SSL_Version_In_Requests/


Top three most useful blog posts I ever wrote, that one.


@t-8ch http://cloudshark.org/captures/1f95da7bf5a7 - this is a capture from inside the debian box. And you are right: I do not see the RST in there. What baffles me, is that my iptables is completely open. Everything else works as it should, except non-existent domains. 

Your answers:

SNI : Indeed register.blib.us is not SNIed, but any existing name is: for example https://rrc.imp.blib.us should be.

I am not using gevent, unless requests automatically picks it up when ndg-.., pyasn1,... is installed. I did install the dependencies in urllib3 that helps with SNI. 


@pikumar requests doesn't automatically pick anything up. @t-8ch was only asking because you hadn't answered earlier and wanted to make sure this wasn't one of the few issues we're starting to see with requests + gevent.


@pikumar Yes, but your `Client Hello` should contain the `servername` extensions. This should happen if you have the mentioned dependencies installed (ndg-, pyasn1, pyopenssl). So it seems the installation of those isn't correct.

However this has no relation to the missing `RST` packets. And it's most likely out of scope for this bugtracker.

I would nevertheless like to know if you in fact used the pyopenssl codepath (missing SNI extension suggests otherwise) because the request with specified timeout should probably not hang forever.