https GET request fails with "handshake failure"

Related to #1083, perhaps. Standard `requests.get()` for this particular site/page `https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html` results in:

```
>>> import requests
>>> requests.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
```

Using `request-toolbelt`'s `SSLAdapter` to try various ssl versions, they all fail, it would seem... see following tracebacks.

TLSv1:

```
>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
```

SSLv3:

```
>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
```

SSLv2:

```
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)
```

Note the last one gives a `Connection reset by peer` error, which differs from the others, but I'm pretty sure SSLv2 isn't supported by the server anyhow.

For fun, I tried to pass through some more appropriate headers through on the last request as well:

```
>>> headers = {
...     'Accept': u"text/html,application/xhtml+xml,application/xml",
...     'User-Agent': u"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36",
...     'Accept-Encoding': u"gzip,deflate",
...     'Accept-Language': u"en-US,en;q=0.8"
... }
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html', headers=headers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)
```

No dice there either. Here's what the HTTPS connection info in Chrome on Mac looks like:

![screen shot 2014-04-26 at 10 35 21 am](https://cloud.githubusercontent.com/assets/101148/2809354/44aa40b4-cd69-11e3-88f1-fabf089fe238.png)

I'm not positive, but some googling indicates it's likely a cipher list issue, which is more urllib3, I think?

I tried to modify `DEFAULT_CIPHER_LIST` in `pyopenssl`, but started running into import errors. At this point it seemed like things were just broken, and there wasn't really a proper way to approach fixing this yet.

Version information:
OSX Mavericks
Python 2.7.5
OpenSSL 0.9.8y 5 Feb 2013 - (from `python -c "import ssl; print ssl.OPENSSL_VERSION"`)
requests 2.2.1
requests-toolbelt 0.2.0
urllib3 1.8


Sadly, this is unrelated to the issue you identified, and entirely down to the crappy OpenSSL that OS X ships with by default. Version 0.9.8y has some real problems with performing SSL handshakes, and some servers don't tolerate it well. Using Python 3 on my OS X box (therefore using a newer OpenSSL) reveals that there's no problem.

You have two options:
1. Install OpenSSL from Homebrew, then install a new version of Python 2 from Homebrew which will automatically link against the Homebrew-provided OpenSSL.
2. Install OpenSSL from Homebrew, and then install PyOpenSSL against that new version by running  `env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install PyOpenSSL`.


Ah, looks like I was following a red herring then - I don't plan on deploying anything on OSX anyhow. Looks like I'll move my testing to a linux virtualbox. Apologies for this long-winded issue!


No need to apologise, asking that question was the right thing to do: it's bizarrely specific knowledge to know that OS X has this problem. =)


Ok, this is a bummer. I created an Ubuntu 14.04 server 32bit Virtualbox image via Vagrant and this is all still happening except for the SSLv2 case, where it fails because the protocol isn't included in the OpenSSL version in Ubuntu 14.04 (by design, I believe - SSLv2 is old and outdated).

Versions:
Ubuntu 14.04 32bit (via Vagrant/Virtualbox combo)
Python 2.7.6
requests==2.2.1
requests-toolbelt==0.2.0
urllib3==1.8.2

EDIT: forgot the OpenSSL version...

python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.1f 6 Jan 2014

TLSv1:

```
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
```

SSLv2:

```
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
```

SSLv23:

```
>>> import requests
>>> from requests_toolbelt import SSLAdapter
>>> adapter = SSLAdapter('SSLv23')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/vagrant/.virtualenvs/techtown/local/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
```

Perhaps this is a cipher list issue then? Or is the OpenSSL version used here still problematic?


I am absolutely willing to put in some time to help debug this if necessary... provided you guys give me some direction.


VM is downloading. I can't reproduce this on ArchLinux.
The stacktraces indicate this but I'd like to be sure: You are _not_ using PyOpenSSL but only the stdlib?


@t-8ch Thanks for taking a look at this, I'm a bit confused. OpenSSL makes my life really hard =(


@t-8ch I haven't installed PyOpenSSL if that's what you're asking?

I would have assumed (perhaps incorrectly) that `pip install requests` should give me everything I need to successfully call `requests.get('...')` on an HTTPS page.  Which, of course, it works for the most part, just not for this site for some reason. 


@jaddison It _mostly_ does. Unfortunately, Python 2.7s standard library sucks hard and doesn't support some features, such as SNI.

I wonder if this is SNI...


@jaddison There are two different codepaths behind the scenes. You shouldn't have to care about those, but it helps to know when debugging.

However I can now reproduce this on ubuntu. But only o Py2. On Py3 everything is fine.
I suspect @Lukasa is right and the server fails when the client is not using SNI.


It bothers me that an absence of SNI fails in multiple different ways depending on the server in question.


I did notice this change between OpenSSL 1.0.1f and 1.0.1g (https://www.openssl.org/news/openssl-1.0.1-notes.html):

`Add TLS padding extension workaround for broken servers.`

EDIT: Ahh, nevermind - the bug shouldn't vary between Py 2 and 3, I'd think.


@jaddison To test whether this is SNI, you'll need to [install the SNI requirements](https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484) for Python 2.


@Lukasa was right. Compare:

``` bash
$ openssl s_client -connect docs.apitools.com:443                              
CONNECTED(00000003)
139846853338768:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:762:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 517 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

$  openssl s_client -connect docs.apitools.com:443 -servername docs.apitools.com
... happy handshake here
```


To elaborate: The second command enables the SNI functionality of `openssl s_client`.

You can a) switch to python3 b) install extra dependencies.
The stdlib has at the moment no way to do SNI.


Thanks for the quick feedback. Seeing as there is no bug, I'll close this... again.


Hey, thank you guys !! I installed python3 on my mac and boom, it works.


Just want to chime in and say that I experienced this issue on OS X 10.9.5, Python 2.7.7 and OpenSSL 0.9.8zc.

I was able to fix my handshaking issue by:
1. Installing a newer-than-stock OpenSSL on my machine via `brew install OpenSSL`
2. Compiling and installing the `cryptography` package linked against the new OpenSSL (`env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography`)  
3. Installing requests with SNI support by doing `pip install requests[security]`


Thanks, @Microserf. I'm pretty much running the same specs (10.9.5, Python 2.7.6 installed via Homebrew but compiled with system provided OpenSSL 0.9.8zg) and this was my entire process for getting `requests` up and running for Django:

```
brew install openssl
```

Install `requests` with a bunch of [SNI stuff](https://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl), compiled against our new install of OpenSSL. The `[security]` option simply installs `pyopenssl ndg-httpsclient pyasn1`

```
env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install requests[security] urllib3
```

And we're good to go:

``` python
"""
This may or may not be needed. See:
https://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl
"""
# from urllib3.contrib import pyopenssl
# pyopenssl.inject_into_urllib3()

import requests
# r = requests.get(...)
```


Is there a definitive answer on how to get this working on ubuntu? I'm running into this issue, and it looks like the only answer here concerns how to get this working on a Mac. Upgrading our entire codebase to python 3 is not an option.


OK, I may have just answered my own question. What I did boils down to:

```
sudo apt-get install libffi-dev
pip install pyOpenSSL ndg-httpsclient pyasn1
```


@lsemel thank you, that just saved me a bunch of time


@lsemel Are your sure? I tried it on Ubuntu 15.10 and it still doesn't work with Python 2.7.10.

It works with Python 2.7 on Travis CI:
https://travis-ci.org/playing-se/swish-python


Got it to work now! I simply uninstalled pyOpenSSL:
`pip uninstall pyOpenSSL`


Maybe we should only pyopenssl.inject_into_urllib3() if Python version is less than 2.7.9? pyOpenSSL seems to break stuff on Ubuntu and Windows if Python version is 2.7.10.


PyOpenSSL should not be breaking anything. If it does, that's a bug that should be reported. 


I will have to look into this, but is there any good reason to inject pyopenssl into urllib3 if Python version is 2.7.9 or newer?

I am thinking of something like this:

```
# Check if Modern SSL with SNI support
try:
    from ssl import SSLContext
    from ssl import HAS_SNI
except ImportError:
    # Attempt to enable urllib3's SNI support, if possible
    try:
        from .packages.urllib3.contrib import pyopenssl
        pyopenssl.inject_into_urllib3()
    except ImportError:
        pass
```


Yeah, frequently there is. For example, on OS X most Pythons link against the system OpenSSL, which is version 0.9.8zg. PyOpenSSL, however, will link against a much newer OpenSSL (1.0.2). That makes using PyOpenSSL a substantial security improvement.

Additionally, PyOpenSSL gives us much better access to OpenSSL, allowing us to secure it more effectively. 


OK, I have played around with this a little now.

It WORKS with pyopenssl BUT not if ndg-httpsclient is installed.

However, I can get it work with ndg-httpsclient if I uninstall pyasn1 giving me these warnings:

```
/usr/lib/python2.7/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled.  Import error is: No module named pyasn1.type
  warnings.warn(import_error_msg)
/usr/lib/python2.7/dist-packages/ndg/httpsclient/ssl_peer_verification.py:25: UserWarning: SubjectAltName support is disabled - check pyasn1 package installation to enable
  warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG)
/usr/lib/python2.7/dist-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled.  Import error is: No module named pyasn1.type
  warnings.warn(import_error_msg)
```

Same behavior on Ubuntu 15.10 and Windows 10 with Python 2.7.10 installed.


That's because without ndg-httpsclient the PyOpenSSL support isn't used.