TLS SNI Support

It seems I'm having problems using requests with servers that need TLS SNI support. When will requests have this feature? Is this something that is planned? 

Thanks,
--Ram


Not posisble in Python 2.x on top of the stdlib's ssl module.


Twisted has it, right? Perhaps time to depend on pyopenssl or some other ssl module?


There's a pull request for urllib3 adding SNI support for Py32+, but still needs test coverage. https://github.com/shazow/urllib3/pull/89


shzow: Thanks for the pointer. I'm unfortunately using Twisted/Flask, which means I'm stuck to 2.7. Twisted it seems can do something like this, but i've not seen examples, and it would need pyopenssl + twisted decoding to do this. Am not an expert in either to pull this off. At this point, just thinking of wrapping some command line utility like wget... :(


PyOpenSSL should be able to do it on 2.x. So there is hope.


I could make it work with pyopenssl. But I still am having a hard time getting it done in twisted correctly. http://pbin.be/show/719/ -- kind of works. But then one has to build an API on it for it to be any useful. I still cant figure out how to pass the correct context to Agent. 


Lukasa: was this resolved? 


No. We're blocked on shazow/urllib3#89.


Merged. Carry on. :)


Note that it's only supported on Py32+ for now. Should be enough to get started though.


Another fair warning: Some tests on Py32 are failing on urllib3 master right now. Some help resolving this would be greatly appreciated (and might affect this functionality): https://github.com/shazow/urllib3/issues/128


works for me on archlinux x64 on py26, py27, py32, py33


Ah I should have qualified that this is on OSX. Scumbag OSX.


t-8ch: Are the sni tests passing on py26/27/32 and 33 for you? 


The SNI tests aren't even run on py2, but on py32 and py33 it works.
There is no way to do SNI with the ssl module of the standard library before py32.


@shazow: Presumably you have no interest in making urllib3 depend on PyOpenSSL?


I'd be happy to have a separate library which adds SSL via PyOpenSSL support to urllib3.


It looks like @t-8ch is knocking it out of the park with the work on urllib3. I'll let him tell us when this can be closed.


urllib3 has optional support for SNI with python2, though with a few optional dependencies. (See [urllib3#156](https://github.com/shazow/urllib3/pull/156)).

I belive this should work using:

```
from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3
```

After adding this to `packages` in `setup.py`

```
requests.packages.urllib3.contrib
```

Note that the `contrib` package is not imported by default.

There are basically two alternatives to enable SNI in requests:
- Create function to manually enable SNI that basically runs the above code. (the downside is that users will need to explicitly enable SNI support).
- Try to import the pre-requisites. If they exists, enable SNI, else don't.

I've no problem implementing any of both alternatives, though I'd prefer to know which would be the prefer one for requests. I personally vote for the second alternative.

BTW: can we reopen the issue, now that we have support from urllib3?


I actually prefer the former because the people who need it are the ones who know they'll need it and they're few enough that it won't cause too many complaints. However, if we're going to be consistent with the existing API, the latter would be the way to implement it. Currently, urllib3 (and requests) will provide the API to send HTTPS requests without the ssl module present but will fail in the case where the ssl module is not available. In other words, the API is there and you can use it but it just won't work and that is indicated by an exception.

As to re-opening this, that's up to @kennethreitz. The issue of course (with all of this) is that we're currently under a feature freeze (#1165, #1168) so I'm not sure if the work is even worth doing because it might not be accepted.


Let me be a bit clearer: what I meant by the second alternative was "Try to use SNI if the optional deps are available, but still use SSL as we always have it they aren't.". This shouldn't break anything existing up to now.

As for reopening (or not) the issue, IMHO, this issue shouldn't be taken lightly. requests can become useless without SNI support for many scenarios. In particular, multiple domains on a single IPv4 host without SNI become impossible. And additional IPv4 addresses are out of the questions for many users (due to their cost).

In any case, this is half feature, half bug, though I'll await @kennethreitz's reply regarding reopening the issue. 


Hugo: I just did pip install -U requests, and when I can't import contrib.
How do I fix that? (Importerror : No module named contrib).

On Fri, May 3, 2013 at 1:12 PM, Hugo Osvaldo Barrera <
notifications@github.com> wrote:

> urllib3 has optional support for SNI with python2, though with a few
> optional dependencies. (See urllib3#156https://github.com/shazow/urllib3/pull/156
> ).
> 
> I belive this should work using:
> 
> from requests.packages.urllib3.contrib import pyopenssl
> pyopenssl.inject_into_urllib3
> 
> Note that the contrib package is not imported by default.
> 
> There are basically two alternatives to enable SNI in requests:
> - Create function to manually enable SNI that basically runs the above
>   code. (the downside is that users will need to explicitly enable SNI
>   support).
> - Try to import the pre-requisites. If they exists, enable SNI, else
>   don't.
> 
> I've no problem implementing any of both alternatives, though I'd prefer
> to know which would be the prefer one for requests. I personally vote for
> the second alternative.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/749#issuecomment-17406518
> .


request's setup.py doesn't include that package (the source it's there, but it's excluded when installing/packaging), so it's not installed, that's why I mentioned:

> After adding this to `packages in setup.py`
> requests.packages.urllib3.contrib

You'll basically need to install from source.
Check out my pull request if you're interested, making this work is just a couple of lines. :)


> "Try to use SNI if the optional deps are available, but still use SSL as we always have it they aren't."

SSL is not always there though. That was my point. Currently we try and use it but fail hard and fast if we don't have it and a user tries to make an https request. The way you had described it though, I was under the impression SNI would require the user to pass some extra notion to urllib3 and you were focusing on just the set-up for it. If all that is necessary is #1347 then I'm 100% behind this.


It would be really nice if this was the default (SNI Support) and requests
just works without any extra code or setup from source. For urllib3 its 2
lines of code. Why not just add that to requests. Am I day dreaming? :)

On Fri, May 3, 2013 at 10:43 PM, Ian Cordasco notifications@github.comwrote:

> "Try to use SNI if the optional deps are available, but still use SSL as
> we always have it they aren't."
> 
> SSL is not always there though. That was my point. Currently we try and
> use it but fail hard and fast if we don't have it and a user tries to make
> an https request. The way you had described it though, I was under the
> impression SNI would require the user to pass some extra notion to urllib3
> and you were focusing on just the set-up for it. If all that is necessary
> is #1347 https://github.com/kennethreitz/requests/issues/1347 then I'm
> 100% behind this.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/749#issuecomment-17426626
> .


@pythonmobile spin a top. If it doesn't stop spinning, you're dreaming.

![inception](https://gs1.wac.edgecastcdn.net/8019B6/data.tumblr.com/tumblr_lzpbw5SP2X1r94e9jo1_500.gif)


Leaving a comment for the sake of others who may run into the same issue as me:
If you are having trouble making secure requests using the `requests` library, in particular when trying to hit an app hosted on Google App Engine, this may be why. The error I was seeing was "EOF occurred in violation of protocol". To diagnose this, try hitting the same host using `s_client` (replace `example.com` with the host in question):

```
openssl s_client -connect example.com:443
openssl s_client -connect example.com:443 -servername example.com
```

If the first command fails with "handshake failure" and the second succeeds, then you are trying to hit a server that uses SNI. The present thread has some good info on how to get `requests` working in this situation. What I did was try to run `from requests.packages.urllib3.contrib import pyopenssl` in ipython, and I kept on pip installing any import errors. YMMV.


@mshang Version 1.2.3 should do this for you already; are you sure you're running that version?


@hobarrera Yes, I'm running 1.2.3. `requests.packages.urllib3.contrib` was there, but that import statements still threw a bunch of import errors. I had to install `ndg-httpsclient` and I forget what else.


Correct. Requests just attempts to import the needed modules, and if it can't gives up and goes on with its life. It might be a good idea to document what you need to use SNI with Requests, though.