Use by default SSL CA certificate bundle from the platform

If pyopenssl is available and you give an empty ca_certs to urllib3,
it will use the default CA of the current platform.

Not ready for merging as this feature requires a bleeding edge version of urllib3 (https://github.com/shazow/urllib3/commit/5c25a73dfb48e4260c44e19e3a50fb5d46832c52)

I post this PR now, to discuss about its implementation and testing improvements.
About testing we could run the project with tox and iterate among different versions of python and with or without dependencies of urllib3.contrib.pyopenssl.

```
pyOpenSSL
ndg-httpsclient
pyasn1
```

thx


Ok, let's think about how this works from the perspective of the standard Requests API. It's intended to be used in one of these ways:

``` python
# Verify certs using the bundled cacerts.
r = requests.get('https://somesslurl.com/')

# Use your own cacerts.
r = requests.get('https://somesslurl.com/', verify='/path/to/cacerts')

# Don't verify.
r = requests.get('https://somesslurl.com/', verify=False)
```

Your PR in its current form changes this behaviour in an alarming and subtle way:

``` python
# Verify certs using the bundled cacerts OR the built-in ones,
# depending on whether you have pyopenssl installed or not.
r = requests.get('https://somesslurl.com/')

# Use your own cacerts.
r = requests.get('https://somesslurl.com/', verify='/path/to/cacerts')

# Don't verify.
r = requests.get('https://somesslurl.com/', verify=False)
```

I don't think we should change which certificates we use based on whether or not you install an _optional_ dependency. Our current behaviour WRT pyopenssl is to enable _additional_ features without changing our current behaviour. That's what you should be aiming for here.


Hi @Lukasa ,
Today if a user import `requests` from global python, and if this package has been patched (like it is intended and encouraged) the user will get the platform CA and not the bundle CA.
This pull request aim to give to the user to the same behaviour when he uses `requests` from its own virtualenv, not from global python. In this regard, I think, this PR is pretty consistent.

Now if we want to make progress. Do you suggest to go in a way that is more explicit for the user ?

``` python
# with pyopenssl
r = requests.get('https://somesslurl.com/', verify=requests.DEFAULT_CA)
r = requests.get('https://somesslurl.com/', verify=requests.BUNDLE_CA)

# without pyopenssl
r = requests.get('https://somesslurl.com/', verify=requests.DEFAULT_CA)
  Traceback
  [...]
  SSLError()
r = requests.get('https://somesslurl.com/', verify=requests.BUNDLE_CA)
```

or you completely reject the idea ?

thx


Woah, woah. Your statement that we "intend and encourage" people to patch Requests to use the system CAs is not true. We _allow_ people to do that, but we have no particular opinions about whether they should. In fact, we now rigorously maintain our own CA bundle, since `pip` uses it.

What you're saying is that on systems where the site-package install of `requests` came from `apt` or `yum` it uses the system CAs. That is the decision of the person who built that package for that repository, and should _not_ be viewed as the viewpoint of the Requests project. We expect people to install requests from `pip`, and when you do that you get the bundled CAs by default.

I don't know whether I completely reject the idea or not. I'm not convinced that this is better than just asking the user to specify the path to the system CAs if they want to use them, which we already allow.


Unfortunately, there is no canonical way to get the System CAs. So, as a user of `requests`, specifying the path of the system CAs is not an option, if I want to provide a library that is platform agnostic. 
Today `requests` does not give real other choice than using the bundle CA.
This is the problem I'm trying to fix.


I don't have a good handle on what the use-case is here. If you were providing a platform-agnostic library, why would you want your users to all be using different CA certs? Surely you'd want to ensure that all your users have the _exact same_ certs on all platforms, to avoid annoying platform-specific bugs?


Basically, we are self signing our certificates. So we declare ourself as trustable CA globally for the system.
Then all ours tools can access our repos.
Our toolkit that needs access to our online resources are for instance `docker`, `docker-py` and `curl`.
Here `Go`, and `libcurl` are by default reading the platform CAs but `docker-py`, that uses `requests` underneath, is not. 
It means I need to patch somehow the setup of all my venv that uses `requests`.
It is too ugly and too much work for me.

As a side note: In terms of security level, the practice of self signing certificates might be considered more secure than relying on well known CA.


OK, so this is my understanding of your situation:
1. You have a number of SSL certificates that have been signed by your own internal CA.
2. Those are distributed to your systems (which are over a number of OSes) via some automated deployment system and are installed to the system CA.
3. You have a number of tools that access your TLS-encrypted endpoints, some of which are based on Requests.
4. You cannot make local code changes to affect the certificate validation of your Requests-based tools on those systems (or you can, but it's going to be a very costly thing to do that you cannot bear that cost).

Does that seem right?

That is a legitimate problem, but I feel like it can be solved by environment variables. Requests exposes the REQUESTS_CA_BUNDLE environment variable which overrides the location of the CA certificates. Is that variable suitable for your use-case? If not, why not?


You perfectly understood my use-case.
I was completely ignorant about REQUESTS_CA_BUNDLE. 
It seems perfect to feel the gap.

I'm very sorry for not having see it before.
And I'm grateful for the time you spent to understand my needs.
I'm closing the PR.


That's not a problem at all, I'm glad we were able to come up with a solution that was good for both of us. I'm glad you're getting lots of use out of Requests!


_Use by default SSL CA certificate bundle from the platform_ is what OP proposes in this issue and that's something I totally agree with. It feels so natural to me that I find it difficult someone could oppose this, yet alone argue that not doing this is better experience. Yet this is what I see here and that's why I decided to share my opinion with you.
@Lukasa states

> I don't think we should change which certificates we use based on whether or not you install an optional dependency. Our current behaviour WRT pyopenssl is to enable additional features without changing our current behaviour. That's what you should be aiming for here.

This argument is misguided here as pretty much everyone agrees that when you're dealing with security you should **by default** be as secure as you are able to be, given the environment you operate in. This means using system CA certificates by default (and fallback to bundled ones if it's not possible or very hard to do) and not bundled ones. I think one might argue that security begs for policy of graceful 
degradation.

Also @Lukasa states

> Surely you'd want to ensure that all your users have the exact same certs on all platforms, to avoid annoying platform-specific bugs?

Surely, if the only goal is to _avoid annoying platform-specific bugs_ then yes. But if the goal is to make requests secure by default (and this is in line with _HTTP library for humans_ motto) then surely you would want to ensure that all your users have _the best certs available on their systems_.

Additional bonus is that you save packagers from having to patch this (mis)feature.

I'm curious what @dstuff and @t-8ch think.


@piotr-dobrogost I'm pretty neutral on this, but I suspect Kenneth will be against it. It adds code complexity (we need a fallback path) and project management overhead (extra bugs filed, both when we change which certificates we use to verify and later when people have failures on specific platforms that do not reproduce on others). Given that we're doing a pretty good job of securing the system already, and that we make it possible to use other certificate bundles, you've got a hell of a job proving that this is worth the switch.


> (...) when people have failures on specific platforms that do not reproduce on others.

I doubt there would be many such failures i.e. caused due to difference in CA certs alone. For instance I do not recall many issues raised in this project by people using requests patched to use system CA certs.


> I do not recall many issues raised in this project by people using requests patched to use system CA certs.

You mean when requests would use a set of locations where the certificates _might_ be? I can remember several and I'm sure a search of issues would result in several issues about it. Most of what I remember are people using distributions of linux that did not use the de facto defaults of other more popular distributions. These people then sent PRs to extend the list of possible locations to include the specific locations for the different versions of that distribution they happened to use. That is exactly what we're trying to avoid in vendoring the certificates. If the SSL library or really anything gave us the information needed to programmatically determine the location without having a hard-coded list of them ourselves, that'd be great.


@sigmavirus24, That's the point of this PR, pyopenssl takes care to provide the default platform CA.

https://github.com/shazow/urllib3/commit/5c25a73dfb48e4260c44e19e3a50fb5d46832c52
http://pyopenssl.sourceforge.net/pyOpenSSL.html/openssl-context.html#l2h-132


@ticosax The section of the documentation you linked to quite literally says 'This method may not work properly on OS X'. Such an admission is tantamount to saying 'This method only sometimes works'. 

Additionally, I can tell you that this doesn't work properly because I've hit exactly this bug in my own project, [hyper](https://github.com/Lukasa/hyper). See issue Lukasa/hyper#9, but in summary, `set_default_verify_paths()` only works on some systems, and otherwise fails _without any way to detect it_, as you can see from [the Python documentation](http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths). Given that PyOpenSSL is only a thin wrapper around SSL, much like the stdlib version, I will assume that PyOpenSSL has exactly the same problems with its method.

BTW, one of those systems is Windows. =)


@Lukasa summed up everything I was about to say in a much nicer way. Count this as a +1 for what he just said.


@sigmavirus24 

> These people then sent PRs to extend the list of possible locations to include the specific locations for the different versions of that distribution they happened to use.

Then all you have to do per @ticosax's remark – _(...) pyopenssl takes care to provide the default platform CA._ – is to direct these people to pyopenssl and close issue right away :)

The theme of this project is that you care about what most people do and you try to make their lives easier. Now, I bet most people use only a handful of systems so there's no problem for pyopenssl project to take care of those. When someone uses unpopular system he should not expect that every piece of software on earth would handle his system. In this case telling him he has to point requests to his CA certs is perfectly fine.

@Lukasa 

> The section of the documentation you linked to quite literally says 'This method may not work properly on OS X'. Such an admission is tantamount to saying 'This method only sometimes works'. 

OS X is a popular OS (unfortunately as I don't like Apple at all) so if it really does not work for it then it's unacceptable. However I don't believe OS X actively hides its CA certs :)


@piotr-dobrogost Gotta read my whole answer. I have personal evidence that this does not work on Windows (though OS X was actually fine), where I got SSL certificate errors when talking to Twitter's servers. Given that there's no error response from the method, we just can't rely on it.


Yup, they're [in the registry](http://msdn.microsoft.com/en-us/library/windows/desktop/aa388136%28v=vs.85%29.aspx), which makes getting at the certs a nightmare. OpenSSL does appear to provide an engine for getting at Microsoft's CryptoAPI stuff, but it's essentially undocumented.

For those who want an illuminating OpenSSL email trail, [here's one](http://openssl.6102.n7.nabble.com/SSL-CTX-set-default-verify-paths-and-Windows-td25299.html). The summary is this:

> `set_default_verify` is effectively `_load_verify_locations` 
> using env vars SSL_CERT_FILE SSL_CERT_DIR if they exist 
> and otherwise `X509_get_default_cert_{file,dir}()` which return 
> a compiled-in file and directory normally file "cert.pem" and 
> subdir "certs" under OPENSSLDIR, which is configurable at build 
> time and can be seen with commandline openssl version -d .

Put another way, there's no intelligence here.

I hereby speak on behalf of the requests project: there's no chance of us using the default system certs, at least not via OpenSSL. Too easy to break, too hard to detect when it does.


> (...) at least not via OpenSSL. Too easy to break, too hard to detect when it does.

Fair enough.

> Put another way, there's no intelligence here.

I believe it's official interface of OpenSSL which uses 3 _well known_ locations when looking for certs. That's not much but it's something.
However this

> Given that there's no error response from the method, we just can't rely on it.

is a blocker, indeed.