Consider using system trust stores by default in 3.0.0.

It's been raised repeatedly, mostly by people using Linux systems, that it's annoying that requests doesn't use the system trust store and instead uses the one that certifi ships. This is an understandable position. I have some personal attachment to the certifi approach, but the other side of that argument definitely has a reasonable position too. For this reason, I'd like us to look into whether we should use the system trust store by default, and make certifi's bundle a fallback option.

I have some caveats here:
1. If we move to the system trust store, we must do so on _all_ platforms: Linux must not be its own special snowflake.
2. We must have broad-based support for Linux and Windows.
3. We must be able to fall back to certifi cleanly.

Right now it seems like the best route to achieving this would be to use [certitude](https://github.com/python-hyper/certitude). This currently has support for dynamically generating the cert bundle OpenSSL needs directly from the system keychain on OS X. If we added Linux and Windows support to that library, we may have the opportunity to switch to using certitude.

Given @kennethreitz's bundling policy, we probably cannot unconditionally switch to certitude, because certitude depends on cryptography (at least on OS X). However, certitude could take the current privileged position that certifi takes, or be a higher priority than certifi, as an optional dependency that is used if present on the system.

Thoughts? This is currently a RFC, so please comment if you have opinions. /cc @sigmavirus24 @alex @kennethreitz @dstufft @glyph @reaperhulk @morganfainberg


I think the system trust stores (or not) essentially boils down to whether you want requests to act the same across platforms, or whether you want it to act in line with the platform it is running on. I do not think that _either_ of these options are wrong (or right), just different trade offs.

I think that it's not as simple on Windows as it is on Linux or OSX (although @tiran might have a better idea). I _think_ that Windows doesn't ship with all of the certificates available and you have to do something (use WinHTTP?) to get it to download any additional certificates on demand. I think that means that a brand new Windows install, if you attempt to dump the certificate store will be missing a great many certificates.

On Linux, you still have the problem that there isn't one single set location for the certificate files, the best you can do is try to heuristically guess at where it might be. This gets better on Python 2.7.9+ and Python  3.4+ since you can use `ssl.get_default_verify_paths()` to get what the default paths are, but you can't rely on that unless you drop 2.6, 2.7.8, and 3.3. In pip we attempt to discover the location of the system trust store (just by looping over some common file locations) and if we can't find it we fall back to certifi, and one problem that has come up is that sometimes we'll find a location, but it's an old outdated copy that isn't being updated by anything. People then get really confused because it works in their browser, it works with requests, but it doesn't in pip.

I assume the fall back to certifi ensures that things will still work correctly on platforms that either don't ship certificates at all, or don't ship them by default and they aren't installed? If so, that's another possible niggle here that you'd want to think about. Some platforms, like say FreeBSD, don't ship them by default at all. So it's possible that people will have a requests using thing running just fine without the FreeBSD certificates installed, and they then install them (explicitly or implicitly) and suddenly they trust something different and the behavior of the program changes.

Anyways, the desire seems reasonable to me and, if all of the little niggles get worked out, it really just comes down to a question of if requests wants to fall on the side of "fitting in" with a particular platform, or if it wants to prefer cross platform uniformity.


> I think the system trust stores (or not) essentially boils down to whether you want requests to act the same across platforms, or whether you want it to act in line with the platform it is running on. I do not think that either of these options are wrong (or right), just different trade offs.

I wouldn't say that either option is _completely_ wrong, but I do think that using the platform trust store is significantly right-er, due to the availability of tooling to adjust trust roots on the platform and the relative unavailability of any such tooling for requests or certifi.  If you go into Keychain Access to add an anchor (or the Windows equivalent) nothing about requests makes it seem like it would be special, that it would be using a different set of trust roots than you had already configured for everything else.


It depends if your audience are people who are familiar with a particular platform or not. I have no idea how to manage the trust store on Windows but I know how to manage the trust store for requests because requests currently chooses being internally consistent cross platform over being externally consistent with any particular platform. 

IOW this change makes it easier for people who are familiar with the system the software is running on at the cost of people who are not. 

Sent from my iPhone

> On Jan 11, 2016, at 4:29 PM, Glyph notifications@github.com wrote:
> 
> I wouldn't say that either option is completely wrong, but I do think that using the platform trust store is significantly right-er


> IOW this change makes it easier for people who are familiar with the system the software is running on at the cost of people who are not.

In the abstract, I disagree.  Of course, we may have distribution or build toolchain issues which make people have to care about this fact, but if it works _properly_ (`pip install`s without argument, doesn't require C compiler shenanigans for the end user) then what is the penalty to people who are not familiar with the platform?


It forces them to learn the differences of every platform they are running on. 

Sent from my iPhone

> On Jan 11, 2016, at 4:59 PM, Glyph notifications@github.com wrote:
> 
> what is the penalty to people who are not familiar with the platform?


So I am in favour of using the system store in general because in most cases [outside of dev] if you're relying on requests or a similar library you're going to expect it to work similar to the rest of the system/tooling. Asking someone to learn the tooling for the system they are deploying on is not unreasonable. If an application uses requests and needs to trust a specific cert (end-user story here), it is usual that it'll handle the installation for that platform at install time (aka OS X or Windows). 

From a developer perspective, it becomes a little more difficult but still not insurmountable as long as we approach this in a way that the developer has clear methods to continue with the same behaviour as today.

As discussed in IRC, perhaps the easiest method to ensure sanity is to really finish up and polish certitude(? was this the tool discussed?) so we can encapsulate the platform/system-specifics clearly and try and ensure requests logic is the same on all of the platforms.


> It forces them to learn the differences of every platform they are running on.

How so? By default, it ought to get the trust roots you expect on every platform. It's not like I need to learn new and exciting things when I launch Safari vs. IE just to type `https://`...


> I think that it's not as simple on Windows as it is on Linux or OSX (although @tiran might have a better idea). I think that Windows doesn't ship with all of the certificates available and you have to do something (use WinHTTP?) to get it to download any additional certificates on demand. I think that means that a brand new Windows install, if you attempt to dump the certificate store will be missing a great many certificates.

You are right about this.  I have verified it on my nearly-pristine Windows VM; in a Python prompt, I do:

``` python
>>> import wincertstore
>>> print(len(list(wincertstore.CertSystemStore("ROOT"))))
```

and get "21".  Visit some HTTPS websites, up-arrow/enter in the python interpreter, and now I get "23".


There's some technical documentation here:

https://technet.microsoft.com/en-us/library/bb457160.aspx


And some more explanation here:

http://unmitigatedrisk.com/?p=259


Frustratingly, I can't find an API that just tells it to grab the certificate store; it seems that verifying a certificate chain that you don't have the root to is the only way that it adds certificates, and it adds them one at a time as necessary.  It baffles me that Microsoft seems to consider storage for certificates a scarce resource.


After hours of scouring MSDN, I give up.  Hopefully someone else can answer this question: https://stackoverflow.com/questions/34732586/is-there-an-api-to-pre-retrieve-the-list-of-trusted-root-certificates-on-windows


In my experience with pip, which attempts to discover the system store and if it can't find it falls back to a bundled copy, I have had to learn how the system store works on platforms that I have no intention on ever running. This is for a fairly simple method of detection (look for file systems) but it absolutely ends up that way. The simple fact is, in my experience most people have absolutely no idea how their system manages a trust store (and if it manages a trust store or not). 

Sent from my iPhone

> On Jan 11, 2016, at 5:22 PM, Glyph notifications@github.com wrote:
> 
> How so? By default, it ought to get the trust roots you expect on every platform. It's not like I need to learn new and exciting things when I launch Safari vs. IE just to type https://...


Look for files on the system*


FWIW, If I could prevent downstream redistributors from forcing pip to use the system store, I would revert the change to look in system locations immediately and only ever use a bundled copy. The UX of that tends to be so much nicer, the only reason we started to trust the system trust stores is because redistributors do patch pip to use the system trust store, so you end up in a situation where people get different trust stores based on where their copy of pip came from (which is likely also a concern for requests).

As an additional datapoint, if I remember correctly, the browsers which are _not_ shipped with the OS tend to not use the system trust store either. According to Chrome's [Root Certificate Policy](https://www.chromium.org/Home/chromium-security/root-ca-policy) they will use the system trust store on OSX and on Windows but they won't use it on Linux. I believe that even where Chrome does use the system store, they still layer their own trust ontop of that to allow them to blacklist certificates if need be. I assume this capability is in place because they do not wholly trust the OS trust stores to remove compromised certificates. If I recall correctly, Firefox does not use the system trust store at all on any OS.

Another question is what even is the "root trust store" on a Linux. The closest that you can get is wherever the system provided OpenSSL (assuming they even provide OpenSSL) is configured to point to. However AFAIK there is no way to determine if you're using a system provided OpenSSL or a copy that someone installed (perhaps via Anaconda?), the additional copy may have stale certificates or no certificates available. If it's stale certificates, then you've successfully lowered the security of requests user's by attempting to follow the system trust store. If it's an empty trust store, how do you determine the difference between "empty because I trust nothing" and "empty because my copy of OpenSSL isn't shipping them" or do I have to manage the certificates I trust using my OS, unless I want to trust nothing then I have to manage the certificates I trust using requests?

I've also found that talks about trying to use the platform certificate trust store on Linux (see [here](https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/)). I've not personally verified the information in this article, however it makes me feel very wary about trying to make using the system trust store anything but an exercise in frustration. 


I think our current approach is the correct one, considering who Requests was built for. 

That being said, it wouldn't hurt to add more documentation/functionality around using system certs for "advanced" users. 


I think talking about configurability is maybe a red herring.  It's useful – necessary even – in certain circumstances, but users who know they need that can usually figure it out.

The more significant issue is that trust root database updates, especially for end-user client software, are both infrequent and extremely important to do in a timely manner.  `certifi` has no mechanism for automatically updating.  Not only will it not come down in a system update, it can't even be done globally; you have to do it once per Python environment (virtualenv, install, home directory, etc, where it's installed).


@glyph I think you may have hit the nail on the head there. That is a significant reason (and addresses the other issues outlined) to use a more centralized location for the cert store.


The flip side is that you have trust stores like Debian which trusted CACert for a long time, and still trusts SPI even though neither of those have gone through any sort of real audit that you'd expect for a trusted CA.


> The flip side is that you have trust stores like Debian which trusted CACert for a long time, and still trusts SPI even though neither of those have gone through any sort of real audit that you'd expect for a trusted CA.

As I understand it, Microsoft trusts their own CA, too, and Apple trusts theirs.  SPI is just Debian's version of that, isn't it?


SPI isn't run by Debian, it's a third party organization similar to that of the Software Freedom Conservancy that Debian happens to be a member of. It'd be more like Microsoft and Apple trusting the CA of their datacenter just because they happened to use them as their data center. In addition, I'm pretty sure that Apple and Microsoft have both passed a WebTrust audit for their root CAs.


To be fair to Debian, I think the _current_ plan is to stop using SPI certificates for their infrastructure and switch to more generally trusted certificates and then stop including SPI and switch to using just the Mozilla bundle without any additions.

That being said, is it even true that they are shipping updates to them? Looking at [packages.debian.org for the ca-certificates packages](https://packages.debian.org/search?keywords=ca-certificates) it shows that the versions there are:
- squeeze (oldoldstable): `20090814+nmu3squeeze1`
- wheezy (oldstable): `20130119+deb7u1`
- jessie (stable): `20141019`
- stretch (testing): `20160104`
- sid (unstable): `20160104`

I haven't looked at the actual _contents_ of these packages, but the version numbers lead me to believe that they are not infact keeping the ca-certificates package up to date.

In addition to that, looking at the open bugs for ca-certificates there are bugs like [#721976](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=721976) which means that the ca-certificates store includes roots which are not valid for validating servers and are only valid for other topics (like email) which means you can't actually use the current ca-certificate package without massaging it to remove those certificates yourself.

Another issue [#808600](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808600) has Comodo requesting a removal of a particular root that they no longer consider to be inscope for the CAB 
Forum's Baseline Requirements. That has been removed from testing and sid, but has not been removed from jessie, wheezy, or squeeze. The maintainers of ca-certificates claim they'll be requesting an upload to jessie and wheezy, but not to squeeze (which still has LTS suport).

That's just from spending a little bit of time looking at one, fairly popular, distribution. I imagine the long tail of the issues with the system provided bundle gets worse the further away from the popular distributions you get. It's not clear to me that it's a reasonable assumption that the certificates included in any random OS are going to be properly maintained.

The other elephant in the room is we're just assuming that because an update is available to their ca-certificates package that someone is going to have pulled it in. As far as I know, most (if not all?) of the Linux systems do not automatically update by default and require configuration to start to do so. There is likely to be a bigger chance of this with Docker in the picture. On the flip side, I think people generally try to update to the latest versions of their dependencies when working on a project.


Debian's cert bundle is almost certainly like ubuntu's, which does not update to the Mozilla bundle that removed 1024-bit roots in order to avoid the pain like that which hit certifi. All of the out-of-date cert bundles are OpenSSL pre 1.0.2, which means they cannot correctly build the cert chain to a cross-signed root without having the 1024-bit cross-signing root still present. I suspect that's the real concern there.


Maybe they should stop shipping an OpenSSL that can't correctly validate certificate chains.


I reached out to Kurt Roebx about backporting the fixes for that, he said
it was a thing he was looking at doing, I have no clue what the timeline is.

On Wed, Jan 13, 2016 at 7:27 AM, Donald Stufft notifications@github.com
wrote:

> Maybe they should stop shipping an OpenSSL that can't correctly validate
> certificate chains.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2966#issuecomment-171276369
> .

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084


So, what if we made the behaviour of looking at the system CA bundle an extra, e.g., `pip install requests[system_ca]` where it would ship without our certificate bundle and instead use certifi. This allows us to keep on "just work" ing for our default user base while supporting the people who need to use the system bundle.


Seems reasonable.


I would start with adding an API for it, before worrying about how to
enable it via installation, you need `requests.get(url,
verify=<something>)` to exist first.

On Wed, Jan 13, 2016 at 7:41 AM, Ian Cordasco notifications@github.com
wrote:

> So, what if we made the behaviour of looking at the system CA bundle an
> extra, e.g., pip install requests[system_ca] where it would ship without
> our certificate bundle and instead use certifi. This allows us to keep on
> "just work" ing for our default user base while supporting the people who
> need to use the system bundle.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2966#issuecomment-171279745
> .

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084


Debian / Ubuntu have more issues. They ignore CKT_NSS_MUST_VERIFY_TRUST flag and throw all trust anchors in one PEM file. The flag overrides extended key usage (EKU) for root certs, e.g. only trust a root cert for S/MIME but not for TLS server auth.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=721976
https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1207004

Apple, Microsoft and Red Hat/Fedora have means to disable certs for a purpose. Red Hat only puts trust anchors for TLS server auth in the default cert bundle. With FreeDesktop.org's p11kit and PKCS#11 bindings the policies can even be modified by user. This doesn't work for OpenSSL yet because OpenSSL uses a static PEM file instead of PKCS#11 to fetch trust anchors.

@glyph Does MS trust their own cert for all EKUs or just for some EKUs like code signing?


@alex would something like

``` py
import requests
from requests import certs

requests.get(url, verify=certs.system_bundle_where())
```

Be a sufficiently good API?