attempt to use the OS's certificate bundle

I thought it might be a good idea to attempt to use the OS's certificate bundle on systems where one is available. I was thinking something like this:
1. It could be dangerous to install `certifi` and then forget about it, in case a CA in it is compromised (like with Diginotar).
2. Might be good not to depend on another package?

Anyway, thanks for your time.

I used github for the smoke test because, amusingly enough, httpbin's certificate didn't verify on my system (Fedora 16):

```
[shivaram@good-fortune ~]$ curl -v https://httpbin.org/get
* About to connect() to httpbin.org port 443 (#0)
*   Trying 107.21.118.103... connected
* Connected to httpbin.org (107.21.118.103) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Peer's certificate issuer is not recognized: 'CN=RapidSSL CA,O="GeoTrust, Inc.",C=US'
* NSS error -8179
* Closing connection #0
* Peer certificate cannot be authenticated with given CA certificates
curl: (60) Peer certificate cannot be authenticated with given CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
```


This, or something similar, would be a great addition for our use too.

We have a few code bases at work that require some hoop jumping to work around system `requests` using system certs[1] and virtualenvs in `tox` runs attempting to use the certs in `certifi`.

I [ended up](https://github.com/ask/python-github2/issues/68#issuecomment-3159947) adding something along the same lines to [github2](https://github.com/ask/python-github2/blob/master/github2/request.py#L41) to work around `httplib2` using bundled certs on some hosts, and it would be nice if I didn't need to worry about it when using `requests` ;)
1. Debian and Gentoo patch system certs support in to their packages, possibly others too.


JNRowe: thanks! I added the FreeBSD path.


I wouldn't mind adding a utility function that makes it simple to use the system's CAs, but I definitely prefer to default to my own packaged CA bundle. It _deeply_ helps reduce unexpected behavior.


That's fair.

Would you prefer a patch where a failure to import `certifi` results in an attempt to use the system CAs (but `certifi` is preferred when present)? That way, administrators and distributions can package `requests` without `certifi` and have it default to the system CAs.


That sounds awesome.


Cool, changed it to prefer the `certifi` bundle.

Tell me if you want me to squash the commits, or remove the unit test (or feel free to do that yourself).


This looks great!

---

Last things:

It'd be great if `DEFAULT_CA_BUNDLE_PATH` took into account the value of the `CURL_CA_BUNDLE` env var (this exists elsewhere in the code now (I think models.py))


Ah, yeah. Right now those environment variables (`REQUESTS_CA_BUNDLE` and `CURL_CA_BUNDLE`) get checked at runtime (i.e., verification time) in `models.py`.

The checks depend on `self.config` (the `TRUST_ENV` variable), though, so maybe they should remain there? Also, it's theoretically possible that someone might monkeypatch the environment variables at runtime and expect the behavior to change, as it does now.


Ah, great point. Agreed.


This is fantastic, thanks!

:sparkles: :cake: :sparkles:


@kennethreitz, quick question.  Do you consider `DEFAULT_CA_BUNDLE_PATH` part of the supported API now?  I'm just wondering if I can call `assert` against it, and expect code to carry on working for the foreseeable future.  Not asking for promises, just a nod ;)

Perhaps, the One Blessed Method for disabling the `certifi` support should be stated in a packaging document?  I'd be surprised if the changes here allowed packagers to use the package as-is, and a continuing variety of solutions is quite difficult to support for naïve `requests` users like me.


I'm not so sure about `DEFAULT_CA_BUNDLE_PATH` itself. I'll probably document using `get_os_ca_bundle_path`. 


@JNRowe Yeah, it's tricky. Perhaps, as the administrator, you could `export REQUESTS_CA_BUNDLE` in `/etc/profile.d` or something similar? That would globally override local installations of `certifi`.

What further changes do you think would be required for packaging?


_EDIT_: Oh wow, sorry guys.  I didn't realise this comment was so long, it'll teach me to keep to the tiny textbox next time.

> @JNRowe Yeah, it's tricky. Perhaps, as the administrator, you could `export REQUESTS_CA_BUNDLE` in `/etc/profile.d` or something similar? That would globally override local installations of `certifi`.

Good idea, and it would work for managed desktop and server deployment.

My question was more concerned with a stable method for testing the environment we're running under on client systems.  I guess if `requests.utils.get_os_ca_bundle_path()` is part of the stable API we could check that, `.startswith('/etc/')` is most of the way there for me.

It is far better than the current situation, needing a lot of code pasting to check for various changes.  On Debian,  the `where()` call is just replaced with `/etc/ssl/certs/ca-certificates.crt`.  On Gentoo, `certifi` is installed with a symlink to the system certs, which necessitates a call to `os.path.realpath()`.

> What further changes do you think would be required for packaging?

Without trying to be deliberately obtuse I'd say "ask the packagers".  I may be totally wrong, the next round of distributor packages could accept this behaviour and ship vanilla packages.  The only reason I doubt it is the automagic nature of the changes, the certs being chosen depending on whether another package is currently installed system-wide or available in `$PYTHONUSERBASE` or somewhere else seems a little off.

Don't get me wrong, I think you and @kennethreitz came to the right conclusion for the common case.  Most people simply don't care about this stuff that deeply, and the ones that do will patch the behaviour anyway.  That is why I resigned myself to checking at runtime, and asked for a little confirmation on an acceptable way to do it.


@JNRowe Thanks, these are really good observations. I'll get in touch with the guy who maintains `requests` for Fedora and see if this change is sufficient for him to package `python-requests` with a dependency on `ca-certificates`.

I understand now why you wanted to be able to test `DEFAULT_CA_BUNDLE_PATH`. I'd put in a tentative +1 for making this part of the API, but my understanding of how stable that would be is shaky at best :-)

By the way, do you know what the standard path to the CA certificates is on Gentoo, and the name of the package that provides them?


> By the way, do you know what the standard path to the CA certificates is on Gentoo, and the name of the package that provides them?

`/etc/ssl/certs/ca-certificates.crt` provided by [app-misc/ca-certificates](http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-misc/ca-certificates/).  It is the Debian package, you even use the same `update-ca-certificates` tool when managing your certs database.  If it wasn't for the differing approach to patching `requests` I could tell the difference between a system and `pip` installation with practically no effort, which is why I thought a blessed solution for forcing system certs in a `README.packagers` would be a good idea.

While we're collecting other locations to test I'll add that [suse](http://rpmfind.net/linux/RPM/opensuse/12.1/noarch/ca-certificates-1-14.12.1.noarch.html) uses `/etc/ssl/ca-bundle.pem`.  And that covers the only three Linux distributions I ever have to worry about :)


Fedora maintainer here, @slingamn this is a clean approach, i was lucky that people allowed me to package python-certifi, i would be happy to remove any dependency on certifi and default to system CA certs.


Looks like I _was_ wrong, the [Debian package](http://packages.debian.org/changelogs/pool/main/r/requests/requests_0.11.2-1/changelog) accepts this behaviour.