Update SSL Certs

We really need to automate this and have an official workflow and policy for security releases.

Fixes #1655 
### TODO:
- [x] Automate CA Bundle generation
- [ ] Establish documented security policy
### See Also:
- [PEP 453 Security Considerations](http://www.python.org/dev/peps/pep-0453/#security-considerations)
- [kennethreitz/cert-playground](https://github.com/kennethreitz/cert-playground)


:+1:


#1460


Is the right thing to do here just to rewrite cURL's Perl script that converts the Mozilla text file into a .pem file? I took a look and it shouldn't be too awful to do.


the problem with mozilla's cert list is that it includes all of the blacklisted certs. We need to automate the removal of those. 

The code at #1460 is interesting —it claims to be both a valid python file and a valid pem file at the same time. 


Oh, I didn't spot he had a parser for Mozilla's certdata. That's interesting. I'll take a look into using that as a springboard. /me assigns to self.


This came up during the review/discussion of PEP 453 (the now accepted PEP to add an ensurepip module to bootstrap pip in Python 3.4). However, that acceptance is currently conditional on meeting an integration timeline that includes resolving this issue as part of the PEP's security considerations: http://www.python.org/dev/peps/pep-0453/#security-considerations

While ensurepip isn't affected by this (since it doesn't talk to the internet), _pip_ is affected when installing from PyPI (including to update itself).

The target date in the PEP is currently December 29th (a bit over two months from now)


/cc @dstufft


Will Pip be relying on Requests' CA Bundle (and to-be-introduced infrastructure)? I'm assuming so, since it needs to make calls out to the internet at large in addition to just PyPi. 


What's the intended workflow for Pip? When a new cert gets blacklisted, will users need to run `$ pip install pip --upgrade`, or would an automatic CA Bundle update be preferred?


Pip will be relying on the CA bundle shipped with requests at this point in time. I have a todo list item to see about the possibility of getting it to use system certificates (would that be something requests itself would be interested in?) but either way it'll need to use the bundled certs on AIX and HP-UX.

As of right now there is no plans for an automatic CA Bundle update, in order to get a new bundle they'll need to get a new pip.


https://github.com/kennethreitz/cert-playground


@dstufft requests explicitly prefers not to rely on the system certs due to massive compatibility headaches the variance in distros often cause. I believe we have a utility function to enable this functionality built-in though. Will update shortly.


So, at this point in time, Pip is blocked only by our currently outdated CA Bundle, nothing else?


On the requests side that's all we require for `ensurepip` yes. The development branch of pip already is using requests with the older ca bundle.


@dstufft one thing to clarify here: historically, we only provide security releases for the latest release. We now have rigid API stability, so this shouldn't effect Pip at all. But, it is something to be aware of. 


e.g. if/when Requests v3.0.0 comes around (not currently roadmapped), I will recommend Pip upgrade to it. 


That sounds more or less reasonable as long as it doesn't remove anything we depend on :)


I'd rather not make this integration this rigid, but, if Pip wanted to provide an expected "api contract", it wouldn't be unwelcome.


The core of the SSL issue here is summed up in this conversation: http://twitter.theinfo.org/164062050971160576#id164067268924420098


I doubt you'll remove any of it, off the top of my head it's timeouts, TLS verification, http proxy support, CONNECT proxy support, and some system for allowing us to treat file:// urls similarly to http urls (the adaper thing works fine for this).


@dstufft are you aware of any sort of alert system for cert blacklisting? Is it possible to automate this? 

I may make a simple service for this.


I'm not aware of anything.

FWIW, Christian Heimes mentioned to me that instead of blacklisting we want white listing based on the CKA_TRUST_\* settings and enhanced key usage OIDs. Apparently there are certificates in the root store that are trusted, but not for HTTPS but only for code signing and such.


agl is a security person on the Chromium project, here's something they wrote to extract mozilla's certificates https://github.com/agl/extract-nss-root-certs/blob/master/convert_mozilla_certdata.go

It's written in go though.


Christian Heimes would be a good person to ping about this - he's been looking into a bunch of SSL issues for CPython, which is how the topic came up in relation to PEP 453 (see https://mail.python.org/pipermail/python-dev/2013-October/129755.html)

https://mail.python.org/pipermail/python-dev/2013-October/129738.html is the original post from Christian regarding cert management that prompted me to ask him to take a look at pip's SSL cert handling (which ended up resulting in the addition to the PEP 453 security considerations section already mentioned above)

http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0.1.5.html has more details regarding what eGenix ended up doing for their cert bundles.


@dstufft interesting. What "transformations" need to be done against Mozilla's raw cert file?

http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt

I believe the workflow I used was taking the certs provided by [curl's caextract](http://curl.haxx.se/docs/caextract.html) and manually removing blacklisted certs.


I'm trying to grok this go code to see what agl is doing to extract a list of trusted certificates.


Go is perfect if it functions properly. @dstufft if you're up for it, could you vet it?


I'm reading it but I don't really know go, but to be honest agl is an actual expert in this area. I'd trust anything he had to say or wrote to do this way more then I'd trust myself.


You know who does know go, right? @Lukasa that's who. ;)


@agl, any input you can provide would be much appreciated!