REQUESTS_CA_BUNDLE does not fall back on system cert chain in Python 3.6?

I'm getting SSL validation errors for certificates signed by well-known root CAs when using the `REQUESTS_CA_BUNDLE` environment variable in Python 3.6 from a project that was working in Python 3.5.

My use case is, let's say I have an app that uses requests to talk to both https://localhost:8080 and https://google.com.  I use a self-signed certificate for comms with localhost and rely on the system's SSL trust store for the comms with google.

Is anyone else seeing this issue?


## Environment

| | Version |
| ----- | ---- |
| OS | macOS Sierra 10.12.3 |
| OpenSSL | 0.9.8zh (the default one that comes with macOS Sierra) |
| Requests | 2.12.1, 2.13.0 |
| Python | 3.6.0 |


## Steps to Reproduce

```
____ [~/code/_experiments/requests_ca_bundle_problem] ____ $ cat openssl.cnf
[req]
default_bits = 2048
default_md = sha256
default_days = 30
prompt = no
distinguished_name = req_distinguished_name

[req_distinguished_name]
countryName = "US"
stateOrProvinceName = "XX"
localityName = "X"
organizationName= "X"
organizationalUnitName = "X"
commonName = "localhost"

[add_san]
subjectAltName = DNS:localhost

____ [~/code/_experiments/requests_ca_bundle_problem] ____ $ openssl req -x509 -new -nodes -config openssl.cnf -extensions add_san -keyout key.pem -out cert.pem
Generating a 2048 bit RSA private key
.........+++
.................................................+++
writing new private key to 'key.pem'
-----

____ [~/code/_experiments/requests_ca_bundle_problem] ____ $ REQUESTS_CA_BUNDLE=cert.pem python3 -c 'import requests; requests.get("https://google.com")'
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
```

I don't think you're actually using 0.9.8zh. What's `python3 -c 'import ssl; print(ssl.OPENSSL_VERSION)'` print?



@Lukasa: Interesting—I'd have thought it's using the system's but apparently it's bundled one.

```
____ [~] ____ $ openssl version
OpenSSL 0.9.8zh 14 Jan 2016

____ [~] ____ $ python3 -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.2j  26 Sep 2016
```

Would that be the cause?  FWIW, the calls to https://localhost:8080 validate successfully when using `REQUESTS_CA_BUNDLE=cert.pem`, it's everything else that fails.

@dbazile are the cert chains for the other websites in `cert.pem`? If not then that is absolutely the expected behaviour.

@sigmavirus24: They're not in `cert.pem`.

But it's not the same behavior that I saw in Python 3.5 (which checked `REQUESTS_CA_BUNDLE` first and fell back on the system's trust store for everything else).

Is there a way to configure this previous behavior without `cat`ing the entire system's trust chain into cert.pem?

Perhaps 3.5 did not use a bundled OpenSSL, but whenever your OpenSSL isn't linked against the (horrific) Apple provided one, you will see exactly what you're seeing now.

So, to clarify: Apple's bundled OpenSSL will check against our cert bundle and then, if validation fails, it will check against the system. This is a dangerous behaviour because it makes it very hard to validate that the correct trust roots are being used. 

Right now Requests does not support using the macOS trust roots. If you want to, the `security` command line tool can be used to extract the system trust roots to PEM. 

I think you're right; Python 3.5 didn't bundle OpenSSL but 3.6 does:

```
____ [terra:~/code/ssl] ____ $ ls /Library/Frameworks/Python.framework/Versions/3.6/lib/*ssl*
/Library/Frameworks/Python.framework/Versions/3.6/lib/libssl.1.0.0.dylib /Library/Frameworks/Python.framework/Versions/3.6/lib/libssl.dylib

____ [terra:~/code/ssl] ____ $ ls /Library/Frameworks/Python.framework/Versions/3.5/lib/*ssl*
ls: /Library/Frameworks/Python.framework/Versions/3.5/lib/*ssl*: No such file or directory
```

I think my solution is to grab and concat all the certs my code will be using into an actual bundle and use that instead.

Thank you much, @Lukasa @sigmavirus24!