SSL: CERTIFICATE_VERIFY_FAILED on valid(?) cert chain with verify="./bundle.crt"

## Story

We are using requests in [MonitOwl](https://github.com/whitehats/monitowl-agent) agent to communicate with API servers, big "thank you" for great software. Few days ago we ordered new certificates for *.monitowl.com and deployed it to new instance, our issuer is `nazwa.pl`. Pointing a webbrowser to https://activeo.monitowl.com works flawlessly, requests seems to have some problems, when setting verify="./cabundle.crt".
## Technical notes

In front of the tornado server handling ssl there is a haproxy checking SNI and redistributing the traffic, part of the config file:

```
        acl app_activeo req_ssl_sni -i activeo.monitowl.com
        use_backend bk_activeo if app_activeo
```

We _need_ to provide own cabundle to have control what is accepted - because of security issues and deployment scripts. We are using debian jessie everywhere.

```
# from issuer https://panel.nazwa.pl/uploads/ssl/nazwaSSL_SHA-2.zip
$ cat monitowlcom.crt nazwasslsha2.pem certumca-ctncasha2.pem gscasha2.pem > ./ca_bundle.crt
# exported from firefox
$ cat monitowl.com nazwaSSL CertumTrustedNetworkCA CertumGlobalServicesCASHA2 > ca_bundle_ff.crt
$  openssl verify -untrusted ./ca.crt  monitowlcom.crt 
monitowlcom.crt: OK
$ c_rehash ./
$ openssl s_client -CApath ./  -connect activeo.monitowl.com:443 -servername activeo.monitowl.com
CONNECTED(00000003)
depth=4 C = PL, O = Unizeto Sp. z o.o., CN = Certum CA
verify return:1
depth=3 C = PL, O = Unizeto Technologies S.A., OU = Certum Certification Authority, CN = Certum Trusted Network CA
verify return:1
depth=2 C = PL, O = Unizeto Technologies S.A., OU = Certum Certification Authority, CN = Certum Global Services CA SHA2
verify return:1
depth=1 C = PL, O = nazwa.pl S.A., OU = http://nazwa.pl, CN = nazwaSSL
verify return:1
depth=0 C = PL, CN = *.monitowl.com, emailAddress = admin@whitehats.pl
verify return:1
[...]
*    Verify return code: 0 (ok)*

$ curl -I --cacert ca_bundle.crt https://activeo.monitowl.com 
HTTP/1.1 200 OK
$ curl -I --cacert ca_bundle_ff.crt https://activeo.monitowl.com 
HTTP/1.1 200 OK

```

As you can see, the `openssl s_client` verifies the connection, curl works without problems. 
## Problem

```
In [1]:  import requests
In [3]: requests.get('https://activeo.monitowl.com', verify=True)      
Out[3]: <Response [200]>
In [4]: requests.get('https://activeo.monitowl.com', verify="./ca_bundle.crt")
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
<ipython-input-3-c92a3091d6ce> in <module>()
----> 1 requests.get('https://activeo.monitowl.com', verify="./ca_bundle.crt")

/home/vagrant/.virtualenvs/test/local/lib/python2.7/site-packages/requests/api.pyc in get(url, params, **kwargs)
     67 
     68     kwargs.setdefault('allow_redirects', True)
---> 69     return request('get', url, params=params, **kwargs)
     70 
     71 

/home/vagrant/.virtualenvs/test/local/lib/python2.7/site-packages/requests/api.pyc in request(method, url, **kwargs)
     48 
     49     session = sessions.Session()
---> 50     response = session.request(method=method, url=url, **kwargs)
     51     # By explicitly closing the session, we avoid leaving sockets open which
     52     # can trigger a ResourceWarning in some cases, and look like a memory leak

/home/vagrant/.virtualenvs/test/local/lib/python2.7/site-packages/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    463         }
    464         send_kwargs.update(settings)
--> 465         resp = self.send(prep, **send_kwargs)
    466 
    467         return resp

/home/vagrant/.virtualenvs/test/local/lib/python2.7/site-packages/requests/sessions.pyc in send(self, request, **kwargs)
    571 
    572         # Send the request
--> 573         r = adapter.send(request, **kwargs)
    574 
    575         # Total elapsed time of the request (approximately)

/home/vagrant/.virtualenvs/test/local/lib/python2.7/site-packages/requests/adapters.pyc in send(self, request, stream, timeout, verify, cert, proxies)
    429         except (_SSLError, _HTTPError) as e:
    430             if isinstance(e, _SSLError):
--> 431                 raise SSLError(e, request=request)
    432             elif isinstance(e, ReadTimeoutError):
    433                 raise ReadTimeout(e, request=request)

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

In [5]: requests.get('https://activeo.monitowl.com', verify="./ca_bundle_ff.crt")    
[same error]
In [7]:  import ssl; ssl.OPENSSL_VERSION
Out[7]: 'OpenSSL 1.0.1k 8 Jan 2015'
In [8]: import sys; print (sys.version)
2.7.9 (default, Mar  1 2015, 12:57:24) 
[GCC 4.9.2]
In [2]: requests.__version__
Out[2]: '2.7.0'

```

On server side (tornado):

```
 SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:581)
```

I asked some friends to test it with the bundle exported from webbrowser and:
Python==2.7.9 + OpenSSL==1.0.1k => FAILS
Python==2.7.10 + OpenSSL==1.0.1k => FAILS
Python==2.7.9 + OpenSSL==1.0.1l => FAILS
Python==2.7.10 + OpenSSL==1.0.1p => WORKS
Python==2.7.10 + OpenSSL==1.0.2d => WORKS
Python==2.7.9 + OpenSSL 1.0.2d  => WORKS
I'm aware that python verifies the certs [differently](http://stackoverflow.com/questions/27804710/python-urllib2-ssl-error/27826829#27826829) than web browsers.

Do you have any clues what is wrong with the bundle? What else can I check? It's a bug? In requests, openssl, python or urllib3?


I've posted a [$30 bounty for this](https://www.bountysource.com/issues/26878061-ssl-certificate_verify_failed-on-valid-cert-chain-with-verify-bundle-crt).

I've tested the server with [ssl checker](https://www.ssllabs.com/ssltest/analyze.html?d=activeo.monitowl.com&latest) and there was no issues, beside accepting old crypto algorithms..


Sorry about the delay on this: due to some bad luck on your end, both core maintainers were on holiday at the same time when you raised this issue.

What cert are you providing in the bundle?


The StackOverflow link you provided has the core bug summary, but I'm finding that in this case it's not so hard to resolve. Can you try using just this as your cert bundle and reproducing your test?

```
-----BEGIN CERTIFICATE-----
MIIDDDCCAfSgAwIBAgIDAQAgMA0GCSqGSIb3DQEBBQUAMD4xCzAJBgNVBAYTAlBM
MRswGQYDVQQKExJVbml6ZXRvIFNwLiB6IG8uby4xEjAQBgNVBAMTCUNlcnR1bSBD
QTAeFw0wMjA2MTExMDQ2MzlaFw0yNzA2MTExMDQ2MzlaMD4xCzAJBgNVBAYTAlBM
MRswGQYDVQQKExJVbml6ZXRvIFNwLiB6IG8uby4xEjAQBgNVBAMTCUNlcnR1bSBD
QTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM6xwS7TT3zNJc4YPk/E
jG+AanPIW1H4m9LcuwBcsaD8dQPugfCI7iNS6eYVM42sLQnFdvkrOYCJ5JdLkKWo
ePhzQ3ukYbDYWMzhbGZ+nPMJXlVjhNWo7/OxLjBos8Q82KxujZlakE403Daaj4GI
ULdtlkIJ89eVgw1BS7Bqa/j8D35in2fE7SZfECYPCE/wpFcozo+47UX2bu4lXapu
Ob7kky/ZR6By6/qmW6/KUz/iDsaWVhFu9+lmqSbYf5VT7QqFiLpPKaVCjF62/IUg
AKpoC6EahQGcxEZjgoi2IrHu/qpGWX7PNSzVttpd90gzFFS269lvzs2I1qsb2pY7
HVkCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEA
uI3O7+cUus/usESSbLQ5PqKEbq24IXfS1HeCh+YgQYHu4vgRt2PRFze+GXYkHAQa
TOs9qmdvLdTN/mUxcMUbpgIKumB7bVjCmkn+YzILa+M6wKyrO7Do0wlRjBCDxjTg
xSvgGrZgFCdsMneMvLJymM/NzD+5yCRCFNZX/OYmQ6kd5YCQzgNUKD73P9P4Te1q
CjqTE5s7FCMTY5w/0YcneeVMUeMBrYVdGjux1XMQpNPyvG5k9VpWkKjHDkx0Dy5x
O/fIR/RpbxXyEV6DHpx8Uq79AtoSqFlnGNu8cN2bsWntgM6JQEhqDjXKKWYVIZQs
6GAqm4VKQPNriiTsBhYscw==
-----END CERTIFICATE-----
```


Yeah, with your bundle it works. But it's SHA1, not SHA256.

My bundle (SHA256 only):

```
-----BEGIN CERTIFICATE-----
MIIEbjCCA1agAwIBAgIQUooUzakaS0WlY3wDZH9CwTANBgkqhkiG9w0BAQsFADCB
gzELMAkGA1UEBhMCUEwxIjAgBgNVBAoTGVVuaXpldG8gVGVjaG5vbG9naWVzIFMu
QS4xJzAlBgNVBAsTHkNlcnR1bSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEnMCUG
A1UEAxMeQ2VydHVtIEdsb2JhbCBTZXJ2aWNlcyBDQSBTSEEyMB4XDTE0MDkzMDEx
NTQxOFoXDTI0MDkyNzExNTQxOFowUjELMAkGA1UEBhMCUEwxFjAUBgNVBAoMDW5h
endhLnBsIFMuQS4xGDAWBgNVBAsMD2h0dHA6Ly9uYXp3YS5wbDERMA8GA1UEAwwI
bmF6d2FTU0wwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDjAfhBk7mv
Z5MlDV34icf8DwkugIYgm6bKBI4aZKa+SoL7tb64OymtOhZxLX2XRMhKqj84GcUE
jg0yQ0vkG/CS/ssJLqo1t56mZJXi5wFCP8eWYcIJoVvGG1QIe1oG1Umtm1dUeDT+
ORHoejbp5XQsubD4bAw3f/vAaBztEKRxB2fGRa07ruY20z4Tm1cKKXASwexzftzQ
g0F/mGm4bAiIm5nodphpWVdqaMxZDXhe7ix90ziUR33P/gbxFpZDGMhQIk8bcZbN
1Cvq9URgKZ7jLkZdbQIBJGN2vMMOaPMb/nt2cyvNZcyRMZoP4u+p6x88SEul9zlw
ufjsV2Qdbk3PAgMBAAGjggEMMIIBCDASBgNVHRMBAf8ECDAGAQH/AgEAMDIGA1Ud
HwQrMCkwJ6AloCOGIWh0dHA6Ly9jcmwuY2VydHVtLnBsL2dzY2FzaGEyLmNybDBu
BggrBgEFBQcBAQRiMGAwKAYIKwYBBQUHMAGGHGh0dHA6Ly9zdWJjYS5vY3NwLWNl
cnR1bS5jb20wNAYIKwYBBQUHMAKGKGh0dHA6Ly9yZXBvc2l0b3J5LmNlcnR1bS5w
bC9nc2Nhc2hhMi5jZXIwHwYDVR0jBBgwFoAUVJndm//opw6jGZ1bvkJX3zD8jzIw
HQYDVR0OBBYEFLJhdoKVWgE3F5fCdyG17Yv8KfdnMA4GA1UdDwEB/wQEAwIBBjAN
BgkqhkiG9w0BAQsFAAOCAQEAHnOtSeZgWlG3Ly/I0QrG7T0eviYdjtHNbmddiuPR
k9UVLoQtDU53MjKl5YvoMWVfUe3ROymD0gvPfh44luLRFzdZ+IO21z90DGEq4utz
uU4DLKI5uzYffuUOAY3k5839ZnEs/1h0yZ4VHcF95IY0IcQ91W0rc3cm0/1WEj++
bcMm4CZzKkbX61+7pnjIw3eAKotNQ3b6QDOnAgNhZSVBTzBJtnoBL4BsoVubYmbP
wOq/vvWmKj901NYI2Iawm6W58hJrI+xEn/vG4vq2Hpku3YE9CKDq8kqjILNxn9n/
d7w1te4vPp7cUhW+2DVfpjPbYWHbbPhDvEgOYG+lsQVb4w==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEzTCCA7WgAwIBAgIRANBLb+XdW9Ih58dM9kaLMUYwDQYJKoZIhvcNAQELBQAw
fjELMAkGA1UEBhMCUEwxIjAgBgNVBAoTGVVuaXpldG8gVGVjaG5vbG9naWVzIFMu
QS4xJzAlBgNVBAsTHkNlcnR1bSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEiMCAG
A1UEAxMZQ2VydHVtIFRydXN0ZWQgTmV0d29yayBDQTAeFw0xNDA5MTExMjAwMDBa
Fw0yNzA2MDkxMDQ2MzlaMIGDMQswCQYDVQQGEwJQTDEiMCAGA1UEChMZVW5pemV0
byBUZWNobm9sb2dpZXMgUy5BLjEnMCUGA1UECxMeQ2VydHVtIENlcnRpZmljYXRp
b24gQXV0aG9yaXR5MScwJQYDVQQDEx5DZXJ0dW0gR2xvYmFsIFNlcnZpY2VzIENB
IFNIQTIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDHhvLXeA4kyBVY
Dr12JusjW+/HPPWpBmWnSZC9gadbNXCD/GlHrGruRWrKn0YBasbonXi5FLRon717
aCCDt2+7URnJx4mLGt3X73yH3I+T+COuyTFOuLId68jYV4vfBm1N/N1/KohGr8+R
7eT4f4agfXVQ+gDZT3pGTywqc9IId4uHaFjW+mr5vVfu7WjL8A20jOawvlqV2K4/
qSrjPZsqHlzf46LHgbQEK1EjTPsFrfvAQxvGpHiEIAFsS+1d2xPIEJVIxvq+KyGb
W/0SJIj7/SfYo+ItvJ2a4/G2JfoPF9wTCC0N8U5I/KXxjYYXBcIE9pQ16T24T+dr
MfPG2mYtAgMBAAGjggE+MIIBOjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRU
md2b/+inDqMZnVu+QlffMPyPMjAfBgNVHSMEGDAWgBQIds3LB/8k9sXN7buQvOKE
N0Z19zAOBgNVHQ8BAf8EBAMCAQYwLwYDVR0fBCgwJjAkoCKgIIYeaHR0cDovL2Ny
bC5jZXJ0dW0ucGwvY3RuY2EuY3JsMGsGCCsGAQUFBwEBBF8wXTAoBggrBgEFBQcw
AYYcaHR0cDovL3N1YmNhLm9jc3AtY2VydHVtLmNvbTAxBggrBgEFBQcwAoYlaHR0
cDovL3JlcG9zaXRvcnkuY2VydHVtLnBsL2N0bmNhLmNlcjA5BgNVHSAEMjAwMC4G
BFUdIAAwJjAkBggrBgEFBQcCARYYaHR0cDovL3d3dy5jZXJ0dW0ucGwvQ1BTMA0G
CSqGSIb3DQEBCwUAA4IBAQDRhC+yVHfF/IFcNNKbkxX1aVsCZ0fa2t+MxyFcs7KO
qW1fGpOOycughOFnbM+lz4Y3gt5RaOFJTm7YVUZZ3751s5tv8nhXeXfrRIpQN7Cu
+Nei457HlDxEUItPlkYnDbdDes/96T19cICd1TmIPekYRXiyuPW4Wgx6vykmk91x
LkJ0y74TzVtUofVF446qXvea95zNpzYCVMg+AOX3ZZyy9XfST6g4um+cw/Idv31d
bnJdBzMOgHH3uw2YMiZQgDqvNRE+wAs+PTFEIKHmBc/t1n3Shvg9e68M+5ZRM8bE
WGqgLqfreTgCsCQcv9MDYw9TFUbS17RdE6NtiPfszTky
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEtDCCA5ygAwIBAgIRAJOShUABZXFflH8oj+/JmygwDQYJKoZIhvcNAQELBQAw
PjELMAkGA1UEBhMCUEwxGzAZBgNVBAoTElVuaXpldG8gU3AuIHogby5vLjESMBAG
A1UEAxMJQ2VydHVtIENBMB4XDTA4MTAyMjEyMDczN1oXDTI3MDYxMDEwNDYzOVow
fjELMAkGA1UEBhMCUEwxIjAgBgNVBAoTGVVuaXpldG8gVGVjaG5vbG9naWVzIFMu
QS4xJzAlBgNVBAsTHkNlcnR1bSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEiMCAG
A1UEAxMZQ2VydHVtIFRydXN0ZWQgTmV0d29yayBDQTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAOP7faNyusLwyRSH9WsBTuFuQAe6bSddf/dbLbNax1Ff
q6QypmGHtm4PhtIwApf412lXoRg5XWpkecYBWaw8MUo4fNIE0kso6CBfOweizE1z
2/OuT8dW1Vqnlon686to1COGWSfPCSe8rG5ygxwwct/gounS4XR1Gb0qnnsVVAQb
10M5rVUoxeIau/TA5K44STPMdoWfOUXSpJ7yEoxR+HzkLX/1rF/rFp+xLdG6zJFC
d0wlyZA4b9vwzPuOHpdZPtVgTuYFKO1JeRNLukjbL/ly0znK/h/YNHL1tEDPMQHD
7N4RLRddH7hQ0V4Zp2neBzMoylCV+adUy1SGUEWp+UkCAwEAAaOCAWswggFnMA8G
A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFAh2zcsH/yT2xc3tu5C84oQ3RnX3MFIG
A1UdIwRLMEmhQqRAMD4xCzAJBgNVBAYTAlBMMRswGQYDVQQKExJVbml6ZXRvIFNw
LiB6IG8uby4xEjAQBgNVBAMTCUNlcnR1bSBDQYIDAQAgMA4GA1UdDwEB/wQEAwIB
BjAsBgNVHR8EJTAjMCGgH6AdhhtodHRwOi8vY3JsLmNlcnR1bS5wbC9jYS5jcmww
aAYIKwYBBQUHAQEEXDBaMCgGCCsGAQUFBzABhhxodHRwOi8vc3ViY2Eub2NzcC1j
ZXJ0dW0uY29tMC4GCCsGAQUFBzAChiJodHRwOi8vcmVwb3NpdG9yeS5jZXJ0dW0u
cGwvY2EuY2VyMDkGA1UdIAQyMDAwLgYEVR0gADAmMCQGCCsGAQUFBwIBFhhodHRw
Oi8vd3d3LmNlcnR1bS5wbC9DUFMwDQYJKoZIhvcNAQELBQADggEBAI3m/UBmo0yc
p6uh2oTdHDAH5tvHLeyDoVbkHTwmoaUJK+h9Yr6ydZTdCPJ/KEHkgGcCToqPwzXQ
1aknKOrS9KsGhkOujOP5iH3g271CgYACEnWy6BdxqyGVMUZCDYgQOdNv7C9C6kBT
Yr/rynieq6LVLgXqM6vp1peUQl4E7Sztapx6lX0FKgV/CF1mrWHUdqx1lpdzY70a
QVkppV4ig8OLWfqaova9ML9yHRyZhpzyhTwd9yaWLy75ArG1qVDoOPqbCl60BMDO
TjksygtbYvBNWFA0meaaLNKQ1wmB1sCqXs7+0vehukvZ1oaOGR+mBkdCcuBWCgAc
eLmNzJkEN0k=
-----END CERTIFICATE-----
```


The problem with the SHA256 bundle is that the 'root' cert in that case is cross-signed with the SHA-1 cert, but older OpenSSL's wont like that at all. The way older OpenSSL builds a cert chain is that it builds the longest one it can out of the certs provided by the remote server, and then looks for a trusted certificate that signs that chain. If it can't find that certificate, it'll bail out, without checking whether it can build a shorter chain using its trust roots.

This is an ongoing problem with the deprecation of SHA1 root certs: see certifi/python-certifi#26. Unfortunately, the only fixes are to temporarily use the SHA1 root in your trust store or to upgrade to a newer OpenSSL version.


Oh. Shame on you Python.

Thank you, Lukasa for explaining this. As it works for now, closing. Next year, new certificate - hope Python will got support for cross signed certificates.


I was looking for Certum API via cURL [ Certum Certificate](https://sslretail.com/certum/) but it seems to work only via  __soapCall only.