requests not able to validate the self signed certificate but i can use it using curl 

```
>>> requests.get('https://10.0.0.1:443/api', verify='/run/secrets/kubernetes.io/serviceaccount/ca.crt')
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 417, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.5/dist-packages/OpenSSL/SSL.py", line 1426, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.5/dist-packages/OpenSSL/SSL.py", line 1174, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.5/dist-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 424, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/dist-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
```
Python 3.5.2 
pyOpenSSL==16.2.0
requests==2.12.1
```
$ openssl verify /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt: CN = acsk8sca
error 18 at 0 depth lookup:self signed certificate
OK
```
with curl
```
curl -I -X GET https://10.0.0.1:443/api --header "Authorization: Bearer $TOKEN" --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 17 Nov 2016 21:57:41 GMT
Content-Length: 183
```

Can you show me the output of `openssl x509 -in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -noout -text` and the equivalent command run on the certificate being offered by 10.0.0.1 please?


Client:

```
root@deis-controller-1217413231-almny:/app# openssl x509 -in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6985765230565584782 (0x60f26c770c22ef8e)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=acsk8sca
        Validity
            Not Before: Nov 17 00:00:00 2016 GMT
            Not After : Nov 17 00:00:00 2018 GMT
        Subject: CN=acsk8sca
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:aa:35:75:dc:40:18:0d:23:bb:b4:f4:04:b0:42:
                    14:70:4a:03:de:15:9b:60:e7:c2:35:a2:26:fd:ac:
                    f2:f1:46:dd:9a:03:e3:2e:e9:a4:b6:74:f6:3f:2a:
                    e0:1d:7a:21:e4:66:46:d8:66:8a:ea:c9:5b:af:c8:
                    88:f5:9b:a6:54:0e:cd:94:51:18:15:1f:e2:8a:0e:
                    bc:e1:fe:4a:f3:96:e8:8c:7e:f5:02:62:b5:20:ab:
                    ba:2f:d0:43:96:a9:06:fa:8e:d0:f5:15:18:cd:5a:
                    97:72:fd:80:47:33:d2:be:4c:4c:53:53:1d:16:7d:
                    af:5d:a4:cc:29:60:9d:b5:6b:c2:4b:04:04:e0:c4:
                    ea:0c:7c:2d:74:d0:47:8a:c5:70:fe:7f:74:67:80:
                    ba:67:8f:2e:f1:f1:d6:02:ad:26:b1:3a:02:41:8d:
                    ab:b5:a1:d1:60:0e:d5:65:89:e7:e1:8a:5b:e7:80:
                    a3:2d:d3:ca:1a:22:bd:bc:46:c1:92:6f:88:b7:51:
                    e0:71:85:52:49:ba:a6:c8:c3:50:2d:ee:8f:c2:2e:
                    3e:f2:62:b5:b5:31:08:6a:ec:83:25:4f:45:9e:39:
                    b2:3c:b4:de:43:1c:eb:fd:bc:bc:0b:2b:a7:82:49:
                    d8:03:22:e2:b9:f9:00:52:49:d8:3a:b0:38:a1:11:
                    72:cf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:A9:16:EE:AB:29:FD:4E:96:97:C7:DB:E5:15:96:C3:41:DC:12:16:B7
                DirName:/CN=acsk8sca
                serial:60:F2:6C:77:0C:22:EF:8E

            X509v3 Subject Key Identifier: 
                A9:16:EE:AB:29:FD:4E:96:97:C7:DB:E5:15:96:C3:41:DC:12:16:B7
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         0b:aa:35:c1:5b:b3:97:a7:1a:34:d5:3d:08:c8:45:28:5d:95:
         70:8f:ec:e1:c0:08:31:b4:cc:fe:2b:c3:16:46:d2:d4:77:19:
         14:f5:02:67:05:b4:0b:0c:b4:5c:5a:c2:76:f3:73:92:bb:5b:
         13:5b:8c:56:45:e1:6c:08:c0:56:eb:cc:d0:68:f9:4f:5a:8a:
         5d:21:d2:31:56:fd:36:06:8f:71:3b:06:4a:5a:df:e3:b9:73:
         a8:86:59:59:aa:3b:47:04:14:71:a8:ac:09:4b:b0:03:08:22:
         aa:71:dc:16:b2:aa:9a:40:ac:d5:66:d9:5c:ec:c4:19:12:c7:
         27:23:3b:d7:fe:20:87:39:f8:f1:46:3d:4f:bd:c8:10:9f:5c:
         ff:a9:74:e8:f3:6b:37:3c:f7:d0:48:b8:29:7d:42:da:be:15:
         ac:60:2e:cf:ca:9e:3e:08:15:57:32:ac:de:87:f8:c7:a9:87:
         5d:fc:89:16:30:fa:d9:a7:a0:65:ba:03:33:aa:b3:15:72:d0:
         6f:7d:94:6d:9a:cf:18:b4:82:a0:2a:38:04:b5:76:53:f8:e0:
         35:4b:f3:6f:ce:db:c4:19:c8:1e:fd:d6:7f:15:b8:79:92:9f:
         38:31:21:13:8a:36:24:84:a4:5c:e1:20:1a:f0:f9:e2:55:6b:
         0d:66:02:c2
```

server :

```
openssl x509 -in  /etc/kubernetes/certs/apiserver.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 7125429247776946919 (0x62e29c2611ec7ae7)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=acsk8sca
        Validity
            Not Before: Nov 17 00:00:00 2016 GMT
            Not After : Nov 17 00:00:00 2018 GMT
        Subject: CN=apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:af:47:25:b5:d6:0d:2e:79:65:a3:e8:a7:6b:48:
                    53:b8:d6:0a:d1:fd:85:54:cb:ec:f9:5a:a2:75:6e:
                    fc:95:97:02:db:aa:1c:ca:8a:60:c7:28:a7:9e:49:
                    b7:94:16:c1:24:67:fb:08:82:cd:79:45:88:5a:a3:
                    8d:59:3d:3c:ea:d7:55:cc:76:a3:5a:e1:0a:4d:21:
                    c6:c7:f2:a5:37:ff:87:b0:38:0c:a1:19:7d:65:dd:
                    5e:55:2d:1c:f6:05:80:d7:80:cc:e3:89:8b:7c:69:
                    fa:74:4f:d6:aa:12:ab:a1:7c:60:34:7a:dd:ae:29:
                    e0:b9:4a:df:da:36:2c:e2:d7:d9:bb:b6:c8:0d:54:
                    87:19:ca:59:e5:c3:d9:02:7b:df:e1:16:33:09:e7:
                    25:6c:0d:ed:3d:99:48:13:8a:72:f0:87:bb:ca:9c:
                    87:64:81:27:53:bb:34:2a:07:80:38:3b:d0:3a:36:
                    24:15:52:68:c6:df:06:6c:18:4a:b2:ce:72:0a:72:
                    29:de:e1:09:e1:5b:ee:60:0a:4b:5f:14:0d:9e:07:
                    3d:32:0a:c2:65:9b:e6:d2:f5:dd:4a:e5:e8:0a:c8:
                    52:87:18:ef:af:46:f6:45:51:05:f3:3b:df:0a:f0:
                    f3:89:e4:b3:1e:c6:39:fa:c2:e4:0b:da:0b:ca:ba:
                    32:ab
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:A9:16:EE:AB:29:FD:4E:96:97:C7:DB:E5:15:96:C3:41:DC:12:16:B7
                DirName:/CN=acsk8sca
                serial:71:10:DD:F3:88:93:0D:A0

            X509v3 Subject Key Identifier: 
                FA:2E:D5:34:E5:8E:F4:A0:3C:86:5A:85:5F:2E:4E:E2:31:46:2A:92
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:kmalamgmt.centralus.cloudapp.azure.com, IP Address:10.240.255.5, IP Address:10.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
         46:69:ec:5f:a9:f9:bc:e9:26:5a:97:a9:05:45:3b:a4:46:ae:
         d2:7d:a1:3f:ff:43:97:7c:9e:0f:13:87:4e:28:4b:01:dd:e9:
         39:fe:36:76:1f:07:15:3a:9c:56:21:3a:6a:d3:58:6d:cd:ec:
         81:3e:4d:92:79:ec:cf:e4:0e:d5:56:01:5b:c5:bb:8e:8a:ad:
         4a:2c:1b:34:6f:5f:30:03:88:24:6a:09:d3:b9:79:b1:0e:03:
         01:cf:5d:66:4e:d0:e3:0f:0e:e5:b8:cf:7b:6a:58:25:5e:f0:
         d7:9a:00:5b:f7:37:4b:a9:bf:e4:ab:30:9c:fd:3c:6e:6b:56:
         f9:cb:8f:11:5a:12:16:d7:8a:10:70:2b:50:10:03:9b:48:34:
         88:1f:0c:1c:60:f3:b1:ca:0c:99:11:da:80:25:b0:c3:43:38:
         92:ee:71:b3:24:be:fa:bd:3b:0a:d8:61:89:d7:27:c5:c3:a1:
         2f:65:61:41:5e:1e:f2:ec:a1:ae:d9:55:48:6b:f2:64:49:d5:
         bd:1a:cb:f4:ca:c0:e7:35:46:4c:8b:79:2f:83:10:8c:52:1b:
         a6:db:b1:17:ce:02:9a:35:40:78:aa:78:19:5e:0b:12:47:8f:
         07:ca:e0:bf:ca:77:62:4a:9b:d5:f7:09:c8:b5:43:b9:b8:22:
         f0:a6:7f:21
```


Hrm. What's the output of `openssl verify -CApath /var/run/secrets/kubernetes.io/serviceaccount/ca.crt </path/to/server/certificate>`?


```
openssl verify -CApath client.crt /etc/kubernetes/certs/apiserver.crt
/etc/kubernetes/certs/apiserver.crt: CN = apiserver
error 20 at 0 depth lookup:unable to get local issuer certificate
```


Is `client.crt` in this case the `ca.cert` you were using earlier? (The changing cert names make this a bit tricky).


yes the client.crt is the same as `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`


Ok, so that result suggests that OpenSSL is failing to validate the cert. Can you please provide `openssl version` and `curl -V` for me?


```
/app# openssl version
OpenSSL 1.0.2g  1 Mar 2016
root@deis-controller-1217413231-almny:/app# curl -V
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets 
```


to verify my client cert shouldn't i do 

```
openssl verify -CApath  /etc/kubernetes/certs/apiserver.crt client.crt
client.crt: CN = acsk8sca
error 18 at 0 depth lookup:self signed certificate
OK
```


@kmala No. `client.crt` is the _root_ certificate, and so is not the CA. The `client.crt` is the CA that should be in the CApath.


Ok so that is probably why curl succeeded in validating: it isn't using OpenSSL for its cert validation, but Requests is.

My _suspicion_ is that the problem here is that the certificate serial number in the `Authority Key Identifier` field does not appear to match the serial number in the issuer certificate, but I'm not _sure_ of that. I'm going to page in @reaperhulk here to validate my answer.


It's quite possible that's the problem. The AKI is indeed wrong for serial number (fine for dirname and keyid) and OpenSSL will attempt to use AKI to build chains in some scenarios.

On a side note it's weird to have a self-signed CA that has an AKI. That's not a useful extension in that context.


Thanks @Lukasa @reaperhulk .Its indeed issue with the certificate. I used a certificate  without AKI and it works fine.