SSL certificate caching

I have a question regarding caching of ssl certificates. I switched over to a sha2 certificate which has 2 intermediate certificates. This all works fine and requests is working like expected.
The problem is the size of the certificates. It was a 2KB certificate before, but now with the 2 intermediate certificates in there it's at 7KB. This doesn't seem like much, but the server I'm running it on has over 25million requests each day. With the content send per requests well below 10KB. Practically doubling my traffic by just adding the new certificate.

I tried searching on how the certificate are used. Is it a 1 time thing and then cached on the client side?, is the certificated send with each request? etc. But I keep finding the basics of how and what certificates do.

Could you tell me how `requests` uses the certificate and if there is a way on somehow caching it in the session object for example. Making sure it doesn't get requested with each call.
Maybe I'm asking something weird here, so please let me know if it just "how http works" ;)


@RuudBurger The certificate chain is sent with each SSL handshake, which is performed for each new connection. If you use a `requests` Session it will use a connectionpool, which reuses connections for each host.
As you are talking to the same host only one handshake will be performed.


It's not "how HTTP works", it's "how SSL/TLS works". =)

TLS/SSL authenticates a single _TCP connection_. The handshake is done at the beginning, including certificate transfer, and then never again. However, it's very important that the certificate chain be sent at the start of each connection. Thus, for connections that don't trust your intermediate certificates (but only trust the root), you _must_ send the whole cert chain on each connection.

Thus, the general way to improve this is to make sure connections last a while. Make sure your server doesn't send `Connection: close` on its responses, and that wherever possible you re-use connections. Requests will do this if you use a `Session` object. If you aren't already, try that: it should minimise some of your cost.

If requests is your only client (that is, you aren't worried about whether third parties have errors verifying the trust chain), you _can_ improve the situation further. The way to do it is to tell requests to trust the intermediate certificate directly. That means putting your intermediate certificate into a single file in `pem` format, and then passing the path to that file as the argument to `verify` on your requests calls. Then your server only needs to provide its certificate: requests will verify that it trusts the certificate that issued it, and all should be well.


@Lukasa Instead of trusting the intermediate, would it not be easier then to use a single self-signed cert and thrust it directly or just pin the fingerprint?


That's absolutely an alternative option. Pinning the fingerprint is harder, I don't know how you'd do it with requests' API, but the self-signed cert would work. Similarly, trusting the cert directly is also an option.


@RuudBurger I assume the 7kb you measured are the PEM-encoded files. This will be less when transmitted over the wire.


Wow thanks for the quick response guys.
Is there a way to test and see if a connection is re-used in requests session? so I can find the best timeout from the server side to close the connection. I do have a single session object for all outgoing requests so this should already be the case, just need to find out it is.

With so many people connecting I don't know if I can keep a lot of connections open though, need to figure out how nginx handles this.

Yeah the 7kb is the combined size of the certificates. A regular response body of my api is 1.7KB gzipped. I don't think the handshake is compressed though, read somewhere there were issues with that so nginx disabled it by default.
My traffic went from 5GB/hour to over 10GB/hour, this is why I noticed.


The easiest way to check whether requests is re-using the connection is to turn on DEBUG level logging. urllib3 will log out its connection pool logic. Post the logs here and I can tell you how to read them. =)


Don't know if these are it, seems a bit low on the info ;)

```
02-22 13:59:30 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt1262416/, data: []
02-22 13:59:30 INFO Starting new HTTPS connection (1): api.couchpota.to
02-22 13:59:30 DEBUG "GET /info/tt1262416/ HTTP/1.1" 200 None
02-22 13:59:30 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt0800369/, data: []
02-22 13:59:30 INFO Resetting dropped connection: api.couchpota.to
02-22 13:59:30 DEBUG "GET /info/tt0800369/ HTTP/1.1" 200 None
02-22 13:59:30 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt2494280/, data: []
02-22 13:59:30 INFO Resetting dropped connection: api.couchpota.to
02-22 13:59:30 DEBUG "GET /info/tt2494280/ HTTP/1.1" 200 None
```

without connection: close

```
02-22 14:00:36 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt1262416/, data: []
02-22 14:00:36 INFO Starting new HTTPS connection (1): api.couchpota.to
02-22 14:00:36 DEBUG "GET /info/tt1262416/ HTTP/1.1" 200 None
02-22 14:00:36 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt0800369/, data: []
02-22 14:00:36 DEBUG "GET /info/tt0800369/ HTTP/1.1" 200 None
02-22 14:00:37 INFO [hpotato.core.plugins.base] Opening url: get https://api.couchpota.to/info/tt2494280/, data: []
02-22 14:00:37 DEBUG "GET /info/tt2494280/ HTTP/1.1" 200 None
```


That's what we needed. =)

The relevant messages are 'Starting new HTTPS connection' and 'Resetting dropped connection'.

Each time you see one of them it means we're doing the handshake again. 'Resetting dropped connection' is an indication that we still have the socket object, but that we believe it to be closed. What you want is to see as few 'Starting new HTTPS connection' messages as possible, and ideally no 'resetting dropped connection' messages.

What's in the headers on the request and response?

If you want, you can correlate these with tcpdump showing connection closures to verify what I'm saying.


https://github.com/RuudBurger/CouchPotatoServer/blob/master/couchpotato/core/plugins/base.py#L195-L200 these are the default headers for the request so I need to change the "connection close" part.

This is the response:

```
Content-Encoding    gzip
Content-Type    application/json; charset=utf-8
Date    Sun, 22 Feb 2015 13:06:33 GMT
Server  nginx
Strict-Transport-Security   max-age=31536000; includeSubdomains;
Vary    Accept-Encoding
X-Powered-By    CouchPotato (3007)
```

Thanks for the help, I'll do some testing and see if I can keep the connections open a bit longer. Offload the traffic a bit. 


Correct, `Connection: close` won't help you here. =)


I can't believe I kept the `connection: close` in there for so long. I think this is code from before I used requests and the session object. Not re-using the connections..
Only enabling this took a normal "api use scenario" from 6.5s to 1.8s.. 

I'll keep an eye out if the fixes helped with the traffic. But because the propagation of the update in the app takes a while I don't know until a couple of weeks when all versions have the fix. Will let it know here, for other people coming by.


Did and update to all users, most active users are using the `keep-alive` fix now. Traffic is normal and server load is way down. Thanks for the help!


No problem, glad I could help!