Close keep-alive connections after configurable period (aka keep-alive timeout)

I am experiencing a race condition with a (Squid) proxy server, best described by @rdharrison2 in https://github.com/kennethreitz/requests/issues/2364#issuecomment-186214402:

> In my case I'm making a large number of periodic https requests to a apache2 server with connection keep-alive timeout 5s. There is a race between the server closing a timed out persistent connection and the client making a request on the same connection. Given a bit of network latency the client can often attempt to send a request on the socket before it gets notified it as been closed.

Rather than implementing a retry logic, I would like to ensure that my requests `Session` keep-alive "timeout" is lower than that of proxy's. But I cannot seem to find out how.

Either I'm blind or it's just not supported?

Requests doesn't have a keep-alive timeout, because such a notion doesn't make sense. HTTP "keep-alive" doesn't involve any socket activity: it just involves each end of the connection promising to keep it open as long as they can.

In this case, what happens is that when you go to make a request, Requests checks whether the socket appears to still be open (assuming it was on a keep-alive connection) by asking whether it's readable. If it is, that means the remote peer has closed the socket. If it isn't, then Requests goes ahead and sends the request on that connection. There is a small timing window between these two events during which a connection closure can be received.

In this instance, a retry is the correct thing to do. However, you don't have to implement retry logic yourself: just use urllib3's. Create and mount new [HTTPAdapter](http://docs.python-requests.org/en/master/api/#requests.adapters.HTTPAdapter) that has `max_retries` set to a value greater than 0.


@Lukasa Thanks for the quick reply. I guess using `max_retries` would make the code a bit more fault-tolerant in generic sense.

However, to fail-fast at least when developing, running tests or continuous integration, I would rather not retry but simply avoid using those connections that are in risk of being closed (I know proxy server's keep-alive timeout so I could just set my value to a second or two lower than that). Do you see this as totally stupid thing to do?


> Do you see this as totally stupid thing to do?

The bigger problem is that this doesn't actually solve the issue.

Specifically, attempting to match your keep-alive timeout to the proxy's timeout can misbehave in a few different ways. The first, and most egregious, is that there is no reason to assume the clock as requests sees it will match the clock as the proxy sees it. Clock drift in distributed systems is well established, and two ends of a TCP connection form a distributed system: for that reason, it's all too easy for the view of the world that Requests has to become skewed from the view of the world that the proxy has, which can lead you to falling back into the same problem you have right now.

Another issue is that the proxy's keep-alive value is presumably not a _promise_ but a _target_. In particular, most servers reserve the right to close connections earlier than the keep-alive timeout if they need to (e.g. if they are running out of file handles). Once again, that means the configurable timeout is problematic.

A third concern is that the Python standard library doesn't provide access to a monotonic clock until Python 3.3, meaning that such a feature would have alarming failure modes during DST switches on earlier versions of Python.

A fourth concern is that different systems may handle monotonic clocks differently, which means that during DST switches we can still encounter problem 1 even when we _do_ have a monotonic clock.

A fifth concern is that a keep-alive switch like this becomes an attractive nuisance. Users will be inclined to set it to try to avoid problems like this, but it will be a bad solution to that problem _and_ it will reduce the efficiency of their connection pooling by forcing perfectly good connections to close.

Given that retries in this situation are totally safe, and that they are a complete solution, and that they represent a substantial efficiency gain, and that the support for them is already built into the code, and that they work across all versions of Python on all systems, I'm not quite sure what advantage you perceive you'll get out of having the keep-alive switch.


Well founded reasoning, I'll take your advice and use `HttpAdapter` with `max_retries`.

A few thoughts out of curiosity:
1. I wouldn't think clock drifting is a real problem here, because it's often quite minor and would not have any real effect when talking about days and weeks rather than seconds and minutes.
2. Interesting.
3. I wasn't even aware of such concept... Looks like there's a [backport in pypi](https://pypi.python.org/pypi/monotonic), but yeah it would be an extra dependency and doesn't look like it would run on PyPy and so on.
4. Cool. I mean, not cool.
5. True...


> 1. I wouldn't think clock drifting is a real problem here, because it's often quite minor and would not have any real effect when talking about seconds and minutes rather than days and weeks.

Heh, if only.

The reality is that clock drift can happen in all kinds of really exciting ways, all of which can affect real programs. Rather than go into much detail here, if you're interested I'll point you at [this blog post](https://aphyr.com/posts/299-the-trouble-with-timestamps) by the excellent Kyle Kingsbury (not tagged on GitHub because he won't care about this issue) which will go into very real detail about the many, many ways that clocks on computers can go wrong; and I'll also point you at [Julia Evans' post on clock skew](https://jvns.ca/blog/2016/02/09/til-clock-skew-exists/) which contains lots of good resources.

The high level thing here is that while clock drift _can_ be minor, it can also be _very major_, and in general it is best to avoid relying on a clock to achieve the affects you want if you can rely on something else instead.

Any application you're going to want to deploy in production should have retry logic in it somewhere, and urllib3 makes _very powerful_ retry logic available if you want it. The urllib3 [`Retry` object](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry) can be passed to the `max_retries` parameter if you want fine-grained control and some serious, hard-core resilience in the face of certain kinds of errors.


Uh, Aphyr always gives me goosebumps... I'll check your pointers. Thanks for the guidance!


@Lukasa I'm facing a similar issue where the server keep-alive connections for 30s, and I get sporadic errors when the client reuses a connection that the server is. Unfortunately, I'm making a POST request, and I cannot safely use the urllib3 `Retry` classes without risking causing duplicate updates. 

I'm wondering if I set a smaller `timeout` value when making the post request using a `Session` can help make sure the client close the connection before the server does? Essentially I'm trying to see if there is a way to ensure the connection is always closed by the client before the server.