Use of Session for repeated requests subject to TCP timing problem

This is a bug that I originally experienced using the InfluxDB-python client and reported [here](https://github.com/influxdb/influxdb-python/issues/103). Upon further investigation, I've been able to reproduce it using only the requests library (which InfluxDB-client uses).

Running a very simple script making repeated requests to a remote URL will result in a failed request if a certain timing condition is met,  resulting in a thrown `requests.exceptions.ConnectionError`. 

Here is the script I used to reproduce the issue:

```
import requests
import time 
session = requests.Session()

while 1:
    try:
        response = session.request(
            method='GET',
            url="http://mediaqueri.es/",
            timeout=15
        )
        print response.status_code, response.text[0:30]
    except Exception as e:
        print e, type(e)
    time.sleep(5)
```

The site I'm GETting is just a publicly available site that happens to be running on the Flask framework. About one in ten requests to the site will fail, and the following line will be printed by the exception handler shown above: `HTTPConnectionPool(host='mediaqueri.es', port=80): Max retries exceeded with url: / (Caused by <class 'httplib.BadStatusLine'>: '') <class 'requests.exceptions.ConnectionError'>`

This exception message confused me initially, because the BadStatusLine is something that would usually indicate a HTTP status being returned that is not considered valid, which would be unusual for such a basic request to a web server. After inspecting the actual TCP traffic with Wireshark, I've found that something else is happening. Here is the packet sequence for two requests being made five seconds apart, with the second one failing:
![screenshot from 2015-02-17 17 00 15](https://cloud.githubusercontent.com/assets/1697414/6239135/87e61a8c-b6c6-11e4-9b54-1776fbaab130.png)
The first request proceeds as you would expect, with a TCP connection being opened with a SYN, and then a GET and a HTTP response being received. The use of connection pooling means that the connection is kept open between the first and second and after five seconds the second GET is sent out. 

The issue is that the Flask web server also seems to close inactive TCP connections after five seconds, so right after that GET is sent out and probably while it's still in flight, a FIN is received from the server and duly ACKed, which closes that TCP connection without receiving the HTTP response. At that point, some RST packets arrive (indicating that a packet arrived on a socket after a FIN was sent?) and they seem to somehow end up causing a BadStatusLine HTTP error to be thrown.

I understand that is a fairly involved series of events, so thank you if you've made it this far. It might even be possible that this bug is at an even lower level than the requests library, somewhere in urllib3 perhaps. 

In any case though, having Session requests fail because they happen to be in sync with the rate at which TCP connections close is a pretty obscure problem for an end user to handle. At the least I think throwing an exception that lists BadStatusLine as the cause is not the correct behavior. The client side is ACKing the FIN it receives from the server, which means it should not expect to receive an HTTP response on that connection after that point.


So I think the fundamental problem here is that neither requests nor urllib3 have any way of knowing what's happening with the actually underlying TCP connection. urllib3 at its best wraps sockets to enable TLS but doesn't really directly manage sockets beyond that very thoroughly.

It isn't as if urllib3 or requests is acknowledging the FIN that it receives. That's generally how the socket behaves without us having to do that. I think a way around this is to enable TCP Keep-Alive but I'm not entirely convinced this is a solution that needs to live in requests proper.

I think @Lukasa is on vacation or just generally taking a break so I'll have to look into this more tonight. Thanks for the very very detailed issue @pddenhar!


BTW, @pddenhar I'm also in MSN. =D


Hello from the UW CS dept! I'm about to leave for the day but I'll certainly be interested to see where the issue goes. It's been a hard thing to track down because it's not really anyone's "fault"; TCP connections closing down are a totally normal part of the TCP state machine. I think somewhere things are just getting a little strange as the closed connection still results in a malformed HTTP response getting "handled". 

I think my fix for the time being will be not using sessions on high latency devices where the FIN and HTTP GET on pooled connections are likely to end up in flight at the same time. 


@pddenhar I wonder if the `TCPKeepAliveAdapter` in [this PR](https://github.com/sigmavirus24/requests-toolbelt/pull/59) helps any with your problem.


> The client side is ACKing the FIN it receives from the server, which means it should not expect to receive an HTTP response on that connection after that point.

Sadly that's not really how this works. The flow of events is this:
1. Second HTTP request is emitted by your machine and traverses the network.
2. The data is received, but not yet processed by Flask.
3. Flask closes the connection.
4. The TCP connection is torn down, either forcefully or abruptly.

We know that step 2 happened because the packet is FIN+ACK, not just FIN. This ACK applies to a previously received data-packet, and the only un-ACKed packet that can be is the one that transmitted the second request. This is a bit irrelevant though.

What matters is the sequence of socket calls. They go:
1. connect (optional, not done in this case)
2. send, send, send, send...
3. recv, recv, recv, recv...
4. GOTO 2

Here's the thing. When a TCP connection is closed by the remote end, the visible effect of this is that the recv call returns 'short' (with whatever data was received before the FIN), and subsequent recv calls on the same FD throw exceptions. In this case, as no data was received the recv call will return a length-zero read. httplib parses this as an early connection close and raises `BadStatusLine`. The RST packets are a red herring here: they don't cause this behaviour. The FIN by itself would do it.

Unfortunately, above the socket layer we cannot tell the difference between the server pulling down a connection before the application has processed the request and the server pulling it down after. `httplib` has no way of knowing that this closure means the request wasn't processed at all, versus the case where the request is processed and the response to it is to close the socket.

I just don't think there's anything we can do here.


I'm not sure there's much we can do to fix this, so I'm proposing we close it.


I am facing this issue without using sessions. I am calling `requests.{get,post,put,delete}` on a single `host:port` and after a few requests I get `BadStatusLine` exception. I am connecting over `https` with `verify=False` as the server is sending a self-signed cert. 

I believe that the underlying libraries still re-use TCP connections even when one isn't using a single session object for making requests. Otherwise, I wouldn't have faced this issue.

@narendraj9 that is only true for extremely old versions of Requests. You've provided no information to indicate this is still a problem. Even so, you should file a new issue if you can show that to be the case.

The version of python-requests that I am facing the issue is `2.11.1` on python `2.7.5`.

Right. So 2.11.1 was released in August of 2016 (per [PyPI](https://pypi.org/project/requests/#history)). That's over 2 years old at this point. That is unsupported from our point of view.

Okay. Thanks for the information. I tried looking at the code to see if it
is re-using TCP connections but couldn't find that it is. If it is possible
for you to tell me when it stopped re-using connections for a non-session
call on requests obejct, it will be really helpful.

Narendra Joshi

On Mon, 8 Oct 2018 17:02 Ian Stapleton Cordasco, <notifications@github.com>
wrote:

> Right. So 2.11.1 was released in August of 2016 (per PyPI
> <https://pypi.org/project/requests/#history>). That's over 2 years old at
> this point. That is unsupported from our point of view.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/requests/requests/issues/2448#issuecomment-427800208>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADbemgJMuYnv9qqZGCtHSuFNFJ_JIBAwks5uize0gaJpZM4DhzA->
> .
>