Setting stream=False does not immediately result in a closed connection

When making a request to a server, if the server response contains a `Connection: close` header the TCP connection should be closed when the request ends, thus resulting in a `tcp:fin` being sent from the client to the server. This behavior is not happening. I'll note that the connection is not available for additional HTTP requests because it has been closed on the server side.

After looking through the past issues I discovered issue #520 which lead me to believe this behavior would occur when adding `prefetch=True` for requests version < 1.0 and now `stream=False` for version >= 1.0. However, it seems that neither of these methods actually result in the closing of the TCP connection on the client side.

Also, one thing the #520 issue discussion neglected to consider (maybe the code does) is that many load balancers explicitly close the TCP connection and send the `connection: close` in spite of what headers the client sends. This is the issue I'm running into as the load balancer (haproxy) will not allow me to make additional requests until my previous connection is properly closed.

I realize this probably extends into urllib3, however since I'm using requests directly I'll initially file the issue here.

Is there something I am missing? Assuming not, and for the time being, how can I forcefully close the TCP connection from a response object?


Thanks for the report; we're definitely interested in investigating and fixing problems like this.

This is the test case from #520: [https://gist.github.com/4332611]

I do not observe a problem when running it under master. Could you see if you experience a problem? And if not, could you try and modify it (or write a new test using similar techniques) that reproduces the bug?


Took me awhile to find a server with a load balancer that behaves similarly to my local setup. Replace the `requests.head` line with `requests.get("http://appfolio.com", prefetch=True)` to see connections in the CLOSE_WAIT state for version `0.14.2`.

I'm trying to find an example that works with `1.0.3`. I actually discovered that when testing locally, `requests.get` appears to block in `1.0.3` until the connection closes, but I've yet to find a globally-reachable server that shows the behavior.


Hmm, doesn't reproduce for me against `0.14.2`: [https://gist.github.com/4479f51af35453402a9c]


Oh, I botched the URL: it reproduces for me with `http://appfolio.com`. I'll take a look later.


Personally I'm not really too concerned with the `0.14.2` behavior as I intend on using `1.0.3`. Unfortunately for `1.0.3` I have yet to be able to reproduce the error other than when haproxy is used in my virtualbox VM in-front of a paste uswsgi server. When I tried installing haproxy locally with a similar configuration I was not able to reproduce. This behavior leads me to believe it may be an issue with virtualbox. I'll continue to look around.


Yeah, that makes sense. I think the connection management architecture (inside urllib3) hasn't changed much between those versions, so if I can diagnose the issue on 0.14.2 it may illuminate a problem on 1.x.


After some attempts, I was finally able to reproduce the blocking behavior with requests `1.0.3`. I wrote a simple HTTP server that behaves similarly to the setup that I'm testing against. Primarily it responds with `HTTP/1.0` indicating that the connection cannot be reused, and more over the server doesn't process additional connections until all previous connections are terminated properly. That is the client closes their end of the connection ([server source](https://gist.github.com/4333673)).

After running the server, the client code that demonstrates the blocking is:

```
import requests

for x in range(5):
    print 'Making request %d' % x
    tmp = requests.get("http://127.0.0.1:8000/", stream=False)
    print 'Done making request %d' % x
```

This example demonstrates the importance of not waiting until garbage collection to close connections. With `HTTP/1.0` or `HTTP/1.1` + `Connection: close` headers, the connections should be immediately closed when EOF is received.


I was tracking through how a request is sent earlier. I think we need to call `release_conn` on `Response.raw` when we know we can close the connection


@bboe your server and your test works fine.


@sigmavirus24 Do you mean you are unable to reproduce the problem?


@bboe exact opposite. I was just going to edit that since I realized it was a bit vague. Even calling release_conn doesn't work.


@sigmavirus24 Oh okay good. I was hoping I wouldn't have to find the set of platform specific settings to reproduce :) Does this mean you are going to look into this a bit then?


@bboe looking into it right now.


This looks like it might be urllib3. If you modify your loop to do:

``` python

for x in range(5):
    print "Request %d started" % x
    t = requests.get('http://localhost:8000', stream=False)
    t.raw._fp.close()
    print "Request %d finished" % x
```

Then it won't block. We can probably fix it ourselves, but that becomes ugly.


@sigmavirus24 Thanks for the quick fix. I'll use that for now.


@bboe glad to help. I'm trying to figure out what the best approach might be with urllib3 for now and how to test it.


We could add on to this: https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L172 that line to close it.

I'm in the process of adding a close method to a urllib3 HTTPResponse to make that cleaner and nicer.


Pending shazow/urllib3#132 we can add that line I mentioned after calling `release_conn` and `r.raw.close()`. This should solve the issue completely.

The other thing is that `Response.close()` doesn't return anything since `release_conn` doesn't return anything. If we want, we can also stick `self.raw.close()` in there. Any objections @kennethreitz ?


@sigmavirus24 I'm thinking the issue should be fixed in urllib3. See https://github.com/kennethreitz/requests/blob/master/requests/models.py#L466

It seems that the entire content is received by requests, thus urllib3 should be aware of whether or not the connection is reusable as this point. If the connection is not re-usable, the socket should be closed.


See the pull request I referenced. @shazow and I are discussing whether this 
behaviour should implicitly be part of the `release_conn` method.


PR shazow/urllib3#133 fixes this issue, and as it turns out will also fix issue #771 for all python releases prior to December 15, 2012 (that includes current releases).


@bboe your PR on urllib3 was accepted and we just updated urllib3 in requests, care to close this? 


Awesome, thanks!