contextlib.closing(response) is not enough when using sessions and stream = True

The requests documentation suggests using the following idiom to partially read response bodies (e.g. when stream = True):

``` python
from contextlib import closing

with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
    # Do things with the response here.
```

I thought this would be applicable to sessions, but it seems it's not enough.
The following program reliably raises an exception (BadStatusLine) even while it uses contextlib.closing, and the only way to stop that from happening is to consume the whole response content:

``` python
import contextlib
import random
import requests

urls = [
    'http://reddit.com',
    'http://news.ycombinator.com',
    'http://github.com',
    'http://bitbucket.org',
]

try:
    session = requests.Session()

    while True:
        try:
            with contextlib.closing(session.get(random.choice(urls), stream = True)) as r:
                # uncomment the line below and the errors dissapear:
                # r.content
                pass

        except Exception as ex:
            print(str(ex))

except KeyboardInterrupt:
    pass
```

Is this the expected behavior or actually a bug? (to be fair the documentation doesn't actually say that the idiom is also applicable to requests coming from sessions).


I think this is expected, but it could arguably be worked around.

The problem is that urllib3 doesn't know whether we consumed the response, so it doesn't know that the socket has unread data in it. We should be able to support that case, I think.


Wait. I have a question.

In your first example, you have a comment "Do things with the response here". If you put that in your while loop with a "pass" statement, do you see the same behaviour? If so, this issue seems a bit misleading as it is currently presented.


Hi.

> In your first example, you have a comment "Do things with the response here". If you put that in your while loop with a "pass" statement, do you see the same behaviour?

The errors disappear if:

``` python
# I just use requests.get instead of sessions:
with contextlib.closing(requests.get(random.choice(urls), stream = True)) as r:
    # no need to consume the whole content, no errors:
    pass
```

``` python
# I use sessions but I consume the whole response body:
with contextlib.closing(session.get(random.choice(urls), stream = True)) as r:
    r.content
```

In other words, the problem is that `closing(requests.get(...), stream = True)` is always safe regardless of the content, but `closing(session.get(...), stream = True)` is not, you must consume the response content.


Just so that everything is as clear as possible:

The [documentation](http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow) states that `closing(requests.get(..., stream = True))` is safe. This works well, but it may lead an user to assume that `closing(session.get(..., stream = True))` is also safe.

This is not true as can be seen by running the program in my first post. It eventually raises an exception with a BadStatusLine. When using a session, you have to consume the content, calling `response.close()` is not enough.

Unfortunately, this may lead to hard to reproduce bugs in user's code. You need a program like the one in the first post (or in my use case a multithreaded link crawler) to reproduce it consistently. 

I can see two solutions:
- Be explicit in the documentation and state that `closing(session.get(...), stream = True)` is unsafe.
- Always consume the content when `response.close()` is called.


I think blindly consuming the content when `response.close()` is called is a bad idea, because it could potentially take an unbounded amount of time to complete. If `response.close()` is called without having drained the response, we should close the underlying connection instead.


Yeah, that seems a way better solution.


Fixed by #2610.