ResourceWarning still triggered when warnings enabled

I can reproducibly get ResourceWarnings when running incomplete requests with [iter_lines](http://docs.python-requests.org/en/latest/api/#requests.Response.iter_lines).

this is my test code:

``` python
#!/usr/bin/python3

from contextlib import closing
import re
import time
import warnings

import requests

print("requests %s" % requests.__version__)

max_bytes = 640*1024
def test_get_title_close(url):
    response = requests.get(url, stream=True)

    try:
        content = ''
        for line in response.iter_lines(decode_unicode=True):
            content += line
            if '</title>' in content or len(content) > max_bytes:
                break
    except (UnicodeDecodeError, TypeError):
          # Fail silently when data can't be decoded
          return
    finally:
        # need to close the connexion because we have not read all the data
        response.connection.close()
    response.connection.close()

    print('title found: %s' % re.sub(r'^.*<title>', '', re.sub(r'</title>.*$', '', content)))

def test_get_title_closing(url):
    with closing(requests.get(url, stream=True)) as response:
        try:
            content = ''
            for line in response.iter_lines(decode_unicode=True):
                content += line
                if '</title>' in content or len(content) > max_bytes:
                    break
        except (UnicodeDecodeError, TypeError):
              # Fail silently when data can't be decoded
              return
        finally:
            # need to close the connexion because we have not read all the data
            response.close()
            # also trying the undocumented ones
            response.connection.close()

        print('title found: %s' % re.sub(r'^.*<title>', '', re.sub(r'</title>.*$', '', content)))

warnings.filterwarnings('default')
test_get_title_closing('http://example.com/') # this should not warn, but does
test_get_title_close('http://example.com/') # this should not warn, but does also
```

it's taken from this pull request: https://github.com/sopel-irc/sopel/pull/988 - which tries to switch an irc bot from using a homegrown http library to the more standard "requests". the original code does not yield such warnings even though it also does HTTPS connections.

here is the output:

```
$ python3 test_requests_fail.py
requests 2.9.1
title found: Example Domain
../test_requests_fail.py:53: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET6, type=SocketType.SOCK_STREAM, proto=6, laddr=('2607:f2c0:f00f:8f00:ea9a:8fff:fe6e:f60', 39628, 0, 0), raddr=('2606:2800:220:1:248:1893:25c8:1946', 80, 0, 0)>
  test_get_title_closing('http://example.com/') # this should not warn, but does
title found: Example Domain
../test_requests_fail.py:54: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET6, type=SocketType.SOCK_STREAM, proto=6, laddr=('2607:f2c0:f00f:8f00:ea9a:8fff:fe6e:f60', 39630, 0, 0), raddr=('2606:2800:220:1:248:1893:25c8:1946', 80, 0, 0)>
  test_get_title_close('http://example.com/') # this should not warn, but does also
```

notice how i tried my best to close the resources, using various methods, none of which work. there's something definitely fishy about the way this works.

this is on Debian Jessie 8.2, with the latest requests installed from pip (2.9.1), or jessie-backports (2.8.1-1~bpo8+1) and Python 3.4.2.

oddly enough, older requests versions do _not_ have that problem: 2.4.3 from plain old jessie is actually fine!

so it sure looks like something broke since 2.4.3 in the fix proposed in #1882 and #2004.


Requests works very hard to prevent you closing a response. This is because it gets in the way of our connection pooling. If you _really must_ close the connection, you need to close the underlying urllib3 connection: use `response.raw.close()`.

The reason the resource warnings are back is because we're much better about not returning partially complete connections to the connection pool. The lack of warnings in 2.5.4 was actually symptomatic of a problem: weirdly, the warnings are an indication that everything is working as intended.


`response.raw.close()` does not work either here, i still get the warning.

i don't get it: ResourceWarnings seem like a problem, not an indication that all is well. in this case, i deliberately want to close the connection before it finishes, because i do not want to download all of the request! how can i do that? and why do we leave sockets lying around like this? if it wasn't for the (not always deterministic) garbage collector, this would be a leaking resources like a sieve in a loop...

in other words: how is having a warning a sign that all is well? or how can it be a problem that there are no warnings?

and why are there functions in the API that basically do absolutely nothing? both the `close()` API and the `with closing(...)` suggestions are basically useless right now.

so i think this is at least a documentation issue...


Do you have a minimal reproduction case? Closing the fp really should work. Can you demonstrate it in a small chunk of code I can investigate please?


i have a reproducible test case above. i tried adding `response.raw.close()` after `response.connection.close()`, without luck. but since you asked:

``` python
#!/usr/bin/python3

from contextlib import closing
import re
import time
import warnings

import requests

print("requests %s" % requests.__version__)

max_bytes = 640*1024
def test_get_title_close(url):
    response = requests.get(url, stream=True)

    try:
        content = ''
        for line in response.iter_lines(decode_unicode=True):
            content += line
            if '</title>' in content or len(content) > max_bytes:
                break
    except (UnicodeDecodeError, TypeError):
          # Fail silently when data can't be decoded
          return
    finally:
        # need to close the connexion because we have not read all the data
        response.connection.close()
        # also trying the undocumented ones
        response.close()
        response.raw.close()

    print('title found: %s' % re.sub(r'^.*<title>', '', re.sub(r'</title>.*$', '', content)))

def test_get_title_closing(url):
    with closing(requests.get(url, stream=True)) as response:
        try:
            content = ''
            for line in response.iter_lines(decode_unicode=True):
                content += line
                if '</title>' in content or len(content) > max_bytes:
                    break
        except (UnicodeDecodeError, TypeError):
              # Fail silently when data can't be decoded
              return
        finally:
            # need to close the connexion because we have not read all the data
            response.connection.close()
            response.close()
            response.raw.close()

        print('title found: %s' % re.sub(r'^.*<title>', '', re.sub(r'</title>.*$', '', content)))

warnings.filterwarnings('default')
test_get_title_closing('http://example.com/') # this should not warn, but does
test_get_title_close('http://example.com/') # this should not warn, but does also
```


So the failure of `raw.close()` is because the http response object is claiming it _is_ closed. This requires a bit more digging on my end to work out why the object is lying.

I'll take a look this weekend and see what I can see.


Ok, so the problem here is that both the underlying HTTP connection _and_ the HTTP response have references to the socket, so the socket doesn't actually close until both references close it. Right now, urllib3's response `close` method doesn't do that, it only closes the response, and as a result the pooled connection remains alive until it passes out of scope.

I consider that a urllib3 bug, so I've opened shazow/urllib3#779. In the meantime, you'll want to call both `response.raw.close()` and `response.raw._connection.close()`. Those two calls alone should solve your problem.

This is also why the documented API functions aren't working as you expect: they also call `response.raw.close()`, and are not expecting that that would fail to clean up the connection.

This should be resolved in a future release when a fix for the urllib3 issue is merged. Sorry about the bug!


On 2016-01-09 11:56:02, Cory Benfield wrote:

> This should be resolved in a future release when a fix for the urllib3 issue is merged. Sorry about the bug!

should this bug be kept open then?

## thanks for the followup

The United States is a nation of laws:
badly written and randomly enforced.
                        - Frank Zappa


@anarcat why should we track an issue in a dependency?


@sigmavirus24 is right. There is an issue open tracking this problem. We have zero code change required to use that fix, all we have to do is update our dependency (which we'll do in the next 2.X release just as part of the release process). The correct issue to track is the one on urllib3.


okay, then i guess i'l watch that other issue.