Sockets remain in CLOSE_WAIT

```
Python 2.7.4 (default, Sep 26 2013, 03:20:56)                                                                          
[GCC 4.7.3] on linux2                                                                                                  
Type "help", "copyright", "credits" or "license" for more information.
 >>> import requests 
 >>> requests.__version__                                                                                               
'2.1.0'
>>> response = requests.get('http://www.yahoo.de')
```

then check:

```
$ lsof -i
python  22881 xubuntu    6u  IPv4 545643      0t0  TCP 192.168.1.242:47049->ir2.fp.vip.ir2.yahoo.com:http (ESTABLISHED)
```

then : 

```
>>response.close()
```

and check again lsof -i

```
python  22881 xubuntu    6u  IPv4 545643      0t0  TCP 192.168.1.242:47049->ir2.fp.vip.ir2.yahoo.com:http (CLOSE_WAIT)
```

the sockets close only when I exit the interpreter !

Doing the same with urllib2:

```
>>res = urllib2.urlopen('http://yahoo.de')
```

lsof -i

```
python  22881 xubuntu    4u  IPv4 546690      0t0  TCP 192.168.1.242:48715->ir1.fp.vip.ir2.yahoo.com:http (ESTABLISHED)

>>res.close() 
```

now the socket is closed!

Can somebody explains this behavior?


Yes. =) Requests uses connection pooling to avoid the overhead of repeatedly creating and connecting sockets. This saves substantial system resources at the cost of keeping a few file descriptors open. Put another way: this is working exactly as designed. =)


Thanks for the response.
But what if I am doing a lot of parallels requests?
And if that is the case what is the purpose of the close method?


We have a maximum number connections in the pool: in this case, 10 per scheme. The close method returns a socket to the pool. =)


@Lukasa try that:

```
responses = [ requests.get('http://www.greven.de') for _ in xrange(60) ]
```

do an 

```
lsof -i|grep 212.29|wc -l
```

you should get 60

then "release" the connections to the pool:

```
[r.close() for r in responses]
```

do the same 

```
lsof -i|grep 212.29|wc -l
```

There are still 60 sockets in CLOSE_WAIT state.

I think that something is not as expected.

Can you verify that it is ok?
Thanks


Hmm, @shazow, thoughts?


The connections seems to be reused when you do it like that:

```
for _ in xrange(60):
    r = requests.get('http://www.greven.de')
    r.close()
```


Hmm. I wonder if, when you consider this and #1967, we have a bug with releasing connections back to the pool.


Also this look strange and maybe related:

```
>> r = requests.get('http://www.greven.de')
>> r.raw.closed
```

   True

```
python  22881 xubuntu    6u  IPv4 562317      0t0  TCP 192.168.1.242:45158->195.14.212.29:http (ESTABLISHED)
```

althought the socked is ESTABLISHED .

and then:

```
>>> r.close()
>>> r.raw.closed
True 
```


Hm. One thing worth noting that by default, a pool will supply additional connections despite being empty. So, if you have a pool of size 10, and you ask for 50 connections concurrently, then it will create 50 new connections but still only return 10 of them to the pool (the rest will be closed and discarded). 

I'm not sure what Requests' default pool size is. I suspect it might be 1. If you're expecting to run N concurrent requests, I suggest creating a PoolManager with size N. You can also bottleneck on the pool size by using `block=True` on the pool. (These are urllib3 things, you'll need to translate them for Requests-land).


This is what is supposed to do.
If that works as expected then when we do N  parallel requests we should have N sockets open.
When the requests have fiinished and read the content the sockets should close or returned to the pool.

So if the pool size is 10 (as Lukasa wrote)  we have the following:
- if N <= 10 all the connections should be returned to the pool
  - if N > 10 then N - 10 connections should be closed and 10 will be returned to the pool.

So after that we should at most have 10 sockets in wait_close state.

Am I correct for the above?


So the for loop is allowing previous responses to be garbage collected. The list comprehension does not allow for that. I suspect the problem is strongly related to that


Aha, yes, the garbage collector is at fault here:

``` python
>>> for _ in xrange(60):
...     r = requests.get('http://www.greven.de')
...
>>>
```

``` bash
[vagrant@localhost sipp]$ lsof -i|grep 212.29|wc -l
1
```

The one socket being held here is the one I have a reference to, via the variable `r`. We can see this with the list comprehension:

``` python
>>> responses = [ requests.get('http://www.greven.de') for _ in xrange(60) ]
```

``` bash
[vagrant@localhost sipp]$ lsof -i|grep 212.29|wc -l
61
```

Now I've got the 60 requests from the LC, plus the one from before. If I empty the list of stored responses, letting them get GC'd:

``` python
>>> responses = []
```

``` bash
[vagrant@localhost sipp]$ lsof -i|grep 212.29|wc -l
1
```

This is obviously true, we don't close the sockets until the GC gets them. =) Unless we can see something else, there is no bug here.


The for loop is first doing the request then closes the connection and then does the other request.

The list comprehension does the request then does the other requests ...

When all the requests have been made it closes them one by one

So it is a different thing.
The list comprehension should create a connection foreach request.
So it should create N connections.
Then it closes/releases these connections.

The for loop creates a connections and closes/releseases it.
The next iteration should resuse the same socket since it is released back to the pool.

So here I am expecting to create only one socket in the fist iteration and resuse it in the N-1 others


@gosom The key is that you're storing the responses in the list comprehension. These responses have references to the socket, which prevents it getting garbage collected. We don't aggressively close sockets because there's no requirement to do that, but we do aggressively release them to the pool (where appropriate).


@Lukasa 

you are right it works correct this way.
But when you explicitly close the connections there should be only MAX_CONNECTION_POOL_SIZE open correct?


Whether they are actually 'open' is not guaranteed: some servers close the connection, so we just hold on to the socket object. But yes, that should be the maximum we hold on to _once garbage collection has occurred_. 


```
>>> import requests        
>>> responses = [requests.get('http://www.greven.de') for _ in xrange(30)] 

$lsof -i|grep python|grep -v grep|wc -l 
 30

>>> [r.close() for r in responses] 
>>> responses = []

$lsof -i|grep python|grep -v grep|wc -l 
 30
```

Shouldn't the garbage collection cleanup?
Then only 10 sockets should be in close_wait state


I don't see that behaviour on my machine, I have all my sockets cleaned up.


True, I tried in another machine (physical machine, I was testing in a vm before).
and it is ok.
It cleanups all the sockets.

But still the close() method does not seem to do anything

An more if all the sockets are cleaned where is the reusage?

One more question the 
 r.raw.closed property what is the purpose?


1. The `.close()` method is documented to return the connection back to the pool, and is also mentioned not to be expected to be called.
2. Connections are re-used if the server doesn't close them, but most do. We do, however, re-use the socket objects, thereby saving a few syscalls.
3. You should try not to think about it. =) Generally, it reflects whether the backing connection is closed or not.

The advice when using Requests is not to worry about this stuff: we handle it for you.