chunked encoding bug when release connection

https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L404
I encounter a bug that will be fixed when remove this line.
Since the preload_content=False in HTTPResponse.from_httplib, we cannot _put_conn back to pool (where it's closed)

``` python
                low_conn = conn._get_conn(timeout=timeout)
                try:
                    low_conn.putrequest(request.method,
                                        url,
                                        skip_accept_encoding=True)
                    .....
                    r = low_conn.getresponse()
                    resp = HTTPResponse.from_httplib(
                        r,
                        pool=conn,
                        connection=low_conn,
                        preload_content=False,
                        decode_content=False
                    )
                except:
                    # If we hit any problems here, clean up the connection.
                    # Then, reraise so that we can handle the actual exception.
                    low_conn.close()
                    raise
                else:
                    # All is well, return the connection to the pool.
                    conn._put_conn(low_conn)
```

the code trigger this bug (just illustrate):

```
monkey.patch_all()

def data_stream(n):
    for i in range(n):
        x = str(i)
        gevent.sleep(0)
        yield x

gevent.spawn(requests.post('xxx', data=data_stream(100)))
gevent.spawn(requests.post('xxx', data=data_stream(100)))
gevent.joinall()
```

the first post gets normal response, while the second get empty response (conn closed before read content)
but result is ok if we don't use gevent.spawn


Why aren't you using grequests?


I look grequests code and find that the code is indeed what I do now use gevent.
Also, I need more control with the concurrency, the example code is just a simple demo.
I think this is a bug of requests.


Can you provide something that actually allows us to reproduce this. We aren't going to just remove code because you asked us to. I modified your example a bit so that I could test it, and it in fact produces no problems for me.

``` python
import requests
import gevent
import gevent.monkey

gevent.monkey.patch_all()


def data_stream(n):
    for i in range(n):
        x = str(i)
        gevent.sleep(0)
        yield x

url = 'https://httpbin.org/post'

greenlets = [gevent.spawn(requests.post, url, data=data_stream(100)),
             gevent.spawn(requests.post, url, data=data_stream(100))]
r = gevent.joinall(greenlets)
```


I don't means remove the code is the correct way, just say where the bug is.

I change your code a little and produce the bug.
Note the connection pool size=2

``` python
import sys
import requests
import gevent
import gevent.monkey

gevent.monkey.patch_all()


def data_stream(n, mark='.'):
    for i in range(n):
        x = str(i)*100
        #gevent.sleep(0)
        if i%(n/10)==0:
            sys.stdout.write(mark)
            sys.stdout.flush()
        yield x

sleep = 0

sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=2, pool_maxsize=2)
sess.mount('http://', adapter)

def g(n, mark='.'):
    global sleep
    sleep += 0.2
    #gevent.sleep(sleep)

    url = 'http://httpbin.org/post'
    r = sess.post(url, data=data_stream(n, mark))
    #gevent.sleep(4)
    print '\n>>>>>>', mark, len(r.content)

n = 100
greenlets = [
                 gevent.spawn(g, n, '.'),
                 gevent.spawn(g, n, 'x'),
                 gevent.spawn(g, n, 'o'),
                 #gevent.spawn(g, n, '-'),
                 #gevent.spawn(g, n, '='),
                 #gevent.spawn(g, n, '*'),
            ]
r = gevent.joinall(greenlets)
print 'done'
```

output:

```
xxxxxxxxoooooooo........xx
>>>>>> x 466
..
>>>>>> . 0
oo
>>>>>> o 0
done
```

the 2nd and 3rd response is lost


I haven't run the code yet, but this can (probably) be solved in a few ways:
1. Use grequests because it looks to do The Right Thing
2. Figure out a way to make gevent monkey patch the `queue` module with it's [own](http://www.gevent.org/gevent.queue.html) (although I have yet to test that this will work)
3. (And this will be the easiest) Use a separate session for each "thread". The problem is that the `LifoQueue` from the standard library is **threadsafe** but that does not imply gevent-safe. This will ensure separate queues for your program.

We _could_ add something like

``````
```python
LOCK = threading.Lock()
# ...
     with LOCK:
         conn._put_conn(low_conn)
```
``````

But we probably won't. The reason this might work is because when using `gevent.monkey.patch_all()` you're patching the threading library and so gevent will ensure that this will not hang. I don't have the time to test all of this, but connection pooling is almost certainly the problem here. @shazow do you have any thoughts on this matter?


So actually, the catch here is import/monkey patch order.

I'm willing to be the following will fix this:

``` python
import gevent
import gevent.monkey

gevent.monkey.patch_all()

import sys
import requests
```

@ilovenwd if you don't `patch_all` before import requests, it probably won't replace any usage of `RLock` used in the `LifoQueue` which is imported from `threading`.


I change the code, but still the same issue.

``` python
import gevent
import gevent.monkey
gevent.monkey.patch_all()
import requests
```

the output is exactly the same.

I trace the code line by line, the real issue is that:
because preload_content=False, so the low_conn is put back to pool BEFORE content is consumed,
so other greenlet/thread may re-use this low_conn (and reset it maybe).
after that, the original client of the low_conn read nothing.


you can compare the chunked code with non-chunked code https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L566
by default, it didn't release the conn (by default), so maybe we should not release low_conn in chunked case, but I'm not sure, since I'm not familiar with requests code base.


Right so the solution is to not use the same session. This is the same recommendation we make in the case of threading. We never want to preload the content into memory immediately. If we do it unconditionally that will have some very real memory problems for users.


Let me know if I can still be of help, sounds like you figured it out.


Maybe I should use the diff session per greenlet solution. Thanks for the explanation.
However, I wish requests doc could have sth regards this case with gevents.