read timeouts and iter_lines

I want to be able to consume an HTTP stream of indefinite length, but I also want a way to cleanly exit the read loop that doesn't involve killing the thread. Here's what I've got:

``` python
import threading
import socket
import requests

resp = requests.get('http://some-url/', stream=True, timeout=5)

quit = False
quit_lock = threading.Lock()
it = resp.iter_lines(chunk_size=1)
while True:
    quit_lock.acquire()
    if quit:
        quit_lock.release()
        break
    quit_lock.release()
    try:
        line = next(it)
    except StopIteration:
        break
    except socket.timeout:
        print('read timed out')
        it = resp.iter_lines(chunk_size=1)
        continue
    print(line)
```

The idea here is that a separate thread can set `quit = True` and this read loop will exit within some seconds. The approach works fine on Python 2 but fails on Python 3. On Python 3, once a read has timed out, a subsequent read will fail with the exception:

```
OSError: cannot read from timed out object
```

Is this a bug in the requests library? Or is it simply a limitation of Python 3? Or is there some other workaround?


This seems to be a limitation of Python 3, introduced in [Issue #7322](https://bugs.python.org/issue7322). The core problem here is that the `read` and `readline` methods of Python socket objects use internal buffers, in order to satisfy the read contract that says that reads should only return less than `amt` if the data stream is closed (see also shazow/urllib3#709). These internal buffers can get screwed up if the timeout is thrown during a later `recv` call.

Now, we don't actually _use_ those methods directly, but `http.client` does, and also provides some internal buffers of its own. Those buffers are also at risk of being screwed up.

Finally, `iter_lines` _also_ uses buffers internally, which are _also_ lost when this exception fires. This doesn't matter for you in the "everything is working fine" case, because you only expect the exception to fire once a newline has been received (so the internal buffer is empty), but if anything went wrong it would fire _mid-line_ and you'd lose the whole line.

There are two ways that leap out at me to make this work, only the first of which uses the majority of your code. The first is a bit gross, because it involves reaching _way_ into requests and urllib3, but it should be fairly stable (the objects you're touching aren't moving around a lot). This involves using `select` and timeouts instead. You could write this:

``` python
import threading
import select
import socket
import requests

resp = requests.get('http://some-url/', stream=True, timeout=60)

quit = False
quit_lock = threading.Lock()
it = resp.iter_lines(chunk_size=1)
while True:
    quit_lock.acquire()
    if quit:
        quit_lock.release()
        break
    quit_lock.release()

    try:
        if not select.select([resp.raw._fp.fp], [], [], 5)[0]:
            continue
    except TypeError:
        # Raised if the connection is completed
        break

    try:
        line = next(it)
    except StopIteration:
        break

    print(line)
```

The second approach is to use a proper concurrency framework for this job. Obviously, that's a bit out of scope for this repo so I won't go into it more, though I'm happy to point out some frameworks that would help if that would be useful (e.g. asyncio, Twisted).


Ah yes, I wondered if I could use `select` somehow. Thanks for the tip. Would this also work for HTTPS connections or would there be a different fd to check?


It should be the same fd.


So this approach almost works. On the first iteration, the `select` is not considering the fd readable even though there is data to read. I think what is happening is any initial data is being buffered inside the response object (or some other layer) and so has already been read by the time we poll the socket. Once new data arrives on the socket, it becomes readable, and the client reads the initial data.

Is there a way to test if there is buffered data to try to read?


> Is there a way to test if there is buffered data to try to read?

Oof, harder. I don't immediately see anything here. I can't even see where the buffer is: it's not in `socket` or `io` AFAICT, so I'd have to guess it was in `http.client`, but I can't see it in there either. Someone, somewhere, has buffered your data above the socket level, but I don't know who did it.