Closing a connection from another thread

Hello, 

Is there any way to close an ongoing connection, for instance I want application to quit while a download of a big file is going on, or an upload. 

I managed to close it this way, but hoping to find a better one

```
def close():
    time.sleep(5)
    r.raw._fp.close()

t = threading.Thread(target=close).start()
print "getting"
s = requests.Session()
r = s.get("http://download.thinkbroadband.com/1GB.zip", stream = True)
for line in r.iter_content(1024):
    log.debug("got it: %s", len(line))
print "done"
```


For everyone else interested, this stems from [this discussion](http://stackoverflow.com/questions/16390243/closing-python-requests-connection-from-another-thread/16400574?noredirect=1#comment23529567_16400574). I'm personally :-1: on the idea, but I'd like to hear @kennethreitz and @Lukasa chime in.


I think the right way to do this is a combination of message passing and streaming downloads. The main thread should be able to send a message to the download thread that can say 'when you can, stop downloading'. Alongside that, the downloading thread should use `iter_content` in a loop to download the data. This loop should abort when the 'stop downloading' message is received, and then call `Response.close()`. **I haven't tried this**, but that ought to work.


Yeah, that works fine. Here is an example script that will do what you want.

Please please please don't just use this as-is. It's suitable for running on your own box if you want to be sure that it works, but it's not really thread-safe in any meaningful sense. The GIL mostly protects us here, but still, _caveat emptor_. Even though you aren't paying.

``` python
#!/usr/bin/env python
# A test script to verify that you can abort streaming downloads of large
# files.
import threading
import time
import requests

stop_download = False

def download(url):
    r = requests.get(url, stream=True)
    data = ''
    content_gen = r.iter_content()

    while (stop_download == False):
        try:
            data =  r.iter_content(1024)
        except StopIteration:
            break

    if (stop_download == True):
        print 'Killed from other thread!'
        r.close()

if __name__ == '__main__':
    t = threading.Thread(target=download, args=('http://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/9.1/FreeBSD-9.1-RELEASE-amd64-dvd1.iso',)).start()
    time.sleep(5)
    stop_download = True
    time.sleep(5) # Just so you believe the above line caused the stoppage.
```


@Lukasa why you no protect the critical variable?


Just a demo script, doesn't need to be super-protected. Also I'm pretty sure that the GIL covers any kind of problems in this case. Anything more complicated and it wouldn't, but for simple comparison and one-off assignment, probably fine. =)


Of course, what I meant to say is: "I, too, like to live dangerously."


I was mostly kidding hence the meme-speak


And hence my meme-speak reply. =D


@Lukasa You are still reading in chunks and you have no control while that is going on, and timing of your socket close request depends on state of communication, not good. Imagine a long polling, push message use case, or connection about to timeout in 10 seconds, still you cannot drop it.

For sockets, simply calling close from another thread is a common practice, method should be thread safe of course.


If you really need fine-grained control you can use `1` as the chunk size, which will enable you to stop reading after any given byte: it's just super slow. =D

I can guarantee to you that calling close() from another thread can only happen in one of two ways: either a message is passed into the thread that is handled on the next go around the event loop (which is basically what I just wrote), or the socket is protected by a mutex and the downloading thread checks for whether the socket is open each time it attempts to read from it.

You _have_ to do this using either shared state or message-passing.

(Note: I don't know what you mean by "timing of your socket close request depends on state of communication". Can you elaborate?)


I mean your loop in thread only can check whenever r.iter_content returns, imagine you are waiting a stream, just like twitter api, do you really want to wait until soneone tweets ? 

I can go async using twisted etc, but then again I want to use blocking sockets to lessen complexity, I am running sequential RPC, so I can run reactor loop and wait it to complete. But this seems overkill, 
"Things shouldn’t be this way. Not in Python."


Yeah, I think this use-case is a bit specialised for Requests. Your requirement is that you want to use blocking IO combined with threading, but to be able to kill the network request at any time from outside of the controlling thread, _and_ to have that thread cancel its blocking IO without delay.

To do that we'd have to expose an API to close the underlying socket layer, which is a thing we work very hard to abstract away. (I also don't know how it would interact with the connection pooling used by urllib3.) I think Requests will probably not support this fairly unusual use case, especially while we're in [feature freeze](https://github.com/kennethreitz/requests/issues/1165).


Long-polling is a pretty common use-case in my work. I have multiple long-polls running, and I want to cancel all others when one (or N) return. Currently, I've manually extended the connection pool to keep track of its released connections, so I can close them when requested.

Above, the argument is that you don't want to expose low-level functionality; however, "cancelling long-polls" seems pretty high-level to me. Is cancelling requests via force-closing of sockets a feature that requests want to implement?

Pinging this issue for consideration into inclusion into [`requests` v3](https://3.python-requests.org/): let me know if this should be a new issue.

Edit: I realise that the connection pool is a `urllib3` feature, so I should make a feature-request there