Limit overall execution time

This seems known but timeout in the .get() function only works if the server doesn't respond. Is there a way to have this also apply to the entire request? (grab a MP3 stream, file too large, site is responding but is too slow?) I'm utilizing requests in an application that grabs only 1MB of each page via .iter_content() for detecting MIME but have some issues where sites are so entirely slow that grabbing the 1MB (or whatever the page is until it completes) can easily take >60s.


Thanks for raising this issue!

The answer is 'yes', sort of. When not streaming the download we apply the supplied `timeout` value to both the connection attempt and the read. However, currently, when streaming the download we only apply the timeout to the connection attempt.

There's an ongoing discussion about how best to handle read timeouts, but right now we don't do anything useful for you there I'm afraid. Your only real workaround is to read smaller chunks. =(


Gentlefolk,

While I am a brand new user of Requests, I do run into the problem where a long running stream does not timeout, it just hangs. In my case, I'm connecting to the Twitter streaming endpoint using `session.iter_lines()`. The loop is trivial:

```
try:

    stream = twitter_sample(args)

    for tweet_bytes in stream.iter_lines(chunk_size=2048):  # Read a basic tweet at a time.
        q.put(tweet_bytes.decode())  # Decode the tweet and put it on the queue.

except (RequestException, UnicodeError) as e:
    warning('Error: {}'.format(e))
```

`twitter_sample()` looks like:

```
def twitter_sample(args: argparse.Namespace) -> requests.Response:

    twitter = OAuth1Session(args.consumer_key, args.consumer_secret,
                            args.token, args.token_secret)
    endpoint = 'https://stream.twitter.com/1.1/statuses/sample.json'
    headers = {'Accept': 'application/json; charset=utf-8'}
    timeout = 60.0
    # timeout = Timeout(connect=15.0, read=60.0)  # urllib3 Timeout()

    stream = twitter.get(endpoint, stream=True, headers=headers, timeout=timeout)

    return stream
```

As this is a long running stream from Twitter that occasionally just stops, I need to be able to recognize when it does. I can't just "read less". Being inspired by the Urllib3 documentation that takes either a `float` or a `Timeout()` for the `timeout` property, I have tried passing in a `Timeout()` class but it is caught in validation. If you let it through or just sanity checked the parameters, I suspect I would get everything I need. (This is the one feature I am getting out of a custom Twitter library and not Requests. I would prefer to use the general purpose and more performant stack that is Requests.)

Thank you all for the hard work on this package.

Anon,
Andrew


Hi there, and thanks for using requests!

As I mentioned above, there's an ongoing discussion on how best to handle the multiple timeouts. As a temporary workaround before we come up with something better, you can monkeypatch the `TimeoutSauce` internal class.

**Please Note: The following involves monkeypatching a class that is an implementation detail of requests. Be aware that we may change this under your feet without mentioning it in the changelog.**

``` python
import requests
from requests.adapters import TimeoutSauce

class MyTimeout(TimeoutSauce):
    def __init__(self, *args, **kwargs):
        connect = kwargs.get('connect', 5)
        read = kwargs.get('read', connect)
        super(MyTimeout, self).__init__(connect=connect, read=read)

requests.adapters.TimeoutSauce = MyTimeout
```

This code should cause us to set the read timeout as equal to the connect timeout, which is the timeout value you pass on your `Session.get()` call. (Note that I haven't actually tested this code, so it may need some quick debugging, I just wrote it straight into the GitHub window.)

Hopefully this will tide you over until we come to a decision about how to handle this.


@Lukasa,

Thank you for the monkey patch. I have it running on a test stream right now. I understand using this patch is on my own recognizance. That you disavow all knowledge of its existence and will deny that you were ever involved. ;-)

There are three main reasons I've moved to using your general purpose stack. First, I'll be able to use this knowledge elsewhere. (The database I use, Couchbase/CouchDB, uses HTTP for control messages.) Second, Requests API is almost as efficient as the Twitter library I am used to. This is a great accomplishment. Third, Requests is more performant. Requests is able to turn gzip on during the stream. This reduces my bandwidth use by a factor of 4 and, apparently, Twitter gives me more tweets. Yay!

Keep up the good work. I'll watch this repo. I'll try to be aware when you've come to a decision. That said, do not hesitate to ping me. As long running timeouts are hard to test, I'll be happy to start testing the fix when you're team is ready.

Anon,
Andrew


Hi guys,

Thanks for the responses.  Lukasa, outside of the patch there, do you have a gist or sample of the code I could see working in real time?

I briefly mentioned my usage. Here's something more complete: I use requests to basically grab the "information" about a URL pasted. What it does is go to the url, parse what it can based on content.type (what the HTTP server responds with) and also does some guess work with python-magic if there's questions about it. I can see how the overall issue doesn't have a clear solution just yet. 

I was trying something like this:

```
#!/usr/bin/env python
import sys
import requests
from requests.adapters import TimeoutSauce

class MyTimeout(TimeoutSauce):
    def __init__(self, *args, **kwargs):
        connect = kwargs.get('connect', 5)
        read = kwargs.get('read', connect)
        super(MyTimeout, self).__init__(connect=connect, read=read)

requests.adapters.TimeoutSauce = MyTimeout

def get(url, urlread=True):
    try:
        r = requests.get(url, timeout=5, stream=False, allow_redirects=True)
        return r
    except Exception, e:
        print "Error: {0}".format(e)
        return None

url = sys.argv[1]
ff = get(url)

print type(ff), ff
```

When you use it with a pure mp3 stream url like: http://mp3.rtvslo.si/ars

It won't timeout until you KeyboardInterrupt (Ctrl+C)

Thanks.


@Lukasa,

I've had the patch running for two days now and it appears to have properly detected hangups from Twitter. My app then safely restarts the connection.

I'm looking forward to your full featured solution. Rest assured though that it is needed and will be used.

Anon,
Andrew


@reticulatingspline Yes, this won't work for your use-case, I'm afraid. The read timeout function is at the scope of an individual socket `recv()` call, so that if the server stops sending data for more than the read timeout we'll abort.

If you really want to set a maximum length you'll need to use `stream=True` and `iter_content()` in small chunks. The read timeout will then apply to each `iter_content()` call. That, plus some judicious use of the `time` module, should get you the behaviour you want.


@Lukasa 

I was testing this out with it and it seemed to work:

```
class TimeoutException(Exception): pass

@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutException, "Timed out!"
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)
```

I'd call get within the try/except after a with time_limit(10):

That seemed to work.

Same concept?

Thanks.


Yeah, same concept. =)


Any idea on when we can expect a total connection timeout option? The workarounds don't seem as appealing as a simple "total_timeout" option would be.


@mortoray There is no schedule on this feature. Kenneth has expressed ambivalence about the quality of the API, and we won't implement anything until we've got an API he's happy with.


Given that there is no schedule on this feature, having an issue open to track it feels messy.


+1


Further discussion belongs over on https://github.com/sigmavirus24/requests-toolbelt/issues/51