Add option to limit response downloads by size.

Would it be possible to add a "limit" argument that will limit downloads to a certain size? I want my users to be able to make requests to URLs, but I don't want them to download gigabytes through them, I want the request to end after, say, 1 KB.

This will complement the "timeout" argument.


The API is frozen so the short answer is no.

A longer answer can probably be found on other discussions of this exact same feature. If you want to do this for your users you can use the `stream` parameter and measure what you've downloaded thus far yourself. After you've hit your limit, you can close the connect yourself.

Something along those lines would be:

``` python

r = requests.get(url, stream=True)

chunk = r.raw.read(1024)
r.close()
```


Does it matter if the API is frozen? Keyword parameters (maybe together with putting this argument last) make it backwards-compatible.

Will the solution you describe still enable processing of the response? I.e. If the response is less than 1 KB, will response.json() still decode the JSON?


@skorokithakis it does matter. Frozen does not mean that backwards compatible changes are allowable. Generally speaking a frozen API means that no new features can be made. On the other hand, given a frozen API and semantic versioning a backwards compatible change can be made assuming it doesn't add anything to the API.

The solution is just for the cases you mentioned in your issue report. You can always pass the `stream=True` parameter and still use `r.json()`.


I agree that using streaming is the correct approach here. Perhaps we should add some reference documentation around this?


Alright, thanks, I'll close this then.


What you proposed doesn't work, unfortunately:

```
In [35]: requests.get("http://pastebin.com/raw.php?i=jJywkuB5").json()
Out[35]: {u'response': u'Yep all ok'}

In [36]: r = requests.get("http://pastebin.com/raw.php?i=jJywkuB5", stream=True)

In [37]: r.raw.read(1024)
Out[37]: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xabV*J-.\xc8\xcf+NU\xb2RP\x8aL-PH\xcc\xc9Q\xc8\xcfV\xaa\x05\x006\x16xH\x1a\x00\x00\x00'

In [38]: r.close()

In [39]: r.json()
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-39-9c78734cbfe1> in <module>()
----> 1 r.json()

...snip...

JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

In [41]: r.content
Out[41]: ''

In [42]: r.text
Out[42]: u''
```


If you read directly from the raw response object, this will necessarily fail.

What you're doing is not the right way to approach this. If you're downloading JSON that is more than 1kB in size and you truncate it at 1kB, that's just not going to be valid JSON. No JSON decoder will accept it.

You have some contradictory goals. What exactly are you attempting to do?


As you can see above, the JSON is around 30 bytes, so reading a KB shouldn't fail (since the returned data isn't even close to 1 KB). I want to limit downloads to 1 KB, and close the socket and return without processing anything if it's over 1 KB.


Then the correct behaviour is this:

``` python
r = requests.get('http://www.url.com/', stream=True)

if int(r.headers['Content-Length']) > 1024:
    return None
else:
    return r.json()
```


Thanks, that's better, but there are two issues:
1. Will requests stop downloading when it gets that that many bytes? I can see a malicious server saying it will send me 10 bytes and sending 10 GB.
2. This:

```
In [7]: requests.get("http://pastebin.com/raw.php?i=jJywkuB5", stream=True).headers
Out[7]: 
{'cf-ray': 'd15713f2f5606e8',
 'connection': 'keep-alive',
 'content-encoding': 'gzip',
 'content-type': 'text/plain; charset=utf-8',
 'date': 'Fri, 22 Nov 2013 11:57:29 GMT',
 'server': 'cloudflare-nginx',
 'set-cookie': '__cfduid=db28dc4309546661b06adc9f4138883611385121448827; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.pastebin.com; HttpOnly',
 'transfer-encoding': 'chunked',
 'vary': 'Accept-Encoding',
 'x-powered-by': 'PHP/5.5.5'}
```


If you're determined to force it, you have to abandon some of the convenience of requests. You can try this:

``` python
r = requests.get('http://url.com/', stream=True
data = next(r.iter_content(1024))
r.close()

# Work with the data.
```


Right, since all I need is to extract some short JSON from it and fail otherwise, that will be good enough for my purposes, thank you.


Is there any chance to reopen this request? No being able to limit the received size is a significant issue for any user that processes user provided URLs.

I see the stream option mentioned here, but it's a bit inconvenient to use (perhaps it's just a lack of documentaion on how it works). But it also doesn't appear to consider the headers, streaming only the data. A misbhaving server could still send me infinite header data.


@mortoray The way the `stream` option is intended to work is to have the user (in this case yourself) take control of how to handle data downloads. For example, you've got a use case where you want to limit the size of the downloaded body. You can do something like this:

``` python
def download_limited(url, max_size):
    r = requests.get(url, stream=True)

    if int(r.headers.get('Content-Length', 0)) > max_size:
        return None

    data = []
    length = 0

    for chunk in r.iter_content(1024):
        data.append(chunk)
        length += len(chunk)
        if length > max_size:
            return None

    return b''.join(data)
```

If you're concerned about being sent infinite header data you have a very separate problem, which is that Requests uses urllib3 which uses httplib for its header parsing. There's an [open Python issue](http://bugs.python.org/issue16037) which tracks the fact that `httplib` has no maximum limit on the number of headers downloaded. This has been fixed in 2.6 and 3.3, but I don't see a fix on my copy of 2.7.6. I'm going to check out the codebase and confirm, but I suspect this fix hasn't been merged. There's nothing Requests can do for you in that situation.


Yeah, it doesn't look like it's in my copy of Mercurial, oddly.


I'll see if reviewing that patch provides any impetus to get that merged into 2.7.


Is there any way I can get that streamed data back into the response? It'd be convenient if I can still use the rest of the response interface, like json and data. That is, I'm only streaming it to check for the size limit, not because I actually want the stream.


I added the below issue about timeouts and streaming. It limits this method of checking the maximum size.
https://github.com/kennethreitz/requests/issues/1948


Sure: when you've read the whole interface, just do `r._content = b''.join(chunks)`.