[Request] Simple argument to limit the size of a downloaded resource

When we use `stream=True` to cap the size of the response, we lose all the bells and whistles of the response object once the response is consumed. For instance, we can no longer return the response to the caller then inspect `status_code`.

In keeping with _simple is beautiful_, it would be terrific if we could do:

`requests.get(url, max_response_size=1024*1024)`

Thanks in advance for considering it. :)


This is a duplicate of https://github.com/kennethreitz/requests/issues/2327

Please use GitHub's issue search before filing future feature requests.


@sigmavirus24 the link you posted is interesting, but I don't see that it addresses:

> For instance, we can no longer return the response to the caller then inspect status_code.

The linked issue had an insightful explanation by @jvanasco of why this feature is needed, then no response... The issue was closed.

If `requests` means to live up to **http for humans**, then... (quoting the first post):

In keeping with simple is beautiful, it would be terrific if we could do:
`requests.get(url, max_response_size=1024*1024)`


> For instance, we can no longer return the response to the caller then inspect status_code.

What does this mean? Sure you can! `status_code` is a static value set once and then left unchanged, streaming the response has _nothing_ to do with it. Can you elaborate by providing some sample code that demonstrates the problem?

> If requests means to live up to http for humans, then... (quoting the first post):
> 
> In keeping with simple is beautiful, it would be terrific if we could do:
> `requests.get(url, max_response_size=1024*1024)`

Sure.

It'd also be terrific if we could do `requests.get(url, total_timeout=5)`. And also `requests.get(url, start_delay=5, max_retries=10)` (see #2682). And also `requests.get(url, websocket_callback=handler)` (see #2634). And also `requests.get(url, local_port=6667)` (see #2563). And also `requests.get(url, socket_options=[(socket.SO_KEEPALIVE, 1)])` (see #2447). And...

But the problem is now you get this line of code:

``` python
r = requests.post(
    url=url,
    data=data,
    max_response_size=1024*1024,
    total_timeout=15,
    start_delay=5,
    max_retries=10,
    websocket_callback=handler,
    local_port=6667,
    socket_options=[(socket.SO_KEEPALIVE, 1)],
    proxies=proxies,
    timeout=6,
    stream=True,
)
```

That line of code is many things: long, weird, mutually contradictory (I specify timeouts in two different manners there, and I've set stream=True but somehow expect a total timeout to be enforced without it being clear what 'total' really means here, and what the hell is the point of passing SO_KEEPALIVE in this context, and really I'm going to get a websocket upgrade from a POST?). But what it isn't is _simple_.

Do you see the point? There are lots of features we _could_ add whose individual interfaces are simple, and that by themselves represent very little burden. But, if we do that for all of them, then we end up like curl. [Here](http://curl.haxx.se/docs/manpage.html) is curl's manpage. curl is a fantastic tool, but I think we can both agree that its interface is not simple.

Simplicity is about constraint. Simplicity is about _not_ doing things, not about doing them. Keeping a project simple means resisting scope creep. It means resisting attempts to add functionality that can be kept outside the library, when doing so does not cause problems for the vast majority of users.

Requests should do things that have excellent _bang for buck_. We should do things that are a bit complex (so that users implementing it by themselves might get it wrong), and that are extremely broadly useful (so that many users would have to implement it themselves). This feature request is not those things.

If users are really worried about a file being accidentally massive, then they should use `stream=True` and track the size themselves. Otherwise, if you pass a response object around a lot, you might suddenly find you get 'unexpected size' errors being thrown from weird places in your code all because of an accidental LoC elsewhere in your program.

Requests' job is primarily to concern itself with making HTTP easy to use. It's not about ensuring well-formedness of data. This is an _application_ concern, and the _application_ should police it.

I'm entirely happy to have a documentation addition that provides a recipe for how to do this, or an addition to the requests-toolbelt. But requests itself has a higher bar for feature requests, and right now I'm not sure this meets it.


@Lukasa 

Thank you very much for your extraordinarily detailed and cogent reply. 

I fully get your point.

I'm more than happy to impose checks on `requests` from outside. I just thought that what I needed was not possible because I had misunderstood how to do it.

> For instance, we can no longer return the response to the caller then inspect status_code.

What I probably should have said there is that you can't return the response and inspect the text. I now see that I had misunderstood how to use `stream=True`. My understanding now is that `response.text` gets consumed while we stream, so we have to rebuild it separately, but the other attributes of the response are still available. So instead of returning the response, we can return a tuple with the response and the rebuilt text. Is that right?

Your idea of posting a recipe is a good one, as the few lines currently in the docs are not enough for a hobbyist like me.

The following seems to work. It follows a [recipe by Martijn Pieters](http://stackoverflow.com/q/23514256/), except that in Python 3 it only worked for me once I set `content` to a byte string.

Is this how you suggest it should be done?

To work with unicode inside instead of bytes, do I just `content.decode(r.encoding)` at the end? By the way for that page, it doesn't work. I thought `r.encoding` was the same as `chardet`, but I get something different and end up having to `content.decode(chardet.detect(content)['encoding'])` Maybe that detection discrepancy is specific to streams?

```
import requests
import chardet

url = 'http://bbc.co.uk'

max_size = 1024*1024*10 #10MB
r = requests.get(url, stream=True, timeout=10)
r.raise_for_status()

size = 0
content = b''
for chunk in r.iter_content(1024):
    size += len(chunk)
    if size > max_size:
        raise ValueError('Maximum exceeded.')
    content += chunk

print("content type", type(content))                           # <class 'bytes'>
print("response: ", r)                                         # <Response [200]>
print("status code:", r.status_code)                           # 200
print("detected encoding:", r.encoding)                        # ISO-8859-1
detected_encoding = chardet.detect(content)['encoding']
print("it should be the same as chardet:", detected_encoding)  # utf-8
ucontent = content.decode(detected_encoding)
print("converted content", type(ucontent))                     # <class 'str'>
print()
print(ucontent)
```

Thank you again for taking the time to explain the interface design.


> Thank you very much for your extraordinarily detailed and cogent reply.

Thank you. =) Fortunately, I've been thinking a lot about this recently because I'm speaking on exactly this topic at PyCon UK this year!

> What I probably should have said there is that you can't return the response and inspect the text. I now see that I had misunderstood how to use stream=True. My understanding now is that response.text gets consumed while we stream, so we have to rebuild it separately, but the other attributes of the response are still available. So instead of returning the response, we can return a tuple with the response and the rebuilt text. Is that right?

Correct. You can even go further and add the content back to the response object when you're done with it: essentially you're just manually building up the `_content` instance variable, so you can probably just patch it back on when you're done.

> The following seems to work. It follows a recipe by Martijn Pieters, except that in Python 3 it only worked for me once I set content to a byte string.

Yeah, you'll have to do that. `content` is always bytes.

> To work with unicode inside instead of bytes, do I just content.decode(r.encoding) at the end?

You could _try_ doing this:

``` python
r._content = content
r._content_consumed = True
```

That _should_ give you back a completely standard `Response` object at that point. These are _technically_ implementation details, but for the long term I expect them to be entirely safe. That's not a _promise_, but it is a suspicion.

Once you've done that, `r.text` should work exactly as it has done in the past: no need to do the decode.

> Is this how you suggest it should be done?

Yep, pretty much! =)


> I'm speaking on exactly this topic at PyCon UK this year!

Ha, didn't know you were one of our illustrious tutors/mentors/evangelists.
Just downloaded your _Hyperactive: HTTP/2 and Python_ video from Montreal, looking forward to watching it later today. :)

Thank you for giving me some ideas about how to do things differently by accessing `requests` internals.

Wishing you a fun weekend.