Implement a peek method. 

From http://stackoverflow.com/questions/13197854/python-requests-fetching-the-head-of-the-response-content-without-consuming-it/

Using python-requests and python-magic, I would like to test the mime-type of a web resource without fetching all its content (especially if this resource happens to be eg. an ogg file or a PDF file). Based on the result, I might decide to fetch it all. However calling the text method after having tested the mime-type only returns what hasn't been consumed yet. How could I test the mime-type without consuming the response content?

Below is my current code.

```
import requests
import magic


r = requests.get("http://www.december.com/html/demo/hello.html", prefetch=False)
mime = magic.from_buffer(r.iter_content(256).next(), mime=True)

if mime == "text/html":
    print(r.text)  # I'd like r.text to give me the entire response content
```

I've been given the following solution:

```
r = requests.get("http://www.december.com/html/demo/hello.html", prefetch=False)
peek = r.iter_content(256).next()
mime = magic.from_buffer(peek, mime=True)

if mime == "text/html":
    contents = peek + ''.join(r.iter_content(10 * 1024))
    encoding = r.encoding
    if encoding is None:
        # detect encoding
        encoding = chardet.detect(contents)['encoding']
    try:
        textcontent = unicode(contents, encoding, errors='replace')
    except (LookupError, TypeError):
        textcontent = unicode(contents, errors='replace')
    print(textcontent)
```

I was wondering if such a case could be adressed by the api directly, for instance by implementing a peek method, or by making it possible to rewind the iter_content cursor.

Thanks


Sorry about the delay in getting back to you on this!

Is there any reason you can't do this with the HEAD request? For example, can you do:

``` python
import requests

url = "http://www.december.com/html/demo/hello.html'

r = requests.head(url)
if r.headers['Content-Type'] == 'text/html':
    r = requests.get(url)
    # Do stuff
```


## If I'm not mistaken, not every website supports a request with HEAD. But don't mistake that for support of this issue. I'm +0 on this.

Sent from my Android phone with K-9 Mail. Please excuse my brevity.


Yeah, there are some that don't, but if @aleray can handle his particular problem using HEAD then that will work well. =D


Hey,

thanks for your comments. Actually I could use the served content-type but since it is sometimes wrong I wanted to double-check with python-magic, based on the first bytes of the response body... Would there be a way to "cache" the content so I could access the full content of the page several times without having to worry about the response content being consumed by my first look at it?

Thanks


> Would there be a way to "cache" the content so I could access the full content of the page several times without having to worry about the response content being consumed by my first look at it?

`cached_content = reply.content` ?


Yes, except that if I'm facing an 300 Gb ogg video I would rather avoid to download it entirely if the first bytes are enough for what I need to do...


@aleray: you can do that today:

``` pycon
>>> r = request.get('http://...', prefetch=False)
>>> r.raw.read(10)
'xxxxxxxxxx'
```


On Sat, Nov 24, 2012 at 05:56:36PM -0800, Kenneth Reitz wrote:

> @aleray: you can do that today:
> 
> ``` pycon
> >>> r = request.get('http://...', prefetch=False)
> >>> r.raw.read(10)
> 'xxxxxxxxxx'
> 
> ```

I was just thinking of this. With a normal socket (or file-descriptor-like 
API), you can seek backward to the beginning of the file, can you do that with 
the raw socket? If so, the peek method becomes trivial.


I'm unaware. @shazow?


`dir` would imply otherwise. Sorry for getting anyone's hopes up.


Peek normally isn't 100% required. For example, the standard library's `gzip` library has a little-known flag you can pass to the mode that lets it work on streams, without peek.


```
>>> r = request.get('http://...', prefetch=False)
>>> r.raw.read(10)
```

@kennethreitz
But this is not peeking what OP askes for but reading which would influence data returned by the api like `.content` etc., right? Is the untold assumption that a client must remember to prepend data read outside of the api to what api returns later?

It looks like it would be possible to do with slightly modified version of [`tee`](http://docs.python.org/2/library/itertools.html#itertools.tee) from `itertools`. The modification would be to allow for removing given deques so that they would not receive and accumulate further data.

UPDATE

This is not as straightforward as `iter_content` is a generator not an iterator. Still I think it's doable by making `Response.raw` an iterator and teeing it.


@piotr-dobrogost `raw` is a file-like object, no? Can you not iterate over it? If not, that might be a nifty feature to add to urllib3 (if I remember correctly that raw is a urllib3 object).


> Can you not iterate over it?

Sure, you can. I was focused on teeing `iter_content` and for some reason I thought it's not possible as `iter_content` is a generator. However it's possible (http://stackoverflow.com/q/1271320/95735). So the thing to do would be modification of `tee` as described above.


@aleray, did @piotr-dobrogost's solution satisfy your needs?


You could always request a small byte range, determine it's content-type with libmagic, and then re-request it again its entirety if you want. This does involve two requests, but you won't have to slurp down a full 300GB of a movie twice. You also won't have to muck about with raw sockets at all.

```
r = requests.get("http://www.december.com/html/demo/hello.html", headers={'Range': 'bytes=0-256'})
```