no way to read uncompressed content as file-like object

According to the documentation, there are three ways to read the content of the response: `.text`, `.content` and `.raw`. The first two consider the transfer encoding and decompress the stream automatically when producing their in-memory result. However, especially for the case that the result is large, there is currently no simple way to get at the decompressed result in the form of a file-like object, e.g. to pass it straight into an XML or Json parser.

From the point of view of a library that aims to make HTTP requests user friendly, why should a user have to care about something as low-level as the compression type of the stream that was internally negotiated between the web server and the library? After all, it's the library's "fault" if it defaults to accepting such a stream. In this light, the `.raw` stream is a bit too raw for my taste.

Maybe a fourth property like `.stream` might provide a better abstraction level?


`Response.iter_content`


Erm, no, that's an iterator. I was asking for a file-like object, i.e. something that document processors can read from directly.


It would be pretty simple to make a file-like object with `iter_content`


Thanks for the quick reply, BTW.


I agree. Still, it would be even easier for `requests` to provide this functionality. My point is that `.raw` is the wrong level of abstraction for most use cases that want to read from the stream, because it exposes transfer level details.

Personally, I don't see a major use case for iterating line by line or even chunk by chunk over the result of an HTTP request, but I see several major use cases for parsing from it as a file-like object, specifically response formats that require a document parser, such as HTML, XML, Json etc.


Note also that it's much easier to write an iterator that wraps a file-like object, than a file-like object that wraps an iterator.


I came up with the following code. It handles all necessary cases, but I find it rather complex. That's why I said I wanted something like this as part of the library. Users shouldn't have to figure this out themselves.

I think that the code inside of requests' models.py uses the wrong abstraction here. It should decompress the raw stream _before_ it starts with its iteration machinery, not during iteration. Going from a file-like to an iterator just to go back to a file-like is just plain stupid. A single API transformation is more than enough and most users won't care about content iterators anyway.

```
class FileLikeDecompressor(object):
    """
    File-like object that wraps and decompresses an HTTP stream transparently.
    """
    def __init__(self, stream, mode='gzip'):
        self.stream = stream
        zlib_mode = 16 + zlib.MAX_WBITS if mode == 'gzip' else -zlib.MAX_WBITS  # magic
        self.dec = zlib.decompressobj(zlib_mode)
        self.data = ''

    def read(self, n=None):
        if self.dec is None:
            return '' # all done
        if n is None:
            data = self.data + self.dec.decompress(self.stream.read())
            self.data = self.dec = None
            return data
        while len(self.data) < n:
            new_data = self.stream.read(n)
            self.data += self.dec.decompress(new_data)
            if not new_data:
                self.dec = None
                break
        if self.data:
            data, self.data = self.data[:n], self.data[n:]
            return data
        return ''

def decompressed(response):
    """
    Return a file-like object that represents the uncompressed HTTP response data.
    For compressed HTTP responses, wraps the stream in a FileLikeDecompressor.
    """
    stream = response.raw
    mode = response.headers.get('content-encoding')
    if mode in ('gzip', 'deflate'):
        return FileLikeDecompressor(stream, mode)
    return stream
```


Why don't you build the file-like object from `content_iter` as proposed. This could look like:

```
class FileLikeFromIter(object):
    def __init__(self, content_iter):
        self.iter = content_iter
        self.data = ''

    def __iter__(self):
        return self.iter

    def read(self, n=None):
        if n is None:
            return self.data + '\n'.join(l for l in self.iter)
        else:
            while len(self.data) < n:
                try:
                    self.data = '\n'.join((self.data, self.iter.next()))
                except StopIteration:
                    break
            result, self.data = self.data[:n], self.data[n:]
            return result
```


You may want to read my comment again, specifically the paragraph that precedes the code I posted.


Yes, but this solution is still cleaner (and IMO easier) than doing the decompression at a second place because this is already built-in in requests.

But I agree with you in general, a `r.file` (or something like this) has much more use cases than `r.raw`. So I would like to see this included in requests, too. @kennethreitz 


"response.stream" sounds like a good name to me.


This is what response.raw is for :)


That was also what I intuitively thought when I saw it. But then I realised that response.raw is broken because it exposes internal details of the underlying transport layer that users shouldn't have to care about.


The only method they should need is `raw.read`?


Well, yes - except that raw.read() behaves differently depending on the internal negotiations between client and server. It sometimes returns the expected data and sometimes it returns bare compressed bytes.


Basically, `response.raw` is a nice-to-have feature that most users would happily ignore and some power-users could find helpful, whereas a compression-independent `response.stream` is a feature that most streaming users would want.


+1


+1


Is this design bug going to be fixed?


~~not sure how correct or efficient this way is, but for me, the following works~~:

``` python
>>> import lxml  # a parser that scorns encoding
>>> unicode_response_string = response.text
>>> lxml.etree.XML(bytes(bytearray(unicode_response_string, encoding='utf-8')))  # provided unicode() means utf-8
<Element html at 0x105364870>
```


@kernc: That's a bizarre thing to be doing. `response.content` is already a bytestring, so what you're doing here is decoding the content with whatever the hell codec Python chooses, then re-encoding it as utf-8.

This is _not_ a bug, and it is quite definitely not the bug you suggested. If you really need a file-like object, I recommend StringIO and BytesIO.


@Lukasa is correct. `content` should always be a bytestring (in Python 3 it's an explicit bytestring; in Python 2 str == bytes). The only item that is not a bytestring is `text`.


@kennethreitz any news on this? This is a pretty serious design bug and it's best to sort it out early. The more code gets written to work around it, the more costly it becomes for everyone.


This is no design bug, it is just a feature request. And as requests has a [feature freeze](http://docs.python-requests.org/en/latest/dev/todo#feature-freeze) I assume this won't make it in requests anytime soon (if at all)...


I don't think redeclaring a long standing design bug a "missing feature"
makes it go away all that easily. I heard that the author is thinking about
making "requests" part of the Python stdlib. That would be a good
opportunity to fix this.


>  I heard that the author is thinking about
> making "requests" part of the Python stdlib.

Not really: http://docs.python-requests.org/en/latest/dev/philosophy/#standard-library


This is not a bug, it's a feature request. Requests isn't doing anything wrong, it's simply not doing something that is optional. That's the very definition of a feature.

Additionally, preparing for the stdlib is exactly why Requests is in feature freeze. Once Requests is in the stdlib it becomes very hard to make timely bug fixes. As a result, if adding the new feature adds bugs or regresses behaviour, the version in stdlib can't be fixed until the next minor release. That would be bad.


Marc Schlaich, 19.03.2013 08:41:

> > I heard that the author is thinking about
> > making "requests" part of the Python stdlib.
> 
> Not really: http://docs.python-requests.org/en/latest/dev/philosophy/#standard-library

I read it here:

http://python-notes.boredomandlaziness.org/en/latest/conferences/pyconus2013/20130313-language-summit.html

Stefan


I already explained why this is a design bug and not a feature request: the existing API uses the wrong abstraction and leaks negotiation details of the connection into user space that are at the mercy of the remote site, and thus, that the user should not have to care about. That renders the current raw stream reading support hard to use. Essentially, this is a request for fixing a feature that's broken, not a request for a new feature.


Let me sum this up cleanly. The bug is that any real world usage of the raw stream reading feature will have to reimplement a part of the library, specifically the entire conditional stream decompression part, because the feature is useless without it, as soon as the client permits compression. We are talking about code here that is already there, in "requests" - it's just used in the wrong spot. It should be used below the raw reading level, not above it, because the client cannot control if the server honours the accept header or not. Compression should be a transparent negotiation detail of the connection, not something that hurts any user who enables the relevant header.

I cannot think of any use case where the client would be interested in the compressed stream, especially if it cannot predict if the stream will really be compressed or not, as the server can happily ignore the client's wish. It's a pure negotiation detail. That's why raw stream reading uses the wrong abstraction by prefering the extremely unlikely use case over the most common one.