get http content as unicode...

It would be nice if we can request a html page as unicode... In my https://github.com/jedie/django-tools/blob/master/django_tools/utils/http.py i have a simple solution for this. The main code is:

``` python
    def get_content_type(self):
        content_type = self.response_header.get("content-type")
        content_type, params = cgi.parse_header(content_type)
        return content_type, params

    def get_encoding_from_content_type(self):
        content_type, params = self.get_content_type()
        if "charset" in params:
            return params["charset"].strip("'\"")

    def get_encodings_from_content(self, content):
        if self._charset_re is None:
            self._charset_re = re.compile(
                r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I
            )
        return self._charset_re.findall(content)

    def get_unicode(self):
        """
        Returns the requested content back in unicode.
        Tried:
        1. charset from content-type
        2. every encodings from <meta ... charset=XXX>
        3. fall back and replace all unicode characters
        """
        content = self.get_content()

        # Try charset from content-type
        encoding = self.get_encoding_from_content_type()
        if encoding:
            try:
                return unicode(content, encoding)
            except UnicodeError:
                self.tried_encodings.append(encoding)

        # Try every encodings from <meta ... charset=XXX>
        encodings = self.get_encodings_from_content(content)
        for encoding in encodings:
            if encoding in self.tried_encodings:
                continue
            try:
                return unicode(content, encoding)
            except UnicodeError:
                self.tried_encodings.append(encoding)

        # Fall back:
        return unicode(content, encoding, errors="replace")
```

I know a real solution can do more, e.g. what http://chardet.feedparser.org/ does.
But IMHO it's ok for the most request, isn't it?

Is there any interests to merge this? If so, I would try to contribute it ;)


Here is my humble opinion: I would not manipulate content in the response class. I would let developers to apply any hook to response content, and leave content manipulation to application specific needs. A library of hooks may be more interesting.


The `develop` branch currently has this merged in. If the response headers say that a page is a certain encoding, you'll get a unicode object back instead of bytes.

Personally, I think this is a good thing to do. Requests has been automatically decoding gzip'd responses for a while, and that has been received well.

Note that this behavior can be opted-out with a setting or using `resp.fo.read()` directly.

Anyone else have any thoughts?


@kennethreitz: It would have been good if you had written here that you've done the merge here: https://github.com/kennethreitz/requests/commit/8fbb1e6d97cda90d588d4263a18906a52d147fba#requests/utils.py


I understand the use of the charset from the headers, but if you take the path of parsing the content, you have to handle:
1. what content it is, because it may be JSON with a string that matches that regular expression
2. parse it, the real way (because that regular expression may fail on some html)
3. validate it, because even if you can parse (let's say html), parsed content may have 2 meta tags with 2 values of charset.


@monkeython excellent point. Let's just stick to headers. This is an HTTP library after all, not HTML :)


I think that getting the encoding from the meta-tags is a nice feature, but shouldn't be part of a normal request. 

As I said above, this is an HTTP library, not an HTML library. So, I'll only unicodify responses with encoding response headers.

I'm going to leave this functionality in the `requests.utils` module though.