Internal encoding detection doesn't match external chardet call

**requests**' method for detecting a document's encoding when the `Content-Type` header does not have it seems to work differently than external methods.
Referencing #480, **chardet** is used to detect the encoding of a document and presumably the `string.decode()` for converting a `response.content` to `response.text` uses this guess, again if the encoding is not in the HTTP headers.  However, testing reveals that the guess provided by **requests** is subpar compared to a standalone call to `chardet.detect()`.

Using this code:

``` python
import requests
import urllib
import chardet

page_requests = requests.get('http://www2.ocn.ne.jp/~cast/')
page_urllib = urllib.FancyURLopener().open('http://www2.ocn.ne.jp/~cast/').read()

print page_requests.encoding
print chardet.detect(page_urllib)['encoding']
```

produces as output:

```
$ python2 encoding_test.py
ISO-8859-1
SHIFT_JIS
```

This becomes a problem when relying on `response.text` for input, as improperly decoded Python Unicode strings break a lot of other modules.  The present solution would be to rely on `response.content` and do the `response.content.decode(chardet.detect(response.content)['encoding'])` manually, but if **requests** is to have a pre-decoded Unicode string and an `encoding` field, the detection should be as accurate as possible.


That page sends a header `Content-Type: text/html`; since the header contains the string `text`, `requests.utils.get_encoding_from_headers()` returns `ISO-8859-1` and we don't attempt detection with `chardet`.

This looks like the standards-compliant behavior: http://en.wikipedia.org/wiki/ISO/IEC_8859-1#History


Another wrinkle is that the bytes from that page do successfully decode as ISO-8859-1 (Latin-1). If they failed, there might be a case for trying `chardet` and re-decoding with chardet's guess. But in this case, we don't really seem to have a good indication not to trust the header. @ensigntropic, thoughts?


Strictly speaking, all bytes successfully decode as ISO-8859-1 since its character width is only one byte. That is, it's a 1:1 conversion between a byte string and a Unicode string.  That's how I first noticed the problem was with the encoding; the Unicode string wasn't showing Unicode-escaped characters e.g. `\u0000`.

Standards-compliant behaviour is good and all but https://github.com/kennethreitz/requests/issues/480#issuecomment-4859364 mentions an internal **chardet** call (specifically referencing the lack of header), and it seems buggy to instantly fallback on Latin-1 without even attempting a heuristic check a la **chardet**, especially when `resource.text` relies on the `resource.encoding` for conversion.
Again the main issue is a broken Unicode string that's useless because it doesn't contain the correct Unicode data.  If there's going to be a reliance on the encoding of a document for an internal field, it should be as accurate as it can be.

My ideal fix would be to perform the heuristic check first (after discovering the `Content-Type` header lacked `charset`), then fallback to Latin-1 if _all else fails_.  It may not be immediately standards-compliant, but the scenarios where it would break things are possibly far fewer than those where it would help.


If I may offer another example:  http://docs.python.org/library/pdb.html

This page lacks a `charset` in its headers, but not in its `<meta>` tags.  It is UTF-8, and of immediate note is the `<title>` which contains one Unicode em dash (—) and one `&;` escaped em dash (&mdash;).  When decoded as Latin-1, that first em dash breaks; however, **chardet** successfully identifies the page as UTF-8.

``` python
>>> import urllib
>>> import chardet
>>> import re
>>> page = urllib.FancyURLopener().open('http://docs.python.org/library/pdb.html').read()

>>> print re.findall('(?i)(?<=<title>).*(?=</title>)', page.decode('latin-1'))[0]
26.2. pdb â The Python Debugger &mdash; Python v2.7.3 documentation
>>> print chardet.detect(page)['encoding']
utf-8
>>> print re.findall('(?i)(?<=<title>).*(?=</title>)', page.decode(chardet.detect(page)['encoding']))[0]
26.2. pdb — The Python Debugger &mdash; Python v2.7.3 documentation
```


That sounds reasonable. My concern is that `chardet` is quite slow; right now, we only invoke it lazily, when the user wants the Unicode text of the page (the `Response.text` property) and we couldn't extract an encoding from the headers.

I think it's reasonable that `Response.encoding` should contain the page's declared encoding in accordance with the standards (i.e., the current behavior). But maybe `Response.text` should see if the page actually declared Latin-1, or we just assumed Latin-1 because of `text/html`, and in the second case use `chardet`'s response instead of Latin-1.


Requests doesn't care about meta tags. This is an http module, not an html module. 

I chose to trust the server before I trust chardet. The server is lying to you, so requests should raise an exception, not ignore the http standard. You're more than welcome to change the value of `response.encoding`, however. If you set it to `'utf-8'`, then it will work as expected. If you'd like to do this all the time, you can implement it with a hook.


Given that, it essentially makes `response.text` useless at worst, and unreliable at best.  If I can't trust **requests** to encode the `text` field correctly, I'm better off using `content` and doing my own work on it.  That a seeming majority of servers send the charset in the header doesn't make the current behaviour of `response.text`, nor the standard it's based on, reliable.  **chardet** may be slow, but it is more reliable and certainly not limited to HTML `<meta>`.

Setting `response.encoding` does indeed change the conversion of `response.content` to `response.text`, but setting it after the request is kind of just an alias for doing the `string.decode()` myself unless I plan on handing the `response` around to other functions; that and I still have to have client code to determine to _what_ encoding it needs to be set (setting it to `utf-8` all the time will not work for documents encoded in `shift-jis`, etc.), again negating the real usefulness of `response.text` in the first place.

This issue is not constrained to HTML documents; any textual file could be in an encoding that the server doesn't report, and therefore `response.text` will be unreliable every time one doesn't.  I fully understand not wanting to do it for every request, but once the first conversion from `content` to `text` is called, the logic should be in place to make sure the resulting Unicode string is going to be properly encoded.  The byte string of `content` very obviously doesn't need this extra baggage, and if one knows they won't need to access textual data, the encoding detection logic will never trigger.