Error with decode_unicode=True on streamed response

With `requests==2.11.0` I get `AttributeError: 'NoneType' object has no attribute 'readline'` when trying to use `iter_lines` with `decode_unicode=True` on a streamed response.

``` sh
$ ./bin/pip freeze
requests==2.11.0
wsgiref==0.1.2
```

``` python
from contextlib import closing
import requests


with closing(requests.get('http://httpbin.org/stream/20', stream=True)) as response:
    for line in response.iter_lines(chunk_size=30, decode_unicode=True):
        print line
```

``` python
$ ./bin/python case.py

Traceback (most recent call last):
  File "case.py", line 6, in <module>
    for line in response.iter_lines(chunk_size=30, decode_unicode=True):
  File "/Users/jone/temp/requests-stream/lib/python2.7/site-packages/requests/models.py", line 720, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/Users/jone/temp/requests-stream/lib/python2.7/site-packages/requests/utils.py", line 374, in stream_decode_response_unicode
    for chunk in iterator:
  File "/Users/jone/temp/requests-stream/lib/python2.7/site-packages/requests/models.py", line 676, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/jone/temp/requests-stream/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 353, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/jone/temp/requests-stream/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 521, in read_chunked
    line = self._fp.fp.readline()
AttributeError: 'NoneType' object has no attribute 'readline'
```

When removing either `stream=True` or `decode_unicode=True` the script works well.

This issue may be related to #3174.


Well this is weird: it's not immediately apparent to me how `decode_unicode` leads to this behaviour.


Aha, got it.

`decode_unicode=True` was enhanced in the last release to perform the same auto-detection of encoding as `text` does for Requests. Unfortunately, that may involve a call to `apparent_encoding`, which streams in the entire request body and attempts to determine the encoding.

This was done as part of #3362 in an attempt to resolve #3359.

Frankly, this leads me to suspect that perhaps we should revert that change. If `apparent_encoding` is going to read the whole response body then there is genuinely no point in falling back to `apparent_encoding` as part of `iter_*`: any reason to use the method has been totally lost.

I wonder if we should instead revert the fix in #3362, and then do one of the following three things:
1. Use UTF-8 as an unconditional fallback if no specific encoding is declared.
2. Raise an exception if no specific encoding is declared.
3. Allow `decode_unicode` to take an encoding name which is used as a fallback.

@kennethreitz @sigmavirus24?


For you, @jone, at least for the short term I recommend not using `decode_unicode`. It saves relatively little code and probably doesn't really work the way you want it to work anyway.


@Lukasa thanks for looking into it that fast!
Yes, I'm going to decode manually, which is quite an easy change for me. 😉 


I really want to tear out `apparent_encoding` and wish it well on future endeavors because it's nothing but a constant headache for us.

Let's revert #3362 ASAP, release 2.11.1 and take a while to decide on what is the correct behaviour here. I don't like the idea of auto-falling back to UTF-8. I feel like this will cause a spate of more bugs as people find that it is insufficient and want a way to pass in the fallback encoding (which I don't see as an unreasonable request).

I also don't think we can raise an exception until 3.0 if no encoding is specified. That said, I'm _firmly_ against `decode_unicode` taking an encoding name. I'd rather that be a separate parameter, e.g., `encoding`.


I'd like to sit on 2.11.1 until at least tomorrow, to give us another day of bugs coming out of the woodwork.


That's fine. I don't think we need to release it until Monday honestly. We just shouldn't wait to revert #3362.


Not sure if this is relevant to add to this, but previous versions of requests raised `UnicodeDecodeError` but 2.11.0 appears to be raising just a `UnicodeError` when `decode_unicode=True` and the content is not unicode. Is the recommended workaround for this to use requests 2.10.0 until this revert is released?


@maxpowa #3486 reverts this change and should be included in the upcoming 2.11.1 release. As @Lukasa suggested [above](https://github.com/kennethreitz/requests/issues/3481#issuecomment-238489871), it's recommended not to use `decode_unicode=True` for the time being. You can either manually decode the strings as a temporary fix, or continue using 2.10 if your workflow allows it.