Repository: urllib3/urllib3

Issue:
[rfe] provide access to chunks when server responds with `Transfer-Encoding: chunked`
I've opened [an issue](https://github.com/kennethreitz/requests/issues/2449) at [requests](https://github.com/kennethreitz/requests) but was advised to move it here. There it goes:

When server responds with `Transfer-Encoding: chunked`, it would be nice if `urllib3` provided access to chunks sent by server. Therefore upper layers, `requests`, could process those encoded chunks and developers wouldn't need to decode those chunks.

More info:
http://en.wikipedia.org/wiki/Chunked_transfer_encoding
http://mihai.ibanescu.net/chunked-encoding-and-python-requests



Discussion:
Hello again!

So, the core problem here is `httplib`. We can achieve this if we grab direct access to the socket underlying `httplib` and decode the chunking ourselves. @shazow, is this a thing you want?

HMMM!! Stealing the socket from httplib is not a thing I have considered.

If it can be done in a reliable/safe way, that could be worthwhile. :)

...how about going even deeper and ask for RFE in `httplib`?

![tzos8](https://cloud.githubusercontent.com/assets/6292/6263461/18abc880-b7cd-11e4-9e74-3301abaee074.gif)

`httplib` should be viewed as a source of pain from which no light or joy can escape. @shazow has repeatedly asked me to rewrite `httplib`. One day I might even do that. =)

I can see there is already code in `httplib` which [reads chunked responses](https://hg.python.org/cpython/file/2.7/Lib/httplib.py#l590). The problem is, it doesn't return generator. Also, looking at [this blog post](http://blog.dowski.com/2008/04/02/reading-chunked-http11-responses/), one should be able to change response class, which looks kinda hacky (but doesn't require patching httplib)

That might be the better way to go about it.

@Lukasa @shazow 
does it mean that I can write a patch which could be merged at some point (or you would be wiling to work on it?). Just to be clear: I don't mind providing the patch. Shouldn't be that hard, I hope.

Ideally you'd write the patch. I'm busy rewriting httplib. ;)

@TomasTomecek Patches are always appreciated. :) Keep in mind that we have a 100% test coverage policy for merging.

Feel free to open a PR and ping it whenever you'd like us to take a look at it, we're happy to help you along the way.


Resolving patch:
diff --git a/urllib3/exceptions.py b/urllib3/exceptions.py
--- a/urllib3/exceptions.py
+++ b/urllib3/exceptions.py
@@ -162,3 +162,8 @@ class SystemTimeWarning(SecurityWarning):
 class InsecurePlatformWarning(SecurityWarning):
     "Warned when certain SSL configuration is not available on a platform."
     pass
+
+
+class ResponseNotChunked(ProtocolError, ValueError):
+    "Response needs to be chunked in order to read it as chunks."
+    pass
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -1,9 +1,15 @@
+try:
+    import http.client as httplib
+except ImportError:
+    import httplib
 import zlib
 import io
 from socket import timeout as SocketTimeout
 
 from ._collections import HTTPHeaderDict
-from .exceptions import ProtocolError, DecodeError, ReadTimeoutError
+from .exceptions import (
+    ProtocolError, DecodeError, ReadTimeoutError, ResponseNotChunked
+)
 from .packages.six import string_types as basestring, binary_type, PY3
 from .connection import HTTPException, BaseSSLError
 from .util.response import is_fp_closed
@@ -117,8 +123,17 @@ def __init__(self, body='', headers=None, status=0, version=0, reason=None,
         if hasattr(body, 'read'):
             self._fp = body
 
-        if preload_content and not self._body:
-            self._body = self.read(decode_content=decode_content)
+        # Are we using the chunked-style of transfer encoding?
+        self.chunked = False
+        self.chunk_left = None
+        tr_enc = self.headers.get('transfer-encoding', '')
+        if tr_enc.lower() == "chunked":
+            self.chunked = True
+
+        # We certainly don't want to preload content when the response is chunked.
+        if not self.chunked:
+            if preload_content and not self._body:
+                self._body = self.read(decode_content=decode_content)
 
     def get_redirect_location(self):
         """
@@ -269,11 +284,15 @@ def stream(self, amt=2**16, decode_content=None):
             If True, will attempt to decode the body based on the
             'content-encoding' header.
         """
-        while not is_fp_closed(self._fp):
-            data = self.read(amt=amt, decode_content=decode_content)
+        if self.chunked:
+            for line in self.read_chunked(amt):
+                yield line
+        else:
+            while not is_fp_closed(self._fp):
+                data = self.read(amt=amt, decode_content=decode_content)
 
-            if data:
-                yield data
+                if data:
+                    yield data
 
     @classmethod
     def from_httplib(ResponseCls, r, **response_kw):
@@ -351,3 +370,59 @@ def readinto(self, b):
         else:
             b[:len(temp)] = temp
             return len(temp)
+
+    def read_chunked(self, amt=None):
+        # FIXME: Rewrite this method and make it a class with
+        #        a better structured logic.
+        if not self.chunked:
+            raise ResponseNotChunked("Response is not chunked. "
+                "Header 'transfer-encoding: chunked' is missing.")
+        while True:
+            # First, we'll figure out length of a chunk and then
+            # we'll try to read it from socket.
+            if self.chunk_left is None:
+                line = self._fp.fp.readline()
+                line = line.decode()
+                # See RFC 7230: Chunked Transfer Coding.
+                i = line.find(';')
+                if i >= 0:
+                    line = line[:i]  # Strip chunk-extensions.
+                try:
+                    self.chunk_left = int(line, 16)
+                except ValueError:
+                    # Invalid chunked protocol response, abort.
+                    self.close()
+                    raise httplib.IncompleteRead(''.join(line))
+                if self.chunk_left == 0:
+                    break
+            if amt is None:
+                chunk = self._fp._safe_read(self.chunk_left)
+                yield chunk
+                self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+                self.chunk_left = None
+            elif amt < self.chunk_left:
+                value = self._fp._safe_read(amt)
+                self.chunk_left = self.chunk_left - amt
+                yield value
+            elif amt == self.chunk_left:
+                value = self._fp._safe_read(amt)
+                self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+                self.chunk_left = None
+                yield value
+            else:  # amt > self.chunk_left
+                yield self._fp._safe_read(self.chunk_left)
+                self._fp._safe_read(2)  # Toss the CRLF at the end of the chunk.
+                self.chunk_left = None
+
+        # Chunk content ends with \r\n: discard it.
+        while True:
+            line = self._fp.fp.readline()
+            if not line:
+                # Some sites may not end with '\r\n'.
+                break
+            if line == b'\r\n':
+                break
+
+        # We read everything; close the "file".
+        self.close()
+


Test patch:
diff --git a/test/test_response.py b/test/test_response.py
--- a/test/test_response.py
+++ b/test/test_response.py
@@ -2,8 +2,12 @@
 
 from io import BytesIO, BufferedReader
 
+try:
+    import http.client as httplib
+except ImportError:
+    import httplib
 from urllib3.response import HTTPResponse
-from urllib3.exceptions import DecodeError
+from urllib3.exceptions import DecodeError, ResponseNotChunked
 
 
 from base64 import b64decode
@@ -156,10 +160,6 @@ def test_body_blob(self):
 
     def test_io(self):
         import socket
-        try:
-            from http.client import HTTPResponse as OldHTTPResponse
-        except:
-            from httplib import HTTPResponse as OldHTTPResponse
 
         fp = BytesIO(b'foo')
         resp = HTTPResponse(fp, preload_content=False)
@@ -174,7 +174,7 @@ def test_io(self):
 
         # Try closing with an `httplib.HTTPResponse`, because it has an
         # `isclosed` method.
-        hlr = OldHTTPResponse(socket.socket())
+        hlr = httplib.HTTPResponse(socket.socket())
         resp2 = HTTPResponse(hlr, preload_content=False)
         self.assertEqual(resp2.closed, False)
         resp2.close()
@@ -406,11 +406,200 @@ def close(self):
         self.assertEqual(next(stream), b'o')
         self.assertRaises(StopIteration, next, stream)
 
+    def test_mock_transfer_encoding_chunked(self):
+        stream = [b"fo", b"o", b"bar"]
+        fp = MockChunkedEncodingResponse(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+
+        i = 0
+        for c in resp.stream():
+            self.assertEqual(c, stream[i])
+            i += 1
+
+    def test_mock_transfer_encoding_chunked_custom_read(self):
+        stream = [b"foooo", b"bbbbaaaaar"]
+        fp = MockChunkedEncodingResponse(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        r.chunked = True
+        r.chunk_left = None
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+        expected_response = [b'fo', b'oo', b'o', b'bb', b'bb', b'aa', b'aa', b'ar']
+        response = list(resp.read_chunked(2))
+        if getattr(self, "assertListEqual", False):
+            self.assertListEqual(expected_response, response)
+        else:
+            for index, item in enumerate(response):
+                v = expected_response[index]
+                self.assertEqual(item, v)
+
+    def test_mock_transfer_encoding_chunked_unlmtd_read(self):
+        stream = [b"foooo", b"bbbbaaaaar"]
+        fp = MockChunkedEncodingResponse(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        r.chunked = True
+        r.chunk_left = None
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+        if getattr(self, "assertListEqual", False):
+            self.assertListEqual(stream, list(resp.read_chunked()))
+        else:
+            for index, item in enumerate(resp.read_chunked()):
+                v = stream[index]
+                self.assertEqual(item, v)
+
+    def test_read_not_chunked_response_as_chunks(self):
+        fp = BytesIO(b'foo')
+        resp = HTTPResponse(fp, preload_content=False)
+        r = resp.read_chunked()
+        self.assertRaises(ResponseNotChunked, next, r)
+
+    def test_invalid_chunks(self):
+        stream = [b"foooo", b"bbbbaaaaar"]
+        fp = MockChunkedInvalidEncoding(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        r.chunked = True
+        r.chunk_left = None
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+        self.assertRaises(httplib.IncompleteRead, next, resp.read_chunked())
+
+    def test_chunked_response_without_crlf_on_end(self):
+        stream = [b"foo", b"bar", b"baz"]
+        fp = MockChunkedEncodingWithoutCRLFOnEnd(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        r.chunked = True
+        r.chunk_left = None
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+        if getattr(self, "assertListEqual", False):
+            self.assertListEqual(stream, list(resp.stream()))
+        else:
+            for index, item in enumerate(resp.stream()):
+                v = stream[index]
+                self.assertEqual(item, v)
+
+    def test_chunked_response_with_extensions(self):
+        stream = [b"foo", b"bar"]
+        fp = MockChunkedEncodingWithExtensions(stream)
+        r = httplib.HTTPResponse(MockSock)
+        r.fp = fp
+        r.chunked = True
+        r.chunk_left = None
+        resp = HTTPResponse(r, preload_content=False, headers={'transfer-encoding': 'chunked'})
+        if getattr(self, "assertListEqual", False):
+            self.assertListEqual(stream, list(resp.stream()))
+        else:
+            for index, item in enumerate(resp.stream()):
+                v = stream[index]
+                self.assertEqual(item, v)
+
     def test_get_case_insensitive_headers(self):
         headers = {'host': 'example.com'}
         r = HTTPResponse(headers=headers)
         self.assertEqual(r.headers.get('host'), 'example.com')
         self.assertEqual(r.headers.get('Host'), 'example.com')
 
+
+class MockChunkedEncodingResponse(object):
+
+    def __init__(self, content):
+        """
+        content: collection of str, each str is a chunk in response
+        """
+        self.content = content
+        self.index = 0  # This class iterates over self.content.
+        self.closed = False
+        self.cur_chunk = b''
+        self.chunks_exhausted = False
+
+    @staticmethod
+    def _encode_chunk(chunk):
+        return '%X\r\n%s\r\n' % (len(chunk), chunk.decode())
+
+    def _pop_new_chunk(self):
+        if self.chunks_exhausted:
+            return b""
+        try:
+            chunk = self.content[self.index]
+        except IndexError:
+            chunk = b''
+            self.chunks_exhausted = True
+        else:
+            self.index += 1
+        encoded_chunk = self._encode_chunk(chunk)
+        return encoded_chunk.encode()
+
+    def pop_current_chunk(self, amt=-1, till_crlf=False):
+        if amt > 0 and till_crlf:
+            raise ValueError("Can't specify amt and till_crlf.")
+        if len(self.cur_chunk) <= 0:
+            self.cur_chunk = self._pop_new_chunk()
+        if till_crlf:
+            try:
+                i = self.cur_chunk.index(b"\r\n")
+            except ValueError:
+                # No CRLF in current chunk -- probably caused by encoder.
+                self.cur_chunk = b""
+                return b""
+            else:
+                chunk_part = self.cur_chunk[:i+2]
+                self.cur_chunk = self.cur_chunk[i+2:]
+                return chunk_part
+        elif amt <= -1:
+            chunk_part = self.cur_chunk
+            self.cur_chunk = b''
+            return chunk_part
+        else:
+            try:
+                chunk_part = self.cur_chunk[:amt]
+            except IndexError:
+                chunk_part = self.cur_chunk
+                self.cur_chunk = b''
+            else:
+                self.cur_chunk = self.cur_chunk[amt:]
+            return chunk_part
+
+    def readline(self):
+        return self.pop_current_chunk(till_crlf=True)
+
+    def read(self, amt=-1):
+        return self.pop_current_chunk(amt)
+
+    def flush(self):
+        # Python 3 wants this method.
+        pass
+
+    def close(self):
+        self.closed = True
+
+
+class MockChunkedInvalidEncoding(MockChunkedEncodingResponse):
+
+    def _encode_chunk(self, chunk):
+        return 'ZZZ\r\n%s\r\n' % chunk.decode()
+
+
+class MockChunkedEncodingWithoutCRLFOnEnd(MockChunkedEncodingResponse):
+
+    def _encode_chunk(self, chunk):
+        return '%X\r\n%s%s' % (len(chunk), chunk.decode(),
+            "\r\n" if len(chunk) > 0 else "")
+
+
+class MockChunkedEncodingWithExtensions(MockChunkedEncodingResponse):
+
+    def _encode_chunk(self, chunk):
+        return '%X;asd=qwe\r\n%s\r\n' % (len(chunk), chunk.decode())
+
+
+class MockSock(object):
+    @classmethod
+    def makefile(cls, *args, **kwargs):
+        return
+
+
 if __name__ == '__main__':
     unittest.main()