Repository: urllib3/urllib3

Issue:
No tolerance for responses to head requests that indicate chunking.
As shown [here](https://ptpb.pw/r/AFSJnZhspw4EkC13PK8TBHpNqMDH), if we receive a redirect response that has no body, but claims Transfer-Encoding: chunked, we will hang indefinitely waiting for a chunked response we never see.

This has been observed with requests 2.6.2 (containing the new chunked handling) but not with requests 2.6.0 (containing the old broke stuff), which moves along just fine.

Now, it's worth noting that Google has totally screwed up here, because the response is invalid, as this Wireshark trace shows:

```
HEAD /~r/DeveloperChronicles/~3/iauy0NfhAus/hitchhikers-guide-to-modern-android.html HTTP/1.1
Host: feedproxy.google.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.2 CPython/2.7.8 Windows/7

HTTP/1.1 301 Moved Permanently
Location: http://www.devchronicles.com/2015/03/hitchhikers-guide-to-modern-android.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DeveloperChronicles+%28Developer+Chronicles%29
Content-Type: text/html; charset=UTF-8
Date: Tue, 28 Apr 2015 09:56:23 GMT
Expires: Tue, 28 Apr 2015 09:56:23 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Server: GSE
Alternate-Protocol: 80:quic,p=1
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding

```

Note that Google have sent Transfer-Encoding: chunked, but no body. This is, as far as I can tell, against all the relevant RFCs: a zero length body has a valid chunked representation (`0\r\n\r\n`). 301 is also not defined as having no body.

Regardless, however, curl gets this right, so we should too if at all possible. We should also confirm whether this is a problem for older versions of urllib3 or whether they too function correctly.

Arguably this may want to block a requests release, @sigmavirus24.



Discussion:
Actually, no, I'm wrong. From RFC 7231:

> The HEAD method is identical to GET except that the server MUST NOT send a message body in the response (i.e., the response terminates at the end of the header section).

I suspect there was special case code in httplib for this.

I'm going to guess that httplib was checking the request method to determine if it should expect a body

Seems likely to me, we should probably do the same.

There's a lesson being learned here about overriding httplib functions...

https://hg.python.org/cpython/file/293d9964cf6e/Lib/http/client.py#l344 is likely the check. Looks like we should be doing the same-ish.

> There's a lesson being learned here about overriding httplib functions...

Is the lesson: "Although httplib is terrible in so many cases, it still has a lot of handling that we don't know we need"?

> Is the lesson: "Although httplib is terrible in so many cases, it still has a lot of handling that we don't know we need"?

Yes.

Ideally, I'd prefer for urllib3 to not care whether a body is provided or not. :)

Alternatively, https://hg.python.org/cpython/file/293d9964cf6e/Lib/http/client.py#l426 is a way to handle this. Since httplib stores that information for us, and we're already accessing private things on an HTTPResponse, we may as well follow that logic as well.

@sigmavirus24 That looks like the generally correct action.

I've a fix in the works for this.


Resolving patch:
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -432,11 +432,17 @@ def read_chunked(self, amt=None, decode_content=None):
             'content-encoding' header.
         """
         self._init_decoder()
-        # FIXME: Rewrite this method and make it a class with
-        #        a better structured logic.
+        # FIXME: Rewrite this method and make it a class with a better structured logic.
         if not self.chunked:
             raise ResponseNotChunked("Response is not chunked. "
                 "Header 'transfer-encoding: chunked' is missing.")
+
+        if self._original_response and self._original_response._method.upper() == 'HEAD':
+            # Don't bother reading the body of a HEAD request.
+            # FIXME: Can we do this somehow without accessing private httplib _method?
+            self._original_response.close()
+            return
+
         while True:
             self._update_chunk_length()
             if self.chunk_left == 0:


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -616,3 +616,33 @@ def test_httplib_headers_case_insensitive(self):
         HEADERS = {'Content-Length': '0', 'Content-type': 'text/plain'}
         r = pool.request('GET', '/')
         self.assertEqual(HEADERS, dict(r.headers.items())) # to preserve case sensitivity
+
+
+class TestHEAD(SocketDummyServerTestCase):
+    def test_chunked_head_response_does_not_hang(self):
+        handler = create_response_handler(
+           b'HTTP/1.1 200 OK\r\n'
+           b'Transfer-Encoding: chunked\r\n'
+           b'Content-type: text/plain\r\n'
+           b'\r\n'
+        )
+        self._start_server(handler)
+        pool = HTTPConnectionPool(self.host, self.port, retries=False)
+        r = pool.request('HEAD', '/', timeout=1, preload_content=False)
+
+        # stream will use the read_chunked method here.
+        self.assertEqual([], list(r.stream()))
+
+    def test_empty_head_response_does_not_hang(self):
+        handler = create_response_handler(
+           b'HTTP/1.1 200 OK\r\n'
+           b'Content-Length: 256\r\n'
+           b'Content-type: text/plain\r\n'
+           b'\r\n'
+        )
+        self._start_server(handler)
+        pool = HTTPConnectionPool(self.host, self.port, retries=False)
+        r = pool.request('HEAD', '/', timeout=1, preload_content=False)
+
+        # stream will use the read method here.
+        self.assertEqual([], list(r.stream()))