Repository: urllib3/urllib3

Issue:
'Failed to parse headers' warning logged when getting message/rfc822 content
I've been investigating an [issue](https://github.com/boto/botocore/issues/1551) I've recently discovered when retrieving objects from S3. I've now tracked it to `urllib3`; this test case (which I've added to `urllib3/test/with_dummyserver/test_socketlevel.py`) demonstrates it:

```py
class TestOkayHeaders(SocketDummyServerTestCase):

    def _test_okay_header_parsing(self, header):
        self.start_response_handler((
           b'HTTP/1.1 200 OK\r\n'
           b'Content-Length: 0\r\n'
           ) + header + b'\r\n\r\n'
        )

        pool = HTTPConnectionPool(self.host, self.port, retries=False)
        self.addCleanup(pool.close)

        with LogRecorder() as logs:
            pool.request('GET', '/')

        for record in logs:
            assert 'Failed to parse headers' not in record.msg

    def test_header_text_plain(self):
        self._test_okay_header_parsing(b'Content-type: text/plain')

    def test_header_message_rfc822(self):
        self._test_okay_header_parsing(b'Content-type: message/rfc822')
```

The test with `text/plain` passes, while the test with `message/rfc822` fails, and this is logged:

```py
Failed to parse headers (url=http://localhost:36732/): Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7f8fab9373c8>]
Traceback (most recent call last):
  File "/home/user/git/urllib3/src/urllib3/connectionpool.py", line 396, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "/home/user/git/urllib3/src/urllib3/util/response.py", line 68, in assert_header_parsing
    raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
urllib3.exceptions.HeaderParsingError: Unknown, unparsed data: [<http.client.HTTPMessage object at 0x7f8fab9373c8>]
```

While retrieving content of type `message/rfc822` still works, the warning message being logged is incorrect and unhelpful.



Discussion:
Thanks for opening this issue. I'm able to reproduce on Python 3.7.0 and this does look like a bug.

Looks to be within [`urllib3.util.response.assert_header_parsing`](https://github.com/urllib3/urllib3/blob/master/src/urllib3/util/response.py#L38) where we call `get_payload` and we receive a zero-length `HTTPMessage` with no defects that we don't receive for other message types. Could be due to how `message/rfc822` is handled by httplib? Will require some research on my part.
Maybe we need to change the `if get_payload:` branch into something like:

```python
    unparsed_data = None
    if get_payload:
        payload = get_payload()

        # Non-multipart message.
        if isinstance(payload, (bytes, str)):
            unparsed_data = payload

        # Multipart message.
        elif isinstance(payload, list):
            multipart_defects = [x.defects for x in payload if getattr(x, "defects", None)]
            if multipart_defects:
                if defects is None:
                    defects = multipart_defects
                else:
                    defects.extend(multipart_defects)
```

With this change the above unit test passes.
> Could be due to how `message/rfc822` is handled by httplib?

Yes, I believe it is.

In `http.client.parse_headers`, it instantiates `email.parser.Parser` to parse the HTTP headers. Then in `email.feedparser.FeedParser` in the `_parsegen()` method, there's this line:

```py
        if self._cur.get_content_maintype() == 'message':
```

It's in the following code-block that the extra `HTTPMessage` object is generated.

Yeah this is definitely a change we need to make then. I can open a PR and create tests later tonight. :)
Is the proposed fix overkill?

If we change the end of `assert_header_parsing()` as follows:

```py
    if defects:
        print(unparsed_data)  # dummy
        raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)

    if unparsed_data:
        print(unparsed_data)  # dummy
        raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
```

we can do some checks with `coverage` to see which of the two conditions actually arises in the test cases (without the extra `TestOkayHeaders` test).

This shows that the `unparsed_data` check isn't triggered by the test suite. Are there any real world situations (except for the case I've identified in this issue) where `unparsed_data` could be non-empty? If not, then perhaps we shouldn't check it at all, which makes the fix for this issue much easier.

Resolving patch:
diff --git a/src/urllib3/util/response.py b/src/urllib3/util/response.py
--- a/src/urllib3/util/response.py
+++ b/src/urllib3/util/response.py
@@ -59,8 +59,14 @@ def assert_header_parsing(headers):
     get_payload = getattr(headers, 'get_payload', None)
 
     unparsed_data = None
-    if get_payload:  # Platform-specific: Python 3.
-        unparsed_data = get_payload()
+    if get_payload:
+        # get_payload is actually email.message.Message.get_payload;
+        # we're only interested in the result if it's not a multipart message
+        if not headers.is_multipart():
+            payload = get_payload()
+
+            if isinstance(payload, (bytes, str)):
+                unparsed_data = payload
 
     if defects or unparsed_data:
         raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -1289,12 +1289,12 @@ def socket_handler(listener):
 )
 class TestBrokenHeaders(SocketDummyServerTestCase):
 
-    def _test_broken_header_parsing(self, headers):
+    def _test_broken_header_parsing(self, headers, unparsed_data_check=None):
         self.start_response_handler((
            b'HTTP/1.1 200 OK\r\n'
            b'Content-Length: 0\r\n'
            b'Content-type: text/plain\r\n'
-           ) + b'\r\n'.join(headers) + b'\r\n'
+           ) + b'\r\n'.join(headers) + b'\r\n\r\n'
         )
 
         pool = HTTPConnectionPool(self.host, self.port, retries=False)
@@ -1306,26 +1306,52 @@ def _test_broken_header_parsing(self, headers):
         for record in logs:
             if 'Failed to parse headers' in record.msg and \
                     pool._absolute_url('/') == record.args[0]:
-                return
+                if unparsed_data_check is None or unparsed_data_check in record.getMessage():
+                    return
         self.fail('Missing log about unparsed headers')
 
     def test_header_without_name(self):
         self._test_broken_header_parsing([
-            b': Value\r\n',
-            b'Another: Header\r\n',
+            b': Value',
+            b'Another: Header',
         ])
 
     def test_header_without_name_or_value(self):
         self._test_broken_header_parsing([
-            b':\r\n',
-            b'Another: Header\r\n',
+            b':',
+            b'Another: Header',
         ])
 
     def test_header_without_colon_or_value(self):
         self._test_broken_header_parsing([
             b'Broken Header',
             b'Another: Header',
-        ])
+        ], 'Broken Header')
+
+
+class TestHeaderParsingContentType(SocketDummyServerTestCase):
+
+    def _test_okay_header_parsing(self, header):
+        self.start_response_handler((
+           b'HTTP/1.1 200 OK\r\n'
+           b'Content-Length: 0\r\n'
+           ) + header + b'\r\n\r\n'
+        )
+
+        pool = HTTPConnectionPool(self.host, self.port, retries=False)
+        self.addCleanup(pool.close)
+
+        with LogRecorder() as logs:
+            pool.request('GET', '/')
+
+        for record in logs:
+            assert 'Failed to parse headers' not in record.msg
+
+    def test_header_text_plain(self):
+        self._test_okay_header_parsing(b'Content-type: text/plain')
+
+    def test_header_message_rfc822(self):
+        self._test_okay_header_parsing(b'Content-type: message/rfc822')
 
 
 class TestHEAD(SocketDummyServerTestCase):