Repository: urllib3/urllib3

Issue:
Multipart mixed responses generate warnings
If you read a multipart/mixed response, the connectionpool issues a couple of warnings about defects in the message. I'm not sure what the expected, correct behavior is, but these warnings seem spurious.

Stick this perl script somewhere:

```
#!/usr/bin/perl

print "Server: Some Server Name\r\n";
print "Content-Type: multipart/mixed; boundary=36eeb8c4e26d842a\r\n";
print "Content-Length: 178\r\n";
print "\r\n\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "7\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "9\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "11\r\n";
print "--36eeb8c4e26d842a--\r\n";
```

Read it with requests (naturally, you'll have to change the URI to wherever you put the script):

```
import requests, logging

logging.basicConfig(level=logging.WARNING)
logging.getLogger("requests").setLevel(logging.DEBUG)

headers = {'accept': "multipart/mixed"}
r = requests.get("http://localhost:8124/cgi-bin/mpm.pl", headers=headers)

print(r)
```

The following errors are displayed:

```
DEBUG:requests.packages.urllib3.connectionpool:"GET http://localhost:8124/cgi-bin/mpm.pl HTTP/1.1" 200 178
WARNING:requests.packages.urllib3.connectionpool:Failed to parse headers (url=http://localhost:8888/http://localhost:8124/cgi-bin/mpm.pl): [StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()], unparsed data: ''
Traceback (most recent call last):
  File "/home/ndw/.virtualenvs/pyapi/lib/python3.4/site-packages/requests-2.8.0-py3.4.egg/requests/packages/urllib3/connectionpool.py", line 390, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "/home/ndw/.virtualenvs/pyapi/lib/python3.4/site-packages/requests-2.8.0-py3.4.egg/requests/packages/urllib3/util/response.py", line 58, in assert_header_parsing
    raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
requests.packages.urllib3.exceptions.HeaderParsingError: [StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()], unparsed data: ''
```

It took me quite a while to work out that they were spurious (because in real life, the server side that is generating the multipart/mixed is more complicated!)

See also kennethreitz/requests#3001
## 



Discussion:
Actually I'm mistaken, nevermind. (Deleted comment)

Hey all, so I took a quick look at this by taking the text and shoving it into a text file (with a status-line) and serving it via netcat while requesting it with only plain `http.client`. I wasn't very familiar with [`assert_header_parsing`](https://github.com/shazow/urllib3/blob/c325f97b7e892cefe5676e129c35c9be5d8180cd/urllib3/util/response.py#L31) but the reason you're seeing this is because of http.client's parsing of the headers. Here's the code that brought me to this conclusion:

``` py
>>> conn = http.client.HTTPConnection('localhost', 3234)
>>> conn.request('GET', '/')
>>> conn.getresponse()
<http.client.HTTPResponse object at 0x11000e6a0>
>>> r = _
>>> r.headers
<http.client.HTTPMessage object at 0x11003b0b8>
>>> r.headers.defects
[StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()]
```

That said, this doesn't seem to be a problem for http.client or urllib3. This simply issues a log warning which might not be the right log level for this (since it isn't anything more than informational).

That said, @ndw, what would you consider to be an appropriate fix here?

Lowering the warning level so that they require an even "finer" logging level would be fine, I guess, if the conditions that are being reported really are harmless. They're noisy and confusing when the logging level is set to warning.

On the other hand, if I understood what the warnings meant, I might be equally content to amend my code so as not to raise them. However, AFAICT, there is a start boundary, so I don't know why it's "not found" and I couldn't find anything in the MIME RFC that seemed to correspond to a multipart invariancy.

I think this traces all the way back to the email/rfc822 parser, and how it's trying to track your header.  in the python source, I don't think it gets to [line 355](https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Lib/email/feedparser.py#L365), thus it raises the error.

I think the issue is [here](https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Lib/email/feedparser.py#L343). Since it sees something equivalent to a blank line, but not NeedMoreData before it sees your boundary, it raises the error.

To fix, if you can call it that, try adding your boundary before you print any blank lines on your perl script.

Is this issue still outstanding? 
As far as I know, yes.
Hello, I see some fixes, but in 0.77.3 I´m still having this issue, can you please say which files I must change and what to apply the fix?
Kind regards
@McVillano urllib3 never had a 0.77.3 release, I think you're confusing urllib3 and Home Assistant here.

Resolving patch:
diff --git a/src/urllib3/util/response.py b/src/urllib3/util/response.py
--- a/src/urllib3/util/response.py
+++ b/src/urllib3/util/response.py
@@ -1,4 +1,5 @@
 from __future__ import absolute_import
+from email.errors import StartBoundaryNotFoundDefect, MultipartInvariantViolationDefect
 from ..packages.six.moves import http_client as httplib
 
 from ..exceptions import HeaderParsingError
@@ -66,6 +67,25 @@ def assert_header_parsing(headers):
 
             if isinstance(payload, (bytes, str)):
                 unparsed_data = payload
+    if defects:
+        # httplib is assuming a response body is available
+        # when parsing headers even when httplib only sends
+        # header data to parse_headers() This results in
+        # defects on multipart responses in particular.
+        # See: https://github.com/urllib3/urllib3/issues/800
+
+        # So we ignore the following defects:
+        # - StartBoundaryNotFoundDefect:
+        #     The claimed start boundary was never found.
+        # - MultipartInvariantViolationDefect:
+        #     A message claimed to be a multipart but no subparts were found.
+        defects = [
+            defect
+            for defect in defects
+            if not isinstance(
+                defect, (StartBoundaryNotFoundDefect, MultipartInvariantViolationDefect)
+            )
+        ]
 
     if defects or unparsed_data:
         raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)


Test patch:
diff --git a/test/test_util.py b/test/test_util.py
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -748,6 +748,20 @@ def test_assert_header_parsing_throws_typeerror_with_non_headers(self, headers):
         with pytest.raises(TypeError):
             assert_header_parsing(headers)
 
+    @onlyPy3
+    def test_assert_header_parsing_no_error_on_multipart(self):
+        from http import client
+
+        header_msg = io.BytesIO()
+        header_msg.write(
+            b'Content-Type: multipart/encrypted;protocol="application/'
+            b'HTTP-SPNEGO-session-encrypted";boundary="Encrypted Boundary"'
+            b"\nServer: Microsoft-HTTPAPI/2.0\nDate: Fri, 16 Aug 2019 19:28:01 GMT"
+            b"\nContent-Length: 1895\n\n\n"
+        )
+        header_msg.seek(0)
+        assert_header_parsing(client.parse_headers(header_msg))
+
 
 class TestUtilSSL(object):
     """Test utils that use an SSL backend."""
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -1808,3 +1808,40 @@ def socket_handler(listener):
         ) as pool:
             pool.urlopen("GET", "/not_found", preload_content=False)
             assert pool.num_connections == 1
+
+
+class TestMultipartResponse(SocketDummyServerTestCase):
+    def test_multipart_assert_header_parsing_no_defects(self):
+        def socket_handler(listener):
+            for _ in range(2):
+                sock = listener.accept()[0]
+                while not sock.recv(65536).endswith(b"\r\n\r\n"):
+                    pass
+
+                sock.sendall(
+                    b"HTTP/1.1 404 Not Found\r\n"
+                    b"Server: example.com\r\n"
+                    b"Content-Type: multipart/mixed; boundary=36eeb8c4e26d842a\r\n"
+                    b"Content-Length: 73\r\n"
+                    b"\r\n"
+                    b"--36eeb8c4e26d842a\r\n"
+                    b"Content-Type: text/plain\r\n"
+                    b"\r\n"
+                    b"1\r\n"
+                    b"--36eeb8c4e26d842a--\r\n",
+                )
+                sock.close()
+
+        self._start_server(socket_handler)
+        from urllib3.connectionpool import log
+
+        with mock.patch.object(log, "warning") as log_warning:
+            with HTTPConnectionPool(self.host, self.port, timeout=3) as pool:
+                resp = pool.urlopen("GET", "/")
+                assert resp.status == 404
+                assert (
+                    resp.headers["content-type"]
+                    == "multipart/mixed; boundary=36eeb8c4e26d842a"
+                )
+                assert len(resp.data) == 73
+                log_warning.assert_not_called()