Repository: urllib3/urllib3

Issue:
IndexError when handle malformed  http response
I use urllib3 receive a http response like this,
```
HTTP/1.1 200 OK
	Content-Type: application/octet-stream
	Content-Length: 89606
	Content-Disposition: attachment; filename="MB-500Ap_2009-01-12.cfg"
	Connection: close

Brickcom-50xA
OperationSetting.locale=auto
HostName.name=cam
ModuleInfo.DIDO_module=1
ModuleInfo.PIR_module=0
ModuleInfo.WLED=0
SensorFPSSetting.fps=0
ModuleInfo.AUTOIRIS_module=0
ModuleInfo.IRCUT_module=0
ModuleInfo.IRLED_module=0
ModuleInfo.lightsensor=0
ModuleInfo.EXPOSURE_module=0
ModuleInfo.MDNS_module=0
ModuleInfo.PTZ_module=1
ModuleInfo.MSN_module=0
ModuleInfo.WIFI_module=0
ModuleInfo.watchDog_module=0
ModuleInfo.sdcard_module=1
ModuleInfo.usbstorage_module=0
ModuleInfo.sambamount_module=0
ModuleInfo.QoS=0
ModuleInfo.shutter_speed=0
ModuleInfo.discovery_internet=1
ModuleInfo.POE_module=
ModuleInfo.audio_record=1
```
it throws a IndexError,I print the traceback,
```
 req = http_get(url, auth=("admin", "admin"), timeout=timeout, verify=False)
  File "C:\Python27\lib\site-packages\requests\api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "C:\Python27\lib\site-packages\urllib3\connectionpool.py", line 617, in urlopen
    **response_kw)
  File "C:\Python27\lib\site-packages\urllib3\response.py", line 456, in from_httplib
    headers = HTTPHeaderDict.from_httplib(headers)
  File "C:\Python27\lib\site-packages\urllib3\_collections.py", line 312, in from_httplib
    key, value = headers[-1]
IndexError: list index out of range
```
how can I deal with this issue?


Discussion:
I'll see if I can replicate soon and figure out the right strategy. In the original, are those header fields spaced out with `\t`, or individual space characters?
the source response is 
```
'HTTP/1.1 200 OK\r\n\tContent-Type: application/octet-stream\r\n\tContent-Length: 49059\r\n\tContent-Disposition: attachment; filename="configfile.txt"\r\n\tConnection: close\r\n\r\n
```
Okay, so we're correctly dropping those "headers" because they're not actually headers (tab character is disallowed in header field names). This results in a response that has no headers, but we have a piece of logic that expects at least one. I'll need to dig in and see if there are any other places that would be unhappy with a headerless response and if so, what kind of effort would be involved with cleaning that up.

@sigmavirus24, thoughts? I don't think we want to try to figure out what the server means when it sends an invalid header; it's too ambiguous.
@haikuginger we're not correctly dropping them necessarily. We're expecting valid headers. The server is wrong and I think we should raise an InvalidHeader exception of sorts in this case, especially when we can't just try to add it onto the previous header.
In this instance I think the error is coming from httplib, so we may not be able to raise that error easily.
Okay, I'm just getting back to this. This is our code, not httplib, and what's happening is that we're interpreting a line that starts with `'\t'` or `' '` (space) as a line continuation for the last header - and then getting the last header with the `-1` index. It looks like this is obsoleted with RFC-7230, but user agents may still accept it by replacing the whole thing with a single space (not the current `'\r\n'` we currently implement).

I propose that we adjust the implementation such that...
1. Multiline header values are passed per RFC-7230 with a space inserted between lines in the value
2. Items that appear to be multiline headers (preceded by `'\r\n\t'` or `'\r\n '`) raise a specific exception unless preceded by at least one header.

@sigmavirus24, does that seem reasonable to you?

Note to @beruhan: this will not prevent an exception from happening for the response you've listed above, but will make that exception more explicit and helpful, so that you can catch it and take action accordingly.
@haikuginger yeah, 2 is what we're missing and definitely seems reasonable. 👍 

Resolving patch:
diff --git a/urllib3/_collections.py b/urllib3/_collections.py
--- a/urllib3/_collections.py
+++ b/urllib3/_collections.py
@@ -15,6 +15,7 @@ def __exit__(self, exc_type, exc_value, traceback):
     from collections import OrderedDict
 except ImportError:
     from .packages.ordered_dict import OrderedDict
+from .exceptions import InvalidHeader
 from .packages.six import iterkeys, itervalues, PY3
 
 
@@ -305,13 +306,22 @@ def from_httplib(cls, message):  # Python 2
         # python2.7 does not expose a proper API for exporting multiheaders
         # efficiently. This function re-reads raw lines from the message
         # object and extracts the multiheaders properly.
+        obs_fold_continued_leaders = (' ', '\t')
         headers = []
 
         for line in message.headers:
-            if line.startswith((' ', '\t')):
-                key, value = headers[-1]
-                headers[-1] = (key, value + '\r\n' + line.rstrip())
-                continue
+            if line.startswith(obs_fold_continued_leaders):
+                if not headers:
+                    # We received a header line that starts with OWS as described
+                    # in RFC-7230 S3.2.4. This indicates a multiline header, but
+                    # there exists no previous header to which we can attach it.
+                    raise InvalidHeader(
+                        'Header continuation with no previous header: %s' % line
+                    )
+                else:
+                    key, value = headers[-1]
+                    headers[-1] = (key, value + ' ' + line.strip())
+                    continue
 
             key, value = line.split(':', 1)
             headers.append((key, value.strip()))


Test patch:
diff --git a/test/test_collections.py b/test/test_collections.py
--- a/test/test_collections.py
+++ b/test/test_collections.py
@@ -4,6 +4,7 @@
 )
 import pytest
 
+from urllib3.exceptions import InvalidHeader
 from urllib3.packages import six
 xrange = six.moves.xrange
 
@@ -340,8 +341,8 @@ def test_from_httplib_py2(self):
 Content-Type: text/html; charset=windows-1251
 Connection: keep-alive
 X-Some-Multiline: asdf
- asdf
- asdf
+ asdf\t
+\t asdf
 Set-Cookie: bb_lastvisit=1348253375; expires=Sat, 21-Sep-2013 18:49:35 GMT; path=/
 Set-Cookie: bb_lastactivity=0; expires=Sat, 21-Sep-2013 18:49:35 GMT; path=/
 www-authenticate: asdf
@@ -356,6 +357,14 @@ def test_from_httplib_py2(self):
         assert len(cookies) == 2
         assert cookies[0].startswith("bb_lastvisit")
         assert cookies[1].startswith("bb_lastactivity")
-        assert d['x-some-multiline'].split() == ['asdf', 'asdf', 'asdf']
+        assert d['x-some-multiline'] == 'asdf asdf asdf'
         assert d['www-authenticate'] == 'asdf, bla'
         assert d.getlist('www-authenticate') == ['asdf', 'bla']
+        with_invalid_multiline = """\tthis-is-not-a-header: but it has a pretend value
+Authorization: Bearer 123
+
+"""
+        buffer = six.moves.StringIO(with_invalid_multiline.replace('\n', '\r\n'))
+        msg = six.moves.http_client.HTTPMessage(buffer)
+        with pytest.raises(InvalidHeader):
+            HTTPHeaderDict.from_httplib(msg)