Repository: urllib3/urllib3

Issue:
duplicate/multiple headers are lost
When the server returns multiple headers of the same type, duplicated headers are lost. Example:

WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM

Results in only NTLM in the www-authenticate header.



Discussion:
Seems to be a result of https://github.com/shazow/urllib3/pull/544#issuecomment-73410346

I have tests broken in my apps as well but didn't have time to debug. For now I froze urllib3 version to 1.10.1. I might have time to take a look on it later today.

Arg, we need some better tests for this.

Hopefully somebody (@ml31415?) has a chance to take a look at this soon, otherwise I'll roll back #544 and push a hotfix release.

@shazow <3

@lukasa fwiw, I want to hold off on a 2.5.5 (requests) release until this is sorted out

@sigmavirus24 Good call, we should aim to fix this sharpish then.

So looking at the HTTPHeaderDict, there's some real weirdness in its logic that I didn't catch before.

For example, its docstrings are fabrications. Here's the docstring for `HTTPHeaderDict.add()`:

``` python
        """Adds a (name, value) pair, doesn't overwrite the value if it already
        exists.

        >>> headers = HTTPHeaderDict(foo='bar')
        >>> headers.add('Foo', 'baz')
        >>> headers['foo']
        'bar, baz'
        """
```

Here's the code from my terminal:

``` python
>>> from urllib3._collections import HTTPHeaderDict
>>> d = HTTPHeaderDict(foo='bar')
>>> d.add('foo', 'baz')
>>> d['foo']
'baz'
```

The bigger problem is that it's based on a misunderstanding of the RFCs. There's a fixed list of header fields that are allowed to have multiple values in the code: `cookie`, `set-cookie`, and `set-cookie2`. The problem is that RFC 7230 section 3.2.2 reads:

>    A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below).

This frozenset includes the well known exceptions, but does not allow for the fact that any header defined as a comma-separated list (which is many of them) are _also_ allowed.

In the short term, I recommend backing out #544 because it's provably wrong. Given that I need one for `hyper` as well, it might be worth us commonalising a HTTP header dict that doesn't try to be too clever to use as a baseline, with a well-defined suite of unit tests that confirm its behaviour. At that point we could revisit the performance improvements in #544, guaranteeing that function isn't regressed.

In the defense of #544, that lying docstring is a relic from the previous implementation, though it's quite conspicuous that it is unchanged in the middle of a large set of changes.

Would be also good to remove (or at least hide) the 1.10.2 from PyPI.

I have a fix in #562 but I'm pretty sure the tests will fail. I also didn't add any new ones. I tested most of this by hand. Next step is to add actual tests and make sure they all pass

Comma separated values are eaten without problems from #544. For me it still looks like #544 works as specified, and it rather looks like the corresponding server implementations don't. I had asked somewhere in the other discussion, if I should add the frozenset to enforce the duplicate headers only to this well known ones, as it's indeed questionable, if the specs should be followed strictly, when some (many?) implementations don't follow it. Removing this check is not a big deal. In that case, any special headers or frozenset won't be necessary at all, it's a three lines change. #544 was carefully tweaked for spec compatibility and performance, not sure what's the idea of reverting most of it's logic with #562. Yes, the docstring never got fixed, though.

Concerning the change in `__getitem__` on multiple headers, I had tried to follow the previous behaviour as good as possible, still not throwing away information. Only returning duplicate headers on `getlist` would be fine to me, but I guess other complaints will follow.

Another note. Maybe it would be worth to discuss and specify a desired behaviour internally, before doing further hasty changes. It's not `urllib3` doing something wrong here. It should be clearly defined, when and where to ignore the spec, and why, and how to behave in that cases.

@ml31415 I'm not proposing to ignore the spec, I'm proposing that the HTTPHeaderDict doesn't behave correctly because it eats headers that appear twice. The following is a totally valid header list:

```
content-length: 300
server: h2o/1.0.2-alpha1
connection: keep-alive
www-authenticate: basic
www-authenticate: digest
```

When passed to the HTTPHeaderDict, the dict from #544 eats that pair of headers, providing only one, where the previous implementation would have merged them.

Agree that the failure was mine in code review, but we should definitely fix it now.

Agree with @Lukasa, this bug breaks authentication schemes when the server provides multiple authentication methods with multiple "WWW-Authenticate" headers.

Hmm, sorry for messing this up! Looks like it comes down to what the spec means with "entire field value for that header field"? So far, to me the field value was everything followed after the colon to a line end not followed by space. Thinking about it now, looks like I was horribly wrong with my interpretation. 

The fix for this would be, to simply remove the field name check in `add`, a three lines change, or to add all csl-allowed headers in the frozen set. Would the well known exceptions, probably containing cookies with commas, require any special handling? Probably not, comma-joining them on `__getitem__` might still be the best option, and `getlist` is there for help.


Resolving patch:
diff --git a/urllib3/_collections.py b/urllib3/_collections.py
--- a/urllib3/_collections.py
+++ b/urllib3/_collections.py
@@ -20,8 +20,6 @@ def __exit__(self, exc_type, exc_value, traceback):
 __all__ = ['RecentlyUsedContainer', 'HTTPHeaderDict']
 
 
-MULTIPLE_HEADERS_ALLOWED = frozenset(['cookie', 'set-cookie', 'set-cookie2'])
-
 _Null = object()
 
 
@@ -143,7 +141,10 @@ class HTTPHeaderDict(dict):
     def __init__(self, headers=None, **kwargs):
         dict.__init__(self)
         if headers is not None:
-            self.extend(headers)
+            if isinstance(headers, HTTPHeaderDict):
+                self._copy_from(headers)
+            else:
+                self.extend(headers)
         if kwargs:
             self.extend(kwargs)
 
@@ -223,11 +224,8 @@ def add(self, key, val):
                 vals.append(val)
             else:
                 # vals should be a tuple then, i.e. only one item so far
-                if key_lower in MULTIPLE_HEADERS_ALLOWED:
-                    # Need to convert the tuple to list for further extension
-                    _dict_setitem(self, key_lower, [vals[0], vals[1], val])
-                else:
-                    _dict_setitem(self, key_lower, new_vals)
+                # Need to convert the tuple to list for further extension
+                _dict_setitem(self, key_lower, [vals[0], vals[1], val])
 
     def extend(*args, **kwargs):
         """Generic import function for any type of header-like object.
@@ -276,14 +274,17 @@ def getlist(self, key):
     def __repr__(self):
         return "%s(%s)" % (type(self).__name__, dict(self.itermerged()))
 
-    def copy(self):
-        clone = type(self)()
-        for key in self:
-            val = _dict_getitem(self, key)
+    def _copy_from(self, other):
+        for key in other:
+            val = _dict_getitem(other, key)
             if isinstance(val, list):
                 # Don't need to convert tuples
                 val = list(val)
-            _dict_setitem(clone, key, val)
+            _dict_setitem(self, key, val)
+
+    def copy(self):
+        clone = type(self)()
+        clone._copy_from(self)
         return clone
 
     def iteritems(self):


Test patch:
diff --git a/test/test_collections.py b/test/test_collections.py
--- a/test/test_collections.py
+++ b/test/test_collections.py
@@ -144,35 +144,32 @@ def setUp(self):
         self.d.add('cookie', 'bar')
 
     def test_overwriting_with_setitem_replaces(self):
-        d = HTTPHeaderDict()
+        self.d['Cookie'] = 'foo'
+        self.assertEqual(self.d['cookie'], 'foo')
 
-        d['Cookie'] = 'foo'
-        self.assertEqual(d['cookie'], 'foo')
-
-        d['cookie'] = 'bar'
-        self.assertEqual(d['Cookie'], 'bar')
+        self.d['cookie'] = 'bar'
+        self.assertEqual(self.d['Cookie'], 'bar')
 
     def test_copy(self):
         h = self.d.copy()
         self.assertTrue(self.d is not h)
         self.assertEqual(self.d, h)
+        
+    def test_getlist_after_copy(self):
+        self.assertEqual(self.d.getlist('cookie'), HTTPHeaderDict(self.d).getlist('cookie'))
 
-    def test_add_multiple_allowed(self):
-        d = HTTPHeaderDict()
-        d['Cookie'] = 'foo'
-        d.add('cookie', 'bar')
-
-        self.assertEqual(d['cookie'], 'foo, bar')
-        self.assertEqual(d['Cookie'], 'foo, bar')
+    def test_add_well_known_multiheader(self):
+        self.d.add('COOKIE', 'asdf')
+        self.assertEqual(self.d.getlist('cookie'), ['foo', 'bar', 'asdf'])
+        self.assertEqual(self.d['cookie'], 'foo, bar, asdf')
 
-        d.add('cookie', 'asdf')
-        self.assertEqual(d['cookie'], 'foo, bar, asdf')
+    def test_add_comma_separated_multiheader(self):
+        self.d.add('bar', 'foo')
+        self.d.add('BAR', 'bar')
+        self.d.add('Bar', 'asdf')
+        self.assertEqual(self.d.getlist('bar'), ['foo', 'bar', 'asdf'])
+        self.assertEqual(self.d['bar'], 'foo, bar, asdf')
 
-    def test_add_multiple_not_allowed(self):
-        self.d.add('notmulti', 'should be overwritten on next add call')
-        self.d.add('notmulti', 'new val')
-        self.assertEqual(self.d['notmulti'], 'new val')
-        
     def test_extend(self):
         self.d.extend([('set-cookie', '100'), ('set-cookie', '200'), ('set-cookie', '300')])
         self.assertEqual(self.d['set-cookie'], '100, 200, 300')