Repository: pallets/werkzeug

Issue:
WSGIRequestHandler should be able to process multiple accept header with same field name.
From https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html section 4.2 last paragraph it should be possible to combine the multiple headers fields.

The way it is currently implemented it takes into account just one accept header https://github.com/pallets/werkzeug/blob/master/werkzeug/serving.py#L138

example of a request sent by docker client:
```
Host: localhost:5001
User-Agent: docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/json
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip
Connection: close
```
As you can see it sends multiple accept headers.

And this is what we get after headers get processed:

```
'HTTP_ACCEPT': 'application/vnd.docker.distribution.manifest.v2+json'
```

`Expected behaviour`

Other accept headers got lost. It should be possible to combine multiple header fields into one, so eventually we would contain a comma separated list of all accept headers.
Something like:
```
HTTP_ACCEPT': 'application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/json, application/vnd.docker.distribution.manifest.v1+prettyjws'
```

```$ ipython
Python 2.7.12 (default, Aug  9 2016, 15:48:18) 
Type "copyright", "credits" or "license" for more information.

In [1]: import werkzeug

In [2]: werkzeug.__version__
Out[2]: '0.11.11'

```



Discussion:
This problem has surfaced multiple times. IIRC the problem is that there's no general way to combine multiple headers into one (separation by comma is not always RFC-conforming)
Section 4.2 says this, which I think is very clear that multiple values must be comma-joinable when providing multiple headers of the same name.

"It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma."

Is there a case you have in mind where comma separation would not conform?
The `Host` header can neither appear multiple times nor can it contain a comma-separated list of hosts.

Also RFC 2616 is obsoleted by a set of new RFCs. RFC 7230 has the following to say:

>A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below).

So there are indeed cases where conversion from multiple header fields to comma-separated header value is acceptable, but that is not applicable to all headers. This actually makes a difference: If the client sends multiple host headers, would you pick the last one, or reject the entire request? I'm not sure if the difference matters in this particular example, and I'm not claiming that this bug is unfixable, but fixing it would also introduce a significant behavior change.
@untitaker i think the fix should at least apply to accept headers
Sure, a whitelist would solve this.

I just noticed that the piece of code you linked iterates through a dictionary which can't have multiple keys of the same name, so the additional headers are probably lost before that.
@untitaker that's not true, if i try to print self.headers it contains all the accept headers
@untitaker 
```    
        _logger.info('XXXXXXXXXXXXXX')
        _logger.info(self.headers)
        _logger.info(type(self.headers))
        for key, value in self.headers.items():
````

what i see:

```
XXXXXXXXXXXXXX
Host: localhost:5001
User-Agent: docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/json
Accept-Encoding: gzip
Connection: close

<type 'instance'>
```

Fair enough.
Ok, so let's see what other WSGI servers are doing (e.g. gunicorn) and copy their behavior. It would be the absolute worst if one had different behavior between dev and production environments, no matter if the behavior is technically correct or not.
As one data point, I tested django, accessing the ``HTTP_ACCEPT`` value here: https://docs.djangoproject.com/en/1.10/ref/request-response/#django.http.HttpRequest.META

Both when using Apache's mod_wsgi, and django's dev server, it successfully concatenated multiple Accept headers into a comma-separated list.
If `mod_wsgi` does this when using Django, it should behave similarly for Flask/werkzeug. Is this assumption correct?
Yes, it should. Actually headers are concatenated by the Apache at the end of [the parsing step](https://github.com/apache/httpd/blob/31b7b8cb10729126c11c19a25563648a184e011c/server/protocol.c#L1180-L1183).

This behaviour conforms to [RFC3875 (CGI v1.1)](https://tools.ietf.org/html/rfc3875#section-4.1.18):
> If multiple header fields with the same field-name are received then the server MUST rewrite them as a single value having the same semantics.

Thus it conforms to [PEP 333](https://www.python.org/dev/peps/pep-0333/#environ-variables):
> The *environ* dictionary is required to contain these CGI environment variables, as defined by the Common Gateway Interface specification.

So right now the same application under `run_simple` and `mod_wsgi`/`uwsgi` acts differently.
So I guess it's settled. Perhaps somebody could check how gunicorn behaves as well though?
I've been doing some digging, as I wan't to see all `X-Forwarded-For` headers.. but only get the last one.

From what I gather, the issue lies in how `rfc822.Message` parses headers into a `dict`, which is fed into the werkzeug `environ` (all headers are present in their raw form, however).

My point is, I would also like to see this issue addressed, and for more headers than merely the `Accept` header ;)


Resolving patch:
diff --git a/werkzeug/serving.py b/werkzeug/serving.py
--- a/werkzeug/serving.py
+++ b/werkzeug/serving.py
@@ -189,10 +189,12 @@ def shutdown_server():
             'SERVER_PROTOCOL':      self.request_version
         }
 
-        for key, value in self.headers.items():
+        for key, value in self.get_header_items():
             key = key.upper().replace('-', '_')
             if key not in ('CONTENT_TYPE', 'CONTENT_LENGTH'):
                 key = 'HTTP_' + key
+                if key in environ:
+                    value = "{},{}".format(environ[key], value)
             environ[key] = value
 
         if environ.get('HTTP_TRANSFER_ENCODING', '').strip().lower() == 'chunked':
@@ -383,6 +385,35 @@ def log(self, type, message, *args):
                                          self.log_date_time_string(),
                                          message % args))
 
+    def get_header_items(self):
+        """
+        Get an iterable list of key/value pairs representing headers.
+
+        This function provides Python 2/3 compatibility as related to the
+        parsing of request headers. Python 2.7 is not compliant with
+        RFC 3875 Section 4.1.18 which requires multiple values for headers
+        to be provided. This function will return a matching list regardless
+        of Python version. It can be removed once Python 2.7 support
+        is dropped.
+
+        :return: List of tuples containing header hey/value pairs
+        """
+        if PY2:
+            # For Python 2, process the headers manually according to W3C RFC 2616 Section 4.2
+            items = []
+            for header in self.headers.headers:
+                # Remove the \n\r from the header and split on the : to get the field name and value
+                key, value = header[0:-2].split(":", 1)
+                # Add the key and the value once stripped of leading white space. The specification
+                # allows for stripping trailing white space but the Python 3 code does not strip
+                # trailing white space. Therefore, trailing space will be left as is to match the
+                # Python 3 behavior
+                items.append((key, value.lstrip()))
+        else:
+            items = self.headers.items()
+
+        return items
+
 
 #: backwards compatible name if someone is subclassing it
 BaseRequestHandler = WSGIRequestHandler


Test patch:
diff --git a/tests/test_serving.py b/tests/test_serving.py
--- a/tests/test_serving.py
+++ b/tests/test_serving.py
@@ -436,3 +436,37 @@ def app(environ, start_response):
     assert res.read() == b'YES'
 
     conn.close()
+
+
+def test_multiple_headers_concatenated_per_rfc_3875_section_4_1_18(dev_server):
+    server = dev_server(r'''
+    from werkzeug.wrappers import Response
+    def app(environ, start_response):
+        start_response('200 OK', [('Content-Type', 'text/plain')])
+        return [environ['HTTP_XYZ'].encode()]
+    ''')
+
+    if sys.version_info[0] == 2:
+        from httplib import HTTPConnection
+    else:
+        from http.client import HTTPConnection
+
+    conn = HTTPConnection('127.0.0.1', server.port)
+    conn.connect()
+    conn.putrequest('GET', '/')
+    conn.putheader('Accept', 'text/plain')
+    conn.putheader('XYZ', ' a ')
+    conn.putheader('X-INGNORE-1', 'Some nonsense')
+    conn.putheader('XYZ', ' b')
+    conn.putheader('X-INGNORE-2', 'Some nonsense')
+    conn.putheader('XYZ', 'c ')
+    conn.putheader('X-INGNORE-3', 'Some nonsense')
+    conn.putheader('XYZ', 'd')
+    conn.endheaders()
+    conn.send(b'')
+    res = conn.getresponse()
+
+    assert res.status == 200
+    assert res.read() == b'a ,b,c ,d'
+
+    conn.close()