Repository: pallets/werkzeug

Issue:
Werkzeug incorrectly handles multiline headers
According to [RFC 2616](https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html):

> HTTP/1.1 header field values can be folded onto multiple lines if the continuation line begins with a space or horizontal tab. All linear white space, including folding, has the same semantics as SP. A recipient MAY replace any linear white space with a single SP before interpreting the field value or forwarding the message downstream.

However, werkzeug does not accept header values with newlines, even if they abide by this convention.

```
>>> import werkzeug
>>> werkzeug.Headers().add('foo', 'bar\n baz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/local/lib/python2.7/site-packages/werkzeug/datastructures.py", line 1136, in add
    self._validate_value(_value)
  File ".../venv/local/lib/python2.7/site-packages/werkzeug/datastructures.py", line 1143, in _validate_value
    raise ValueError('Detected newline in header value.  This is '
ValueError: Detected newline in header value.  This is a potential security problem
```

Also, this restriction is applied inconsistently.
```
>>> werkzeug.Headers([('foo', 'bar\n baz')])
Headers([('foo', 'bar\n baz')])
```

I ran into this issue when trying to write test cases relating to nginx forwarding of client certificates via headers, so there is a real use case for supporting this properly.


Discussion:
HTTP headers do *not* allow newlines. The section you're quoting talks about folding, which ought to remove newlines from the unfolded value. This:

    """foo
         bar"""

should get unfolded to something like this:

    "foo bar"

Apart from that, Werkzeug doesn't parse HTTP at this level, this is the job of the WSGI server. The only reason it rejects newlines when parsing requests is to catch security issues.
This ticket was motivated by the real-life behavior of Flask in development mode behind an nginx proxy forwarding client certs.  With that setup, I observed newlines in the headers being passed to the application.  But when I attempted to replicate this in a unit test I got the above `ValueError` when building the request headers.

I did a bit more research on this issue and found the following:

* The HTTP spec ([RFC 2616](https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html)) states that newlines in headers MAY be replaced with a single space, not that they are guaranteed to be (see quote above).

* The CGI spec ([RFC 3875](https://tools.ietf.org/html/rfc3875)), which WSGI extends, requires newlines in request headers to be replaced:

    > Similarly, a header field that spans multiple lines MUST be merged onto a single line.

* The WSGI spec ([PEP 333](https://www.python.org/dev/peps/pep-0333/)) also prohibits newlines in response headers:

    > Each `header_value` must not include any control characters, including carriage returns or linefeeds, either embedded or at the end.

So this means that there are two bugs here:

* The werkzeug development server does not correctly normalize folded strings in request headers.

* The `Headers` object is inconsistent about accepting newlines in header values.  Newlines are accepted in the constructor, but not in the `add()` method.  It's probably better for the `Headers` object to remain permissive and perform the validation when actually constructing the WSGI response in `BaseResponse`.  That way forbidding newlines in the constructor won't break compatibility with possibly non-conforming WSGI servers (such as werkzeug's own development server).
Fair enough. There's also #1070 which plays into this.
Fairly sure this was fixed with the fix for #1070. If not, please let me know with a reproducible example.
@davidism As I mentioned in a previous comment, there are actually two bugs here, neither of which has been fixed on the current master branch.

---

The first bug involves how the werkzeug development server handles line-wrapped headers. It can be reproduced with the following server code, which prints the value of the `X-Example` header:
```
from werkzeug.serving import run_simple
from werkzeug.wrappers import Request, Response

def app(environ, start_response):
    request = Request(environ)
    print(repr(request.headers.get('X-Example')))
    response = Response(status=204)
    return response(environ, start_response)

run_simple('localhost', 8080, app)
```
We can then send it a request with a header spanning multiple lines:
```
GET / HTTP/1.1
Host: localhost:8080
Connection: close
X-Example: foo
 bar
```
**Expected server output:**
Header value is merged onto a single line
```
'foo bar'
```
**Actual server output (Python 2):**
```
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 57361)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/home/.../venv/local/lib/python2.7/site-packages/werkzeug/serving.py", line 320, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/home/.../venv/local/lib/python2.7/site-packages/werkzeug/serving.py", line 355, in handle_one_request
    return self.run_wsgi()
  File "/home/.../venv/local/lib/python2.7/site-packages/werkzeug/serving.py", line 238, in run_wsgi
    self.environ = environ = self.make_environ()
  File "/home/.../venv/local/lib/python2.7/site-packages/werkzeug/serving.py", line 217, in make_environ
    for key, value in self.get_header_items():
  File "/home/.../venv/local/lib/python2.7/site-packages/werkzeug/serving.py", line 441, in get_header_items
    key, value = header[0:-2].split(":", 1)
ValueError: need more than 1 value to unpack
----------------------------------------
```
**Actual server output (Python 3):**
Header value contains newline characters disallowed by WSGI spec
```
'foo\r\n bar'
```

---

The second bug has to do with how the `Headers` object handles newline values.
```
>>> from werkzeug import Headers
>>> h1 = Headers([('X-Example', 'foo\r\n bar')])
>>> h2 = Headers()
>>> h2.add('X-Example', 'foo\r\n bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.../venv3/lib/python3.6/site-packages/werkzeug/datastructures.py", line 1166, in add
    self._validate_value(_value)
  File "/home/.../venv3/lib/python3.6/site-packages/werkzeug/datastructures.py", line 1173, in _validate_value
    raise ValueError('Detected newline in header value.  This is '
ValueError: Detected newline in header value.  This is a potential security problem
```

**Expected result:**
Both operations should have the same result, whether a `ValueError` or success.

**Actual result:**
`Headers` constructor allows newlines while `Headers.add()` raises `ValueError`.
I was seeing the `ValueError` recently in one of our projects (Python 2.7 with Flask) which happens to sit behind an nginx proxy. I ended up reverting to `0.14.1` (overriding the version bundled with `Flask`) and my error went away. Thought I'd add since it seems the `0.15.x` branch introduced this problem (or potentially created a new problem with how it was handling request headers).

------UPDATE-------:

I tracked down what headers we were passing and one of them was a multi-line cert in pem format i.e.:
```
SSL_CLIENT_CERT: -----BEGIN CERTIFICATE-----
    MIIFHzCCAwegAwIBAgICEDgwDQYJKoZIhvcNAQELBQAwajELMAkGA1UEBhMCdXMx
    GDAWBgNVBAoMD3Uucy4gZ292ZXJubWVudDEPMA0GA1UECwwGcGVvcGxlMQwwCgYD
    VQQLDANkYWUxEDAOBgNVBAsMB2NoaW1lcmExEDAOBgNVBAMMB0ludGVyQ0EwHhcN
    MTcwODMxMTUwMzEwWhcNMjcwODI5MTUwMzEwWjBwMQswCQYDVQQGEwJVUzEYMBYG
    A1UECgwPVS5TLiBHb3Zlcm5tZW50MRAwDgYDVQQLDAdjaGltZXJhMQwwCgYDVQQL
    ....  
    -----END CERTIFICATE-----
```

Our nginx server was configured like so:
```
proxy_set_header SSL_CLIENT_CERT $ssl_client_cert;
```

We should probably be using `$ssl_client_escaped_cert` instead of `$ssl_client_cert` (since this is deprecated anyway). Not sure if that change will fix the header parsing problem though.

Hoping this helps anyone else that is running into this problem. It appears that `0.15.1` doesn't properly handle a multi-line header like a PEM cert at this time with Python 2.7.


This is a 2.7 issue again caused by the header processing of the development server. I am adding the ability to process folding of headers to the 2.7 compatibility code for request headers.

Resolving patch:
diff --git a/src/werkzeug/serving.py b/src/werkzeug/serving.py
--- a/src/werkzeug/serving.py
+++ b/src/werkzeug/serving.py
@@ -218,6 +218,7 @@ def shutdown_server():
 
         for key, value in self.get_header_items():
             key = key.upper().replace("-", "_")
+            value = value.replace("\r\n", "")
             if key not in ("CONTENT_TYPE", "CONTENT_LENGTH"):
                 key = "HTTP_" + key
                 if key in environ:
@@ -434,7 +435,8 @@ def get_header_items(self):
         This function provides Python 2/3 compatibility as related to the
         parsing of request headers. Python 2.7 is not compliant with
         RFC 3875 Section 4.1.18 which requires multiple values for headers
-        to be provided. This function will return a matching list regardless
+        to be provided or RFC 2616 which allows for folding of multi-line
+        headers. This function will return a matching list regardless
         of Python version. It can be removed once Python 2.7 support
         is dropped.
 
@@ -445,9 +447,26 @@ def get_header_items(self):
             # W3C RFC 2616 Section 4.2.
             items = []
             for header in self.headers.headers:
-                # Remove "\n\r" from the header and split on ":" to get
+                # Remove "\r\n" from the header and split on ":" to get
                 # the field name and value.
-                key, value = header[0:-2].split(":", 1)
+                try:
+                    key, value = header[0:-2].split(":", 1)
+                except ValueError:
+                    # If header could not be slit with : but starts with white
+                    # space and it follows an existing header, it's a folded
+                    # header.
+                    if header[0] in ("\t", " ") and items:
+                        # Pop off the last header
+                        key, value = items.pop()
+                        # Append the current header to the value of the last
+                        # header which will be placed back on the end of the
+                        # list
+                        value = value + header
+                    # Otherwise it's just a bad header and should error
+                    else:
+                        # Re-raise the value error
+                        raise
+
                 # Add the key and the value once stripped of leading
                 # white space. The specification allows for stripping
                 # trailing white space but the Python 3 code does not


Test patch:
diff --git a/tests/test_serving.py b/tests/test_serving.py
--- a/tests/test_serving.py
+++ b/tests/test_serving.py
@@ -510,6 +510,36 @@ def app(environ, start_response):
     conn.close()
 
 
+def test_multiline_header_folding_for_http_1_1(dev_server):
+    """
+    This is testing the provision of multi-line header folding per:
+     * RFC 2616 Section 2.2
+     * RFC 3875 Section 4.1.18
+    """
+    server = dev_server(
+        r"""
+        from werkzeug.wrappers import Response
+        def app(environ, start_response):
+            start_response('200 OK', [('Content-Type', 'text/plain')])
+            return [environ['HTTP_XYZ'].encode()]
+        """
+    )
+
+    conn = httplib.HTTPConnection("127.0.0.1", server.port)
+    conn.connect()
+    conn.putrequest("GET", "/")
+    conn.putheader("Accept", "text/plain")
+    conn.putheader("XYZ", "first-line", "second-line", "third-line")
+    conn.endheaders()
+    conn.send(b"")
+    res = conn.getresponse()
+
+    assert res.status == 200
+    assert res.read() == b"first-line\tsecond-line\tthird-line"
+
+    conn.close()
+
+
 def can_test_unix_socket():
     if not hasattr(socket, "AF_UNIX"):
         return False