Repository: pallets/werkzeug

Issue:
Chunked requests still don't really work
(Updated, because I realised I had been attempting to fix this, so got confused by the original issue and the issue that made me give up)

I'm making a chunked request to an app that uses Werkzeug (current git master). get_input_stream ends up calling `LimitedStream(wsgi.input, min(content_length, max_content_length))`. `min(content_length, max_content_length)` will always be None if using chunked encoding.

In an attempt to make progress I tried using `LimitedStream(stream, content_length or max_content_length)`, but then I end up getting a `ClientDisconnected` exception, because we can't tell the difference between a client disconnecting and the chunked request simply being done, as `wsgi.input.read()` just returns a 0-length string in both cases.


Discussion:
This is more than just the simple bug with `min`. After fixing that, the problem is that doing `wsgi.input.read(max_content_length)` with Werkzeug blocks forever if there's less than `max_content_length` to read. Or using Gunicorn, which supports chunks by parsing and buffering the whole message, the length comes out to less than `max_content_length`, so `LimitedStream` thinks the client disconnected.

I tested this out on Django just to be sure we're not doing something weird, it sees an empty stream too.
The simplest way forward is to provide a middleware that sets `environ['wsgi.input_terminated']` and tell people to use it when using a server that supports chunked transfer.
Which wsgi currently supports `input_terminated`? After burning all day on debugging why the data is empty, then why stream is empty then realizing it's due to chunked stream encoding I now can't find anything that supports this :/
In addition to checking `environ.get('wsgi.input_terminated')` should we also check `environ.get('HTTP_TRANSFER_ENCODING') == 'chunked'` and in that case also return the unwrapped stream directly?  That fixes problems like a simple `curl -H 'Transfer-Encoding: chunked' ...` returning an empty stream.

Per [RFC2616](https://www.ietf.org/rfc/rfc2616.txt)

> 4.4.2
> If a Transfer-Encoding header field (section 14.41) is present and
>      has any value other than "identity", then the transfer-length is
>      defined by use of the "chunked" transfer-coding (section 3.6),
>      unless the message is terminated by closing the connection.

> 4.4.3
> If a Content-Length header field (section 14.13) is present, its
>      decimal value in OCTETs represents both the entity-length and the
>      transfer-length. The Content-Length header field MUST NOT be sent
>      if these two lengths are different (i.e., if a Transfer-Encoding
>      header field is present). If a message is received with both a
>      Transfer-Encoding header field and a Content-Length header field,
>      the latter MUST be ignored.

... noting that `Transfer-Encoding` of `chunked` demands that any `Content-Length` header be ignored, and so no stream-processing should be done based on it.  This is currently the case unless `wsgi.input_terminated` is set.

Should it really be necessary to set `wsgi.input_terminated` in order for normal chunked encoding e.g. of a 10 byte JSON payload to work?

Resolving patch:
diff --git a/werkzeug/formparser.py b/werkzeug/formparser.py
--- a/werkzeug/formparser.py
+++ b/werkzeug/formparser.py
@@ -11,8 +11,7 @@
 """
 import re
 import codecs
-from io import BytesIO
-from tempfile import TemporaryFile
+from tempfile import SpooledTemporaryFile
 from itertools import chain, repeat, tee
 from functools import update_wrapper
 
@@ -38,9 +37,7 @@
 def default_stream_factory(total_content_length, filename, content_type,
                            content_length=None):
     """The stream factory that is used per default."""
-    if total_content_length > 1024 * 500:
-        return TemporaryFile('wb+')
-    return BytesIO()
+    return SpooledTemporaryFile(max_size=1024 * 500, mode='wb+')
 
 
 def parse_form_data(environ, stream_factory=None, charset='utf-8',
diff --git a/werkzeug/serving.py b/werkzeug/serving.py
--- a/werkzeug/serving.py
+++ b/werkzeug/serving.py
@@ -37,6 +37,7 @@
 """
 from __future__ import with_statement
 
+import io
 import os
 import socket
 import sys
@@ -97,6 +98,59 @@ class ForkingMixIn(object):
 can_open_by_fd = not WIN and hasattr(socket, 'fromfd')
 
 
+class DechunkedInput(io.RawIOBase):
+    """An input stream that handles Transfer-Encoding 'chunked'"""
+
+    def __init__(self, rfile):
+        self._rfile = rfile
+        self._done = False
+        self._len = 0
+
+    def readable(self):
+        return True
+
+    def read_chunk_len(self):
+        try:
+            line = self._rfile.readline().decode('latin1')
+            _len = int(line.strip(), 16)
+        except ValueError:
+            raise IOError('Invalid chunk header')
+        if _len < 0:
+            raise IOError('Negative chunk length not allowed')
+        return _len
+
+    def readinto(self, buf):
+        read = 0
+        while not self._done and read < len(buf):
+            if self._len == 0:
+                # This is the first chunk or we fully consumed the previous
+                # one. Read the next length of the next chunk
+                self._len = self.read_chunk_len()
+
+            if self._len == 0:
+                # Found the final chunk of size 0. The stream is now exhausted,
+                # but there is still a final newline that should be consumed
+                self._done = True
+
+            if self._len > 0:
+                # There is data (left) in this chunk, so append it to the
+                # buffer. If this operation fully consumes the chunk, this will
+                # reset self._len to 0.
+                n = min(len(buf), self._len)
+                buf[read:read + n] = self._rfile.read(n)
+                self._len -= n
+                read += n
+
+            if self._len == 0:
+                # Skip the terminating newline of a chunk that has been fully
+                # consumed. This also applies to the 0-sized final chunk
+                terminator = self._rfile.readline()
+                if terminator not in (b'\n', b'\r\n', b'\r'):
+                    raise IOError('Missing chunk terminating newline')
+
+        return read
+
+
 class WSGIRequestHandler(BaseHTTPRequestHandler, object):
 
     """A request handler that implements WSGI dispatching."""
@@ -141,6 +195,10 @@ def shutdown_server():
                 key = 'HTTP_' + key
             environ[key] = value
 
+        if environ.get('HTTP_TRANSFER_ENCODING', '').strip().lower() == 'chunked':
+            environ['wsgi.input_terminated'] = True
+            environ['wsgi.input'] = DechunkedInput(environ['wsgi.input'])
+
         if request_url.scheme and request_url.netloc:
             environ['HTTP_HOST'] = request_url.netloc
 
diff --git a/werkzeug/wsgi.py b/werkzeug/wsgi.py
--- a/werkzeug/wsgi.py
+++ b/werkzeug/wsgi.py
@@ -166,12 +166,16 @@ def get_host(environ, trusted_hosts=None):
 
 def get_content_length(environ):
     """Returns the content length from the WSGI environment as
-    integer. If it's not available ``None`` is returned.
+    integer. If it's not available or chunked transfer encoding is used,
+    ``None`` is returned.
 
     .. versionadded:: 0.9
 
     :param environ: the WSGI environ to fetch the content length from.
     """
+    if environ.get('HTTP_TRANSFER_ENCODING', '') == 'chunked':
+        return None
+
     content_length = environ.get('CONTENT_LENGTH')
     if content_length is not None:
         try:


Test patch:
diff --git a/tests/res/chunked.txt b/tests/res/chunked.txt
new file mode 100644
--- /dev/null
+++ b/tests/res/chunked.txt
@@ -0,0 +1,25 @@
+94
+----------------------------898239224156930639461866
+Content-Disposition: form-data; name="file"; filename="test.txt"
+Content-Type: text/plain
+
+
+f
+This is a test
+
+2
+
+
+65
+----------------------------898239224156930639461866
+Content-Disposition: form-data; name="type"
+
+
+a
+text/plain
+3a
+
+----------------------------898239224156930639461866--
+
+0
+
diff --git a/tests/test_serving.py b/tests/test_serving.py
--- a/tests/test_serving.py
+++ b/tests/test_serving.py
@@ -10,6 +10,7 @@
 """
 import os
 import ssl
+import sys
 import subprocess
 import textwrap
 
@@ -294,3 +295,89 @@ def app(environ, start_response):
         serving.run_simple(hostname='localhost', port='5001',
                            application=app, use_reloader=False)
     assert 'port must be an integer' in str(excinfo.value)
+
+
+def test_chunked_encoding(dev_server):
+    server = dev_server(r'''
+    from werkzeug.wrappers import Request
+    def app(environ, start_response):
+        assert environ['HTTP_TRANSFER_ENCODING'] == 'chunked'
+        assert environ.get('wsgi.input_terminated', False)
+        request = Request(environ)
+        assert request.mimetype == 'multipart/form-data'
+        assert request.files['file'].read() == b'This is a test\n'
+        assert request.form['type'] == 'text/plain'
+        start_response('200 OK', [('Content-Type', 'text/plain')])
+        return [b'YES']
+    ''')
+
+    testfile = os.path.join(os.path.dirname(__file__), 'res', 'chunked.txt')
+
+    if sys.version_info[0] == 2:
+        from httplib import HTTPConnection
+    else:
+        from http.client import HTTPConnection
+
+    conn = HTTPConnection('127.0.0.1', server.port)
+    conn.connect()
+    conn.putrequest('POST', '/', skip_host=1, skip_accept_encoding=1)
+    conn.putheader('Accept', 'text/plain')
+    conn.putheader('Transfer-Encoding', 'chunked')
+    conn.putheader(
+        'Content-Type',
+        'multipart/form-data; boundary='
+        '--------------------------898239224156930639461866')
+    conn.endheaders()
+
+    with open(testfile, 'rb') as f:
+        conn.send(f.read())
+
+    res = conn.getresponse()
+    assert res.status == 200
+    assert res.read() == b'YES'
+
+    conn.close()
+
+
+def test_chunked_encoding_with_content_length(dev_server):
+    server = dev_server(r'''
+    from werkzeug.wrappers import Request
+    def app(environ, start_response):
+        assert environ['HTTP_TRANSFER_ENCODING'] == 'chunked'
+        assert environ.get('wsgi.input_terminated', False)
+        request = Request(environ)
+        assert request.mimetype == 'multipart/form-data'
+        assert request.files['file'].read() == b'This is a test\n'
+        assert request.form['type'] == 'text/plain'
+        start_response('200 OK', [('Content-Type', 'text/plain')])
+        return [b'YES']
+    ''')
+
+    testfile = os.path.join(os.path.dirname(__file__), 'res', 'chunked.txt')
+
+    if sys.version_info[0] == 2:
+        from httplib import HTTPConnection
+    else:
+        from http.client import HTTPConnection
+
+    conn = HTTPConnection('127.0.0.1', server.port)
+    conn.connect()
+    conn.putrequest('POST', '/', skip_host=1, skip_accept_encoding=1)
+    conn.putheader('Accept', 'text/plain')
+    conn.putheader('Transfer-Encoding', 'chunked')
+    # Content-Length is invalid for chunked, but some libraries might send it
+    conn.putheader('Content-Length', '372')
+    conn.putheader(
+        'Content-Type',
+        'multipart/form-data; boundary='
+        '--------------------------898239224156930639461866')
+    conn.endheaders()
+
+    with open(testfile, 'rb') as f:
+        conn.send(f.read())
+
+    res = conn.getresponse()
+    assert res.status == 200
+    assert res.read() == b'YES'
+
+    conn.close()