Repository: pallets/werkzeug

Issue:
POST multipart upload out of memory
Hi, I have an issue with OOM while using multipart upload of huge binary files. This is due to the 
def make_line_iter(stream, limit=None, buffer_size=10 \* 1024):
code in werkzeug/werkzeug/wsgi.py.

Steps to reproduce:

Server  side code:

```
@app.route('/', methods=['POST'])
def upload_file():
    file = request.files['file']
    return '''OK'''
```

client side code (using requests-toolbelt)

```
from requests_toolbelt import MultipartEncoder
import requests


m = MultipartEncoder(
    fields = {'file': ( 'filename', open( '/var/tmp/sasha_test/input.bin' ), 'application/octet-stream' )}
    )

r = requests.post( 'http://localhost:5000/', data = m,
                  headers = {'Content-Type': m.content_type})

print "Done!!! %s" % r
```

input bin is a large binary file created like this:

```
dd if=/dev/zero of=input.bin bs=1MiB count=1K
```

What happens is that the code searches for teh "\r\n" in order to find multipart-part "terminator". It saves in memory 2 last lines.
This causes OOM while uploading large files. 

Thanks.



Discussion:
Was this issue fixed? I cannot reproduce the problem with the steps above

This issue is still not fixed. I am facing the same problem.

Hi,

I have faced the same problem.
I was uploading [`DBF`](http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUC) files which do not contain new-line characters with `Content-Type: multipart/form-data`.

### Steps to reproduce

1) Create a dummy file of a size bigger than your `$WEBSERVER` memory. E.g. For 156 MiB:

```
fallocate -l 156M dummy.bin
```

2) Upload the file:

```
curl -X POST -F file=@dummy.bin http://$WEBSERVER
```

### Error

```
Traceback (most recent call last):
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/app/view/layer.py", line 81, in post
    notification_endpoint = request.form.get('notification_endpoint')
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/local.py", line 338, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/utils.py", line 71, in __get__
    value = self.func(obj)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 488, in form
    self._load_form_data()
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/flask/wrappers.py", line 165, in _load_    value = self.func(obj)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 488, in form
    self._load_form_data()
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/flask/wrappers.py", line 165, in _load_form_data
    RequestBase._load_form_data(self)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 360, in _load_form_data
    mimetype, content_length, options)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 193, in parse
    content_length, options)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 99, in wrapper
    return f(self, stream, *args, **kwargs)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 210, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 520, in parse
    return self.cls(form), self.cls(files)
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/datastructures.py", line 373, in __init__
    for key, value in mapping or ():
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 518, in <genexpr>
    form = (p[1] for p in formstream if p[0] == 'form')
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 477, in parse_parts
    for ellt, ell in self.parse_lines(file, boundary, content_length):
  File "/root/code/gis/uwsgi_emperor/code/gis-api/apps/layers/.pyenv/local/lib/python2.7/site-packages/werkzeug/formparser.py", line 458, in parse_lines
    yield _cont, line[:cutoff]
MemoryError
```

I was wondering why the `buffer` ([source code](https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/wsgi.py#L826)) is needed and not discarded.

Best regards,
Robert

buffer contains the last line, and is used to yield the last unfinished line,

if you take a look at the loop, you will see, that after a sucessfull yield of a line buffer is replaced

for propper error proof multipart mime parsing a more complex parser is needed,
as to allow skipping non-boundary lines that are part of a nested multipart message format thats not text

This bug is still here, and is blocking for our application (we want to accept large video files). Here's a minimal example:

``` python
#!/usr/bin/env python

from werkzeug.wrappers import Request, Response


@Request.application
def application(request):
    _ = request.files['file']
    return Response('Hello World!')


if __name__ == '__main__':
    from werkzeug.serving import run_simple
    import resource

    limit_b = 256 * 1024 ** 2  # Allow only 256 MiB, for this demo.
    for rsrc in (resource.RLIMIT_AS, resource.RLIMIT_DATA, resource.RLIMIT_RSS):
        resource.setrlimit(rsrc, (limit_b, limit_b))

    run_simple('localhost', 4000, application)
```

Start the Werkzeug script, then run these shell commands:

``` bash
$ truncate -s 128M file-128M
$ curl -v --form "file=@file-128M" http://localhost:4000/
```

This crashes the server as follows:

```
 * Running on http://localhost:4000/ (Press CTRL+C to quit)
127.0.0.1 - - [17/May/2016 12:59:30] "POST / HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/serving.py", line 193, in run_wsgi
    execute(self.server.app)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/serving.py", line 181, in execute
    application_iter = app(environ, start_response)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/wrappers.py", line 291, in application
    return f(*args[:-2] + (request,))(*args[-2:])
  File "/home/sybren/workspace/cloud/pillar/upload_memory_bug.py", line 8, in application
    _ = request.files['file']
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/utils.py", line 73, in __get__
    value = self.func(obj)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/wrappers.py", line 520, in files
    self._load_form_data()
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/wrappers.py", line 361, in _load_form_data
    mimetype, content_length, options)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 195, in parse
    content_length, options)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 100, in wrapper
    return f(self, stream, *args, **kwargs)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 212, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 522, in parse
    return self.cls(form), self.cls(files)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/datastructures.py", line 382, in __init__
    for key, value in mapping or ():
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 520, in <genexpr>
    form = (p[1] for p in formstream if p[0] == 'form')
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 479, in parse_parts
    for ellt, ell in self.parse_lines(file, boundary, content_length):
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 401, in parse_lines
    headers = parse_multipart_headers(iterator)
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/formparser.py", line 259, in parse_multipart_headers
    for line in iterable:
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/wsgi.py", line 846, in make_line_iter
    for item in _iter_basic_lines():
  File "/home/sybren/.virtualenvs/pillar/lib/python2.7/site-packages/werkzeug/wsgi.py", line 837, in _iter_basic_lines
    yield _join(new_buf)
MemoryError
```

As you can see, we only upload a 128 MiB file, while allowing Python 256 MiB RAM. This amount of memory should be plenty to handle streaming the file to a temporary file on disk.

@mitsuhiko do you have any idea how to fix this?


Resolving patch:
diff --git a/werkzeug/formparser.py b/werkzeug/formparser.py
--- a/werkzeug/formparser.py
+++ b/werkzeug/formparser.py
@@ -372,7 +372,7 @@ def validate_boundary(self, boundary):
             # the assert is skipped.
             self.fail('Boundary longer than buffer size')
 
-    def parse_lines(self, file, boundary, content_length):
+    def parse_lines(self, file, boundary, content_length, cap_at_buffer=True):
         """Generate parts of
         ``('begin_form', (headers, name))``
         ``('begin_file', (headers, name, filename))``
@@ -387,7 +387,8 @@ def parse_lines(self, file, boundary, content_length):
         last_part = next_part + b'--'
 
         iterator = chain(make_line_iter(file, limit=content_length,
-                                        buffer_size=self.buffer_size),
+                                        buffer_size=self.buffer_size,
+                                        cap_at_buffer=cap_at_buffer),
                          _empty_string_iter)
 
         terminator = self._find_terminator(iterator)
diff --git a/werkzeug/wsgi.py b/werkzeug/wsgi.py
--- a/werkzeug/wsgi.py
+++ b/werkzeug/wsgi.py
@@ -784,7 +784,8 @@ def _make_chunk_iter(stream, limit, buffer_size):
         yield item
 
 
-def make_line_iter(stream, limit=None, buffer_size=10 * 1024):
+def make_line_iter(stream, limit=None, buffer_size=10 * 1024,
+                   cap_at_buffer=False):
     """Safely iterates line-based over an input stream.  If the input stream
     is not a :class:`LimitedStream` the `limit` parameter is mandatory.
 
@@ -808,6 +809,12 @@ def make_line_iter(stream, limit=None, buffer_size=10 * 1024):
                   content length.  Not necessary if the `stream`
                   is a :class:`LimitedStream`.
     :param buffer_size: The optional buffer size.
+    :param cap_at_buffer: if this is set chunks are split if they are longer
+                          than the buffer size.  Internally this is implemented
+                          that the buffer size might be exhausted by a factor
+                          of two however.
+    .. versionadded:: 0.11.10
+       added support for the `cap_at_buffer` parameter.
     """
     _iter = _make_chunk_iter(stream, limit, buffer_size)
 
@@ -831,11 +838,19 @@ def _iter_basic_lines():
             if not new_data:
                 break
             new_buf = []
+            buf_size = 0
             for item in chain(buffer, new_data.splitlines(True)):
                 new_buf.append(item)
+                buf_size += len(item)
                 if item and item[-1:] in crlf:
                     yield _join(new_buf)
                     new_buf = []
+                elif cap_at_buffer and buf_size >= buffer_size:
+                    rv = _join(new_buf)
+                    while len(rv) >= buffer_size:
+                        yield rv[:buffer_size]
+                        rv = rv[buffer_size:]
+                    new_buf = [rv]
             buffer = new_buf
         if buffer:
             yield _join(buffer)
@@ -854,7 +869,8 @@ def _iter_basic_lines():
         yield previous
 
 
-def make_chunk_iter(stream, separator, limit=None, buffer_size=10 * 1024):
+def make_chunk_iter(stream, separator, limit=None, buffer_size=10 * 1024,
+                    cap_at_buffer=False):
     """Works like :func:`make_line_iter` but accepts a separator
     which divides chunks.  If you want newline based processing
     you should use :func:`make_line_iter` instead as it
@@ -865,12 +881,19 @@ def make_chunk_iter(stream, separator, limit=None, buffer_size=10 * 1024):
     .. versionadded:: 0.9
        added support for iterators as input stream.
 
+    .. versionadded:: 0.11.10
+       added support for the `cap_at_buffer` parameter.
+
     :param stream: the stream or iterate to iterate over.
     :param separator: the separator that divides chunks.
     :param limit: the limit in bytes for the stream.  (Usually
                   content length.  Not necessary if the `stream`
                   is otherwise already limited).
     :param buffer_size: The optional buffer size.
+    :param cap_at_buffer: if this is set chunks are split if they are longer
+                          than the buffer size.  Internally this is implemented
+                          that the buffer size might be exhausted by a factor
+                          of two however.
     """
     _iter = _make_chunk_iter(stream, limit, buffer_size)
 
@@ -895,12 +918,24 @@ def make_chunk_iter(stream, separator, limit=None, buffer_size=10 * 1024):
             break
         chunks = _split(new_data)
         new_buf = []
+        buf_size = 0
         for item in chain(buffer, chunks):
             if item == separator:
                 yield _join(new_buf)
                 new_buf = []
+                buf_size = 0
             else:
+                buf_size += len(item)
                 new_buf.append(item)
+
+                if cap_at_buffer and buf_size >= buffer_size:
+                    rv = _join(new_buf)
+                    while len(rv) >= buffer_size:
+                        yield rv[:buffer_size]
+                        rv = rv[buffer_size:]
+                    new_buf = [rv]
+                    buf_size = len(rv)
+
         buffer = new_buf
     if buffer:
         yield _join(buffer)


Test patch:
diff --git a/tests/test_formparser.py b/tests/test_formparser.py
--- a/tests/test_formparser.py
+++ b/tests/test_formparser.py
@@ -154,7 +154,8 @@ def test_streaming_parse(self):
         class StreamMPP(formparser.MultiPartParser):
 
             def parse(self, file, boundary, content_length):
-                i = iter(self.parse_lines(file, boundary, content_length))
+                i = iter(self.parse_lines(file, boundary, content_length,
+                                          cap_at_buffer=False))
                 one = next(i)
                 two = next(i)
                 return self.cls(()), {'one': one, 'two': two}
diff --git a/tests/test_wsgi.py b/tests/test_wsgi.py
--- a/tests/test_wsgi.py
+++ b/tests/test_wsgi.py
@@ -381,6 +381,13 @@ def test_make_chunk_iter_bytes():
                                    buffer_size=4))
     assert rv == [b'abcdef', b'ghijkl', b'mnopqrstuvwxyz', b'ABCDEFGHIJK']
 
+    data = b'abcdefXghijklXmnopqrstuvwxyzXABCDEFGHIJK'
+    test_stream = BytesIO(data)
+    rv = list(wsgi.make_chunk_iter(test_stream, 'X', limit=len(data),
+                                   buffer_size=4, cap_at_buffer=True))
+    assert rv == [b'abcd', b'ef', b'ghij', b'kl', b'mnop', b'qrst', b'uvwx',
+                  b'yz', b'ABCD', b'EFGH', b'IJK']
+
 
 def test_lines_longer_buffer_size():
     data = '1234567890\n1234567890\n'
@@ -388,3 +395,11 @@ def test_lines_longer_buffer_size():
         lines = list(wsgi.make_line_iter(NativeStringIO(data), limit=len(data),
                                          buffer_size=4))
         assert lines == ['1234567890\n', '1234567890\n']
+
+
+def test_lines_longer_buffer_size_cap():
+    data = '1234567890\n1234567890\n'
+    for bufsize in range(1, 15):
+        lines = list(wsgi.make_line_iter(NativeStringIO(data), limit=len(data),
+                                         buffer_size=4, cap_at_buffer=True))
+        assert lines == ['1234', '5678', '90\n', '1234', '5678', '90\n']