Repository: pallets/werkzeug

Issue:
Performance issue for some data handling in MultipartDecoder
When using werkzeug handling file upload, werkzeug use FormDataParser and MultipartDecoder, in here: https://github.com/pallets/werkzeug/blob/edef71c243e1e1396092f7b5f82ddad6c6f766cf/src/werkzeug/sansio/multipart.py#L187 , if for some part of data, the var data_length is always 0, that will always generate a 'werkzeug.sansio.multipart.NeedData', so the source code will run  self.buffer.find(b"--",...) ( https://github.com/pallets/werkzeug/blob/edef71c243e1e1396092f7b5f82ddad6c6f766cf/src/werkzeug/sansio/multipart.py#L182 ) again and again, which will take longer and longer time(I think it is O(n²)), in the end the upload of file will take much longer time than other most of the files.

<!--
Describe how to replicate the bug.

Include a minimal reproducible example that demonstrates the bug.
Include the full traceback if there was an exception.
-->
Generate a file, which will having a b'\r' in a bound of MultipartDecoder.buffer, and all the data after there is no "\r".
Example of this kind of file:
```
def test52():
    with open("uploadfile.dat","wb") as f:
        def _gen():
            yield b"\r"
            for i in range(30000000):
                yield b"1234567890"
        for i in _gen():
            f.write(i)

if __name__ == '__main__':
    test52()
```
Then upload this file with werkzeug: werkzeug.formparser.parse_form_data() , it will reproduce the issue
<!--
Describe the expected behavior that should have happened but didn't.
-->
Need to handle file quickly, but actually very slow

Environment:

- Python version:3.6.5
- Werkzeug version: latest: edef71c243e1e1396092f7b5f82ddad6c6f766cf

There is a workaround, here( https://github.com/pallets/werkzeug/blob/edef71c243e1e1396092f7b5f82ddad6c6f766cf/src/werkzeug/formparser.py#L348 ) ,  buffer_size: int = 64 * 1024, that's too small, to handle several hundrers of MBytes of data, that will run too many times, if this buffer_size can be given by user, for example user give: 16 * 1024 * 1024, that will reduce the time a lot.



Discussion:
@pgjones could you take a look at this? Need to determine if this is actually an issue (I thought we addressed this with the new decoder). Maybe we can just bump up the buffer size. I don't want to make this configurable.
I think I'll need some more information here. The parser will still consume the buffer whilst looking for the next boundary, so I don't think there is any n^2 growth. I'm also unsure if this applies as the example given is invalid multipart data and hence should result in a failed parse.
Because every time parser search for the next boundary, it need to traverse all the data, the traverse time is decided by the data length, suppose buffer size is n, the traverse time is about to connec with: n+2n+3n+4n+5n... mn = (1+2+3...+m)n = (m * (m + 1) / 2)*n, so it is m^2 growth.


Resolving patch:
diff --git a/src/werkzeug/sansio/multipart.py b/src/werkzeug/sansio/multipart.py
--- a/src/werkzeug/sansio/multipart.py
+++ b/src/werkzeug/sansio/multipart.py
@@ -70,6 +70,10 @@ class State(Enum):
 # Header values can be continued via a space or tab after the linebreak, as
 # per RFC2231
 HEADER_CONTINUATION_RE = re.compile(b"%s[ \t]" % LINE_BREAK, re.MULTILINE)
+# This must be long enough to contain any line breaks plus any
+# additional boundary markers (--) such that they will be found in a
+# subsequent search
+SEARCH_EXTRA_LENGTH = 8
 
 
 class MultipartDecoder:
@@ -113,6 +117,7 @@ def __init__(
             % (LINE_BREAK, re.escape(boundary), LINE_BREAK, LINE_BREAK),
             re.MULTILINE,
         )
+        self._search_position = 0
 
     def last_newline(self) -> int:
         try:
@@ -141,7 +146,7 @@ def next_event(self) -> Event:
         event: Event = NEED_DATA
 
         if self.state == State.PREAMBLE:
-            match = self.preamble_re.search(self.buffer)
+            match = self.preamble_re.search(self.buffer, self._search_position)
             if match is not None:
                 if match.group(1).startswith(b"--"):
                     self.state = State.EPILOGUE
@@ -150,9 +155,17 @@ def next_event(self) -> Event:
                 data = bytes(self.buffer[: match.start()])
                 del self.buffer[: match.end()]
                 event = Preamble(data=data)
+                self._search_position = 0
+            else:
+                # Update the search start position to be equal to the
+                # current buffer length (already searched) minus a
+                # safe buffer for part of the search target.
+                self._search_position = max(
+                    0, len(self.buffer) - len(self.boundary) - SEARCH_EXTRA_LENGTH
+                )
 
         elif self.state == State.PART:
-            match = BLANK_LINE_RE.search(self.buffer)
+            match = BLANK_LINE_RE.search(self.buffer, self._search_position)
             if match is not None:
                 headers = self._parse_headers(self.buffer[: match.start()])
                 del self.buffer[: match.end()]
@@ -177,6 +190,12 @@ def next_event(self) -> Event:
                         name=name,
                     )
                 self.state = State.DATA
+                self._search_position = 0
+            else:
+                # Update the search start position to be equal to the
+                # current buffer length (already searched) minus a
+                # safe buffer for part of the search target.
+                self._search_position = max(0, len(self.buffer) - SEARCH_EXTRA_LENGTH)
 
         elif self.state == State.DATA:
             if self.buffer.find(b"--" + self.boundary) == -1:


Test patch: