Repository: netty/netty
Issue: HttpPostMultipartRequestDecoder performance regression

### Expected behavior

#10226 improves performance.

### Actual behavior

The decoder is now at least 100x slower than before iff it is offered a direct buffer.

If the decoder is offered a direct buffer it now creates a bunch of 64b direct buffers during parsing, where before it always used heap buffers.
Using a `PooledByteBufAllocator` that prefers direct buffers did not solve the issue.
Using a `(Un)PooledByteBufAllocator` that prefers heap buffers normalizes performance again.

Given that this issue was not listed in the announcement under important, I would think that this was not intentional.

### Steps to reproduce

Create a HttpPostMultipartRequestDecoder and offer around a 1000 chunks (64k); in 4.1.49 this runs in seconds, whereas in 4.1.50 it takes minutes.

### Minimal yet complete reproducer code (or URL to code)

We have [this](https://github.com/apache/flink/blob/e991de19337591cde444fbede010cb8bdc7f118f/flink-runtime/src/test/java/org/apache/flink/runtime/rest/FileUploadHandlerTest.java) test in Apache Flink that exhibits the problem.
I can try to write a minimal reproducer if requested; may take some time to figure out how to correctly create the requests :/

### Netty version

4.1.50

### JVM version (e.g. `java -version`)

8

### OS version (e.g. `uname -a`)

Windows 10, but we also saw it on ubuntu linux

Issue comments:
My 2 cents:
First change:
- Before, it was a `copy` which limits the max, 
- now is a `buf.alloc.buffer(...).writeByte(buf)`  => Pooled buffer using the original ByteBufAllocator from the `buf`

Second change, which is probably the one you point out:
- Before (2 places): `Unpooled.buffer(64)` (Unpooled implicit)
- After (same): `undecodedChunk.alloc().heapBuffer(64)`

This should use the ByteBufAllocator used for undecoded, which in turns should be (if I understand) the same allocator than original `buf` (so the one from IO) since either it is the same, or a new allocated one using the same allocator.

I could suggest to replace `undecodedChunk.alloc().heapBuffer(64)` by `undecodedChunk.alloc().buffer(64, 64)` (here there should not be any more than 64), but as you noticed that putting a `PooledByteBufAllocator` to direct is not correct, but heap seems ok, then I finally could suggest: `undecodedChunk.alloc().heapBuffer(64, 64)`, which is almost what we have there...

@normanmaurer Any suggestion since I do not have a clear mind on this... ?

@zentol I tried to reproduce using only Netty, but I can't get the numbers you've got.

Could you have a reproducer code with Netty only (almost a Junit or simple example)?
What kind of data are you sending (which forms and numbers)?
And to be sure, your tests shows that Direct buffers are not ok, while Heap buffers are ?
Thanks
@zentol Note that I used 4.1.51 and I can't reproduce the issue you raised... 
@zentol I tried another time, but the numbers I've got are almost the same whatever the allocation:
- PooledHeap, PooledDirect, PooledBuffer (from undecodedChunked
- UnpooledHeap, UnpooledDirect, UnpooledBuffer

I've tried both 1000 chunks for big data (64KB each) so 1000 decoded vars or 1000 chunks for small data (4KB each but 16 times so still chunks of 64KB each) so 16000 decoded vars, and the results are the same : 
- 4KB data size: roughly 230ms each test (16000 data) (highest - lowest = 5ms)
- 64KB data size: roughly 2900ms each test (1000 data), the difference being the data written on disk due to large size (> 16KB)  (highest - lowest = 310ms, but not so relevant since disk access is most of the issue as without disk access, there is no issue)
- 64KB data size: roughly 190ms each test (1000 data), the difference being the data not written on disk (memory only) (highest - lowest = 8ms)

So my conclusion until further information from you is that there is no issue there... Either I'm missing something, either it was fixed in 4.1.51...
@fredericBregier I did not test it with 4.1.51 due to #10425. I will check whether the issue persists in 4.1.52, and if so try to create a reproducer.
Ok, I think I got closer to the problem.

I wrote a reproducing test, which ran fast in 4.1.49 and slow in 4.1.50, but when I moved it into a separate project it suddenly ran fast either way.
I then noticed that the test we used so far also uses a `ResourceLeakDetector` set to `PARANOID`. If I set the level to `ADVANCED` it runs quickly with both versions, but `PARANOID` appears to slow things down way more in 4.1.50+.

Here's the test in question:

```
public class RegressionTest {

	@Rule
	public final TemporaryFolder temporaryFolder = new TemporaryFolder();

	static {
		ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
	}

	@Test
	public void testRegression() throws Exception {
		File file = temporaryFolder.newFile();
		try (RandomAccessFile rw = new RandomAccessFile(file, "rw")) {
			rw.setLength(1024 * 1024 * 5);
		}
		DefaultHttpDataFactory httpDataFactory = new DefaultHttpDataFactory(true);

		HttpRequest httpRequest = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "");
		HttpPostRequestEncoder bodyRequestEncoder = new HttpPostRequestEncoder(httpDataFactory, httpRequest, true);
		bodyRequestEncoder.addBodyFileUpload("file", file, "application/octet-stream", false);

		HttpRequest firstHttpRequest = bodyRequestEncoder.finalizeRequest();
		HttpPostRequestDecoder httpPostRequestDecoder = new HttpPostRequestDecoder(httpDataFactory, firstHttpRequest);

		ByteBufAllocator allocator = PooledByteBufAllocator.DEFAULT;
		for (HttpContent httpContent = bodyRequestEncoder.readChunk(allocator); httpContent != null; httpContent = bodyRequestEncoder.readChunk(allocator)) {
			httpPostRequestDecoder.offer(httpContent);
		}
	}
}
```
@zentol Effctively, with PARANOID level set, chunks are quite high consuming time, while other levels are quite quickier. On my tests (same than before, based on 4.1.52):  whatever the pooled/unpooled heap/direct buffers
- No detector: 225 to 230ms
- Simple: 229 to 233ms
- Advanced; 228 to 245ms
- Paranoid:  147.000ms
So yes, there is a hug gap using Paranoid (almost x600). Maybe you can stay in at most Advanced mode ?
@normanmaurer However I have a question for you, do you think this kind of gap is normal ?
I do know that Paranoid is not for Production, but maybe it is so huge that it could not be usable. Note however that I used Paranoid level for a long time is all my tests and did not found any issue about on my side.
Using Advanced would be workaround we would go for, but this runtime increase on Paranoid does not seem normal. It was perfectly fine to use Paranoid in earlier versions, and it would be unfortunate to weaken our tests now with never versions. :/
sorry for the late reply... The problem with "paranoid" is that it basically creates one stack trace per `ByteBuf` access which can be very expensive. So these kind of performance drops can happen.
@normanmaurer Thanks!
For me, it's fine as is.
If I'm correct, changing the number of items tracked through `io.netty.leakDetection.targetRecords` will not change every access to be checked but just limit the number of retained items ?

I believe that one could set specific values for `io.netty.leakDetection.samplingInterval` (default being 128 to a small value (still power of 2, such as 16 to reduce the PARANOID effect to a smaller part (If I'm correct, the main difference between ADVANCED and PARANOID is that this samplingInterval is ignored within PARANOID ?).

I checked using level = `ADVANCED` and `io.netty.leakDetection.samplingInterval` to: (using Pooled one)
- compared to `PARANOID`: 140.000ms whatever Direct or Heap
- 8 : 140.000ms Direct vs 4.000ms Heap, so almost the same as PARANOID, except using Heap
- 16 : 210ms whatever
- 128 (default) : 200ms whatever Direct or Heap
- 12 (*I know that the code said it should be a power of 2): 590ms Direct 420ms Heap
So maybe 12 could be a good tradeof ? (more trace but not as many as in PARANOID).

Again, on my side, I use PARANOID everywhere in my tests and I'm fine with it. It is more for the question of @zentol .

Note: I used this in my test bench code

    @BeforeClass
    public static void setDefaultAdvancedLevel() {
      System.setProperty("io.netty.leakDetection.samplingInterval", "12");
      ResourceLeakDetector.setLevel(Level.ADVANCED);
    }

Our problem isn't that PARANOID is slower than other modes, that's fine and expected, but that due to _some change that impacts leak detection_ our test is 100x slower in 4.1.50 than before.

I would just like to understand what has changed; if the leak detection was made slower in general, then that's fine, but I'm worried that something else has changed that indirectly causes leak detection to be slower in our specific case.
After merging #11001 I'm getting ~360-400ms for ADVANCED and ~460-770ms for PARANOID when running the test in https://github.com/netty/netty/issues/10508#issuecomment-692037524

So I think we can close this issue?
I hope so ;-)

Pull request review comments:
@fredericBregier can you explain why setting an upper limit improves things here ?
@normanmaurer 
I don't have any real clue, but it seems it does. That's why I'm asking you to look at this why. 

The change is huge on performances in PARANOID mode (using the example test given by the final user), about 10 times better... I test several times, and without it, the times are about 1000ms, while with it, it gives about 100ms.

Using my old test (a bit different, where a lot of items, about 6000, are passed and not only one file), the improvement is there also on PARANOID mode, but not that much (not 10 times better, comparing 146.000 old code vs 105.000 new code). 
There, the issue (and change) is on the `undecodedChunk.write(buf)`.
But if I change it to something like `wrappedBuffers(undecodedChunk, buf)`, the time is close to 1000ms again in PARANOID mode, but increases sensibly in other modes (500ms against 200ms without any change).


It was strange for me to see that comparing:
- `undecodedChunk = isLast? ... : buf.alloc.buffer(buf.readableSize(), buf.readableSize()).write(buf)` and `undecodedChunk = Unpooled.wrapped(undecodedChunk, buf.alloc.buffer(buf.readableSize(), buf.readableSize()).write(buf))`
- vs original ones (still in place at lines 338 and 340

gives impact performances (better for PARANOID, about 1000 times, but worst in other modes, about 2 times). I feel like it was the same in both cases (allocation and writing), but it seems not. Note that in current code, `undecodedChunk` in line 338 is still allocated using no upper bound.

The main "same root cause" seems to be "no upper bound for buffer" when using PARANOID mode, but in different ways.


I feel like this proposal is safe as it enhances a bit the general performances and takes down the PARANOID issue, at least for the user's example.
But I feel also that there is something else, since in my test (high number of items) this is really not enough.

So the reason I try to test it and produce at least this in order to get help.
@normanmaurer 

I added a fake commit to let you run the different cases at once:

This commit is only intend to show the result on tests using 3 cases:
- HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 1: original case
- HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 2: where only upper bound is modified
- HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 3: where write is replaced by wrapped and the last "no upper bounded" buffer is replaced too

Test to run is `testRegressionMultipleLevelLeakDetector`.

On my side results are:

Timer: DISABLED NotUsingDisk1 => 220.995585
**Timer: SIMPLE NotUsingDisk1 => 222.134248**
Timer: ADVANCED NotUsingDisk1 => 208.68989
**_Timer: PARANOID NotUsingDisk1 => 100044.293876_**
Timer: DISABLED UsingDisk1 => 707.480873
Timer: DISABLED UsingDisk1 => 518.695771
Timer: SIMPLE UsingDisk1 => 522.790849
Timer: ADVANCED UsingDisk1 => 585.759164
Timer: PARANOID UsingDisk1 => 105090.135276

Timer: DISABLED NotUsingDisk2 => 136.461289
**Timer: SIMPLE NotUsingDisk2 => 126.793194**
Timer: ADVANCED NotUsingDisk2 => 129.771282
**_Timer: PARANOID NotUsingDisk2 => 973.38734_**
Timer: DISABLED UsingDisk2 => 74.279745
Timer: DISABLED UsingDisk2 => 71.332363
Timer: SIMPLE UsingDisk2 => 114.811827
Timer: ADVANCED UsingDisk2 => 89.290387
Timer: PARANOID UsingDisk2 => 1155.278418

Timer: DISABLED NotUsingDisk3 => 1829.364849
**Timer: SIMPLE NotUsingDisk3 => 1949.512746**
Timer: ADVANCED NotUsingDisk3 => 2120.573799
**_Timer: PARANOID NotUsingDisk3 => 2073.506771_**
Timer: DISABLED UsingDisk3 => 1998.837595
Timer: DISABLED UsingDisk3 => 1952.342478
Timer: SIMPLE UsingDisk3 => 2099.951834
Timer: ADVANCED UsingDisk3 => 2058.028121
Timer: PARANOID UsingDisk3 => 2295.847557

Last fake commit:

This commit is only intend to show the result on tests using 3 cases:
    - HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 1: original case
    - HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 2: where only upper bound is modified
    - HttpPostMultipartRequestDecoder.TEST_TEMP_ITEM = 3: where temptative to reuse more the existing ByteBuf is done
    
Test to run is `testRegressionMultipleLevelLeakDetectorNoDisk` or `testHighNumberCheckLeakDetectorVersions`
    
Current results (stability is correct but numbers depend on host)

I've done various tests:
- using `wrapped` (or through `buf.alloc().compositeByteBuf()`): worst timers
- using fixed allocation (but high enough to fit,the necessary elements, just for test): not good timers (probably because the size is too huge for a lot of cases, and whatever, it is not acceptable to have fixed size, it was just for testing purpose)
- using various ways to discard read bytes: it gives a bit improvement (TEST_TEMP_ITEM=3): almost best results, but not that much

In particular, in the "big" test (testHighNumberCheckLeakDetectorVersions where more than 16000 items are sent, while the other test is only one item from a big file), I am not able to reduce the timer in PARANOID mode, while in "unique" item test ( testRegressionMultipleLevelLeakDetectorNoDisk), the PARANOID decreases a lot.

Even if I cannot understand why setting the upper bound brings such a better improvement, I feel like this is at least a good change.
However, for the other try using discardReadBytes (or tries using wrapped buffers), it seems to not enhanced the timers.

@normanmaurer If you have any clue to continue, I will. I would propose to at least have the minimal change (upper bounds) in order to fix a bit this issue in PARANOID mode (and does not changed a lot, better or worst, for other levels).

_Here are my results (I tried to use profiling to understand the reasons, but the profiling just kill all results, therefore I was unable to deep in buffer implementations to see where is the main difference)._


testHighNumberCheckLeakDetectorVersions
=======================================

HighItemNumberDISABLED1=506.43838600000004, 
HighItemNumberDISABLED2=496.729877, 
HighItemNumberDISABLED3=503.72193200000004, 

**_HighItemNumberSIMPLE1=460.46265600000004, 
HighItemNumberSIMPLE2=475.950721, 
HighItemNumberSIMPLE3=467.69606699999997,_** 

HighItemNumberADVANCED1=465.508472, 
HighItemNumberADVANCED2=611.797092, 
HighItemNumberADVANCED3=467.812221, 

**HighItemNumberPARANOID1=304098.68770899996, 
HighItemNumberPARANOID2=304140.256149, 
HighItemNumberPARANOID3=303301.14581200003,** 

BigItemDISABLED1=426.205971, 
BigItemDISABLED2=441.343974, 
BigItemDISABLED3=420.528978, 

**_BigItemSIMPLE1=420.825118, 
BigItemSIMPLE2=432.968082, 
BigItemSIMPLE3=418.955781,_** 

BigItemADVANCED1=420.391646, 
BigItemADVANCED2=445.089119, 
BigItemADVANCED3=428.97732599999995, 

**BigItemPARANOID1=318339.78715600003
BigItemPARANOID2=319121.713933
BigItemPARANOID3=316866.262051**

testRegressionMultipleLevelLeakDetectorNoDisk
=============================================

Timer: DISABLED NotUsingDisk1 => 195.185296
Timer: DISABLED NotUsingDisk2 => 206.28149
Timer: DISABLED NotUsingDisk3 => 142.354934

**_Timer: SIMPLE NotUsingDisk1 => 185.459894
Timer: SIMPLE NotUsingDisk2 => 82.243331
Timer: SIMPLE NotUsingDisk3 => 83.65489_**

Timer: ADVANCED NotUsingDisk1 => 175.245971
Timer: ADVANCED NotUsingDisk2 => 214.114222
Timer: ADVANCED NotUsingDisk3 => 89.005829

**Timer: PARANOID NotUsingDisk1 => 119286.301819
Timer: PARANOID NotUsingDisk2 => 1604.135822
Timer: PARANOID NotUsingDisk3 => 2489.555437**

Use final here, it would change a lot how the JIT would optimize it.
Use a sys property to set this and just use different runs with different JVMs (probably using a good profiler + JMH bench would be ideal to be sure of the impact/meaning of changes)
@franz1981 Thank you !
I agree that making it final shall be done to achieve final checks. This variable was only intend to allow 3 kinds of tests, not to be keeped of course in final code (and if final, I couldn't change the value of course).

Now, using a final here, 3 different runs (changing value of course) and a good profiler could be far better and ideal, I agree. 

However, as I said, when using a profiler (the one I have is from Oracle - VisualVM-, and I believe it is not the best around, but the only one I've got), the performances drop severely (even in DISCARDED mode), therefore nothing to look at correctly. For my other tasks, this one is great, giving the information I need and with not that much impact. But there, the performances are almost all at the same level than PARANOID, even in DISCARDED or SIMPLE mode.

Can you suggest some tools (free, sadly) that could help in this research?

Sure!

https://github.com/jvm-profiling-tools/async-profiler is probably one of the most complete, accurate and cheap (;)) ones I know.
Highly suggested to add -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints
@njhill now you read my mind bud?:)
2020
It's suspicious to see a while-loop for releasing a reference counted object. Increments and decrements of the ref count are supposed to be pair-wise, following the structure of the code.
Odd formatting that the constants ended up on their own line.
OK
Indeed, and I did not found why in some cases I have a huge number of refCnt(), so this.
However, I will remove it, going back to previous situation (probably bad programming on my side).
Right, wrong indentation through IJ
```suggestion
 *   https://www.apache.org/licenses/LICENSE-2.0
```
Already done ;-)
nit: we use 4 spaces ... Please change everywhere to be consistent with our code-styling 
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
hmm... retainedSlice(...) should be fine. This will do an extra memory copy which I think is not needed.
I think the `b.readable()` for the second arg is not needed. 
OK, I will double check.
As I wrote, there is a "old" bug, rarely catched, when the buffer `undecodedChunk` exceed the memory limit. When it reached this limit, there is a call to `undecodedChunk.discardReadBytes();`. I found this bug accidently, through my other new calls to the very same methods and errors in unit testing.

When this call occurs, the underlying buffers are changed, changing all HttpData values (of course, incorrectly).
To prevent this, I propose to copy it. I know this is higher in memory copy, but correct from final values, instead of current status.
Also, I changed this "high limit only" call to "each time it is relevant" (when new data can be added while there are enough space within the current buffer to fill it with this new data).

Another way would be to igonre totally this high limit and therefore not calling, nowhere, this `undecodedChunk.discardReadBytes();`, in order to keep final consistency.

WDYT?
Not mandatory, but there, it is valid (there is no need to allocate more than this in this specific case).
If you prefer, I could remove it (while valid).
Of course, the second option (no more calls to `discardReadBytes()`) would lead to higher memory usage since the underlying buffer will be kept until the end.
Using "to disk" (or mixed way) management of HttpData can decrease this memory pressure, espcially for File upload (but not limited to), but as in mixed way, some HttpData could stay using the original undecodedChunk, this could lead in wrong data at the end, so my original proposition.

Any direction or idea ?
I think we should only call `undecoded.discardReadBytes()` if `refCnt() == 1`.
It could be a good idea !
I know this will be almost never true (due to `retainedSlice()` that will increment `refCnt()`), but at least, it will give a chance for it (as with `SeekAhead` algorithm that is almost never usable) and obviously limits the copy as much as possible.

I will git it a try. Thanks @normanmaurer for this idea !
Done
Done:

New benchmark results:
Benchmark                                                                           Mode  Cnt  Score   Error   Units
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigAdvancedLevel   thrpt    6  2,248 ± 0,198  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigDisabledLevel   thrpt    6  2,067 ± 1,219  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigParanoidLevel   thrpt    6  1,109 ± 0,038  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderBigSimpleLevel     thrpt    6  2,326 ± 0,314  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighAdvancedLevel  thrpt    6  1,444 ± 0,226  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighDisabledLevel  thrpt    6  1,462 ± 0,642  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighParanoidLevel  thrpt    6  0,159 ± 0,003  ops/ms
HttpPostMultipartRequestDecoderBenchmark.multipartRequestDecoderHighSimpleLevel    thrpt    6  1,522 ± 0,049  ops/ms

Almost the same as in previous code.
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
seems like a leak in the test that was not related to your change... correct ?
Yes, not related. I was just getting from time to time (before and after changes) an error in PARANOID mode, so I tried to fix as much as possible the tests to be more coherent. 
The reason is: as the test is not going through network but directly, so there is no handler that take care of incoming messages, there is no clean of the request after receiving it by the main handler as it should.
Same answear: not related, just fix the test
Same answear: not related, just fix the test
Same answear: not related, just fix the test
Same answear: not related, just fix the test
Same answear: not related, just fix the test
Same answear: not related, just fix the test

Patch that resolved the issue:
diff --git a/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java b/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java
index a366bd31606..b125dfae0e7 100644
--- a/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java
+++ b/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java
@@ -123,6 +123,11 @@ public class HttpPostMultipartRequestDecoder implements InterfaceHttpPostRequest
      */
     private Attribute currentAttribute;
 
+    /**
+     * The current Data position before finding delimiter
+     */
+    private int lastDataPosition;
+
     private boolean destroyed;
 
     private int discardThreshold = HttpPostRequestDecoder.DEFAULT_DISCARD_THRESHOLD;
@@ -337,12 +342,15 @@ public HttpPostMultipartRequestDecoder offer(HttpContent content) {
                     // which is not really usable for us as we may exceed it once we add more bytes.
                     buf.alloc().buffer(buf.readableBytes()).writeBytes(buf);
         } else {
+            int readPos = undecodedChunk.readerIndex();
+            int writable = undecodedChunk.writableBytes();
+            int toWrite = buf.readableBytes();
+            if (undecodedChunk.refCnt() == 1 && writable < toWrite && readPos + writable >= toWrite) {
+                undecodedChunk.discardReadBytes();
+            }
             undecodedChunk.writeBytes(buf);
         }
         parseBody();
-        if (undecodedChunk != null && undecodedChunk.writerIndex() > discardThreshold) {
-            undecodedChunk.discardReadBytes();
-        }
         return this;
     }
 
@@ -980,47 +988,6 @@ private void cleanMixedAttributes() {
         currentFieldAttributes.remove(HttpHeaderValues.FILENAME);
     }
 
-    /**
-     * Read one line up to the CRLF or LF
-     *
-     * @return the String from one line
-     * @throws NotEnoughDataDecoderException
-     *             Need more chunks and reset the {@code readerIndex} to the previous
-     *             value
-     */
-    private static String readLineStandard(ByteBuf undecodedChunk, Charset charset) {
-        int readerIndex = undecodedChunk.readerIndex();
-        ByteBuf line = undecodedChunk.alloc().heapBuffer(64);
-        try {
-            while (undecodedChunk.isReadable()) {
-                byte nextByte = undecodedChunk.readByte();
-                if (nextByte == HttpConstants.CR) {
-                    // check but do not changed readerIndex
-                    nextByte = undecodedChunk.getByte(undecodedChunk.readerIndex());
-                    if (nextByte == HttpConstants.LF) {
-                        // force read
-                        undecodedChunk.readByte();
-                        return line.toString(charset);
-                    } else {
-                        // Write CR (not followed by LF)
-                        line.writeByte(HttpConstants.CR);
-                    }
-                } else if (nextByte == HttpConstants.LF) {
-                    return line.toString(charset);
-                } else {
-                    line.writeByte(nextByte);
-                }
-            }
-        } catch (IndexOutOfBoundsException e) {
-            undecodedChunk.readerIndex(readerIndex);
-            throw new NotEnoughDataDecoderException(e);
-        } finally {
-            line.release();
-        }
-        undecodedChunk.readerIndex(readerIndex);
-        throw new NotEnoughDataDecoderException();
-    }
-
     /**
      * Read one line up to the CRLF or LF
      *
@@ -1030,44 +997,23 @@ private static String readLineStandard(ByteBuf undecodedChunk, Charset charset)
      *             value
      */
     private static String readLine(ByteBuf undecodedChunk, Charset charset) {
-        if (!undecodedChunk.hasArray()) {
-            return readLineStandard(undecodedChunk, charset);
+        final int readerIndex = undecodedChunk.readerIndex();
+        int posLf = undecodedChunk.bytesBefore(HttpConstants.LF);
+        if (posLf == -1) {
+            throw new NotEnoughDataDecoderException();
         }
-        SeekAheadOptimize sao = new SeekAheadOptimize(undecodedChunk);
-        int readerIndex = undecodedChunk.readerIndex();
-        ByteBuf line = undecodedChunk.alloc().heapBuffer(64);
-        try {
-            while (sao.pos < sao.limit) {
-                byte nextByte = sao.bytes[sao.pos++];
-                if (nextByte == HttpConstants.CR) {
-                    if (sao.pos < sao.limit) {
-                        nextByte = sao.bytes[sao.pos++];
-                        if (nextByte == HttpConstants.LF) {
-                            sao.setReadPosition(0);
-                            return line.toString(charset);
-                        } else {
-                            // Write CR (not followed by LF)
-                            sao.pos--;
-                            line.writeByte(HttpConstants.CR);
-                        }
-                    } else {
-                        line.writeByte(nextByte);
-                    }
-                } else if (nextByte == HttpConstants.LF) {
-                    sao.setReadPosition(0);
-                    return line.toString(charset);
-                } else {
-                    line.writeByte(nextByte);
-                }
-            }
-        } catch (IndexOutOfBoundsException e) {
-            undecodedChunk.readerIndex(readerIndex);
-            throw new NotEnoughDataDecoderException(e);
-        } finally {
-            line.release();
+        boolean crFound =
+            undecodedChunk.getByte(readerIndex + posLf - 1) == HttpConstants.CR;
+        if (crFound) {
+            posLf--;
         }
-        undecodedChunk.readerIndex(readerIndex);
-        throw new NotEnoughDataDecoderException();
+        CharSequence line = undecodedChunk.readCharSequence(posLf, charset);
+        if (crFound) {
+            undecodedChunk.skipBytes(2);
+        } else {
+            undecodedChunk.skipBytes(1);
+        }
+        return line.toString();
     }
 
     /**
@@ -1085,201 +1031,73 @@ private static String readLine(ByteBuf undecodedChunk, Charset charset) {
      *             Need more chunks and reset the {@code readerIndex} to the previous
      *             value
      */
-    private static String readDelimiterStandard(ByteBuf undecodedChunk, String delimiter) {
-        int readerIndex = undecodedChunk.readerIndex();
+    private String readDelimiter(ByteBuf undecodedChunk, String delimiter) {
+        final int readerIndex = undecodedChunk.readerIndex();
         try {
-            StringBuilder sb = new StringBuilder(64);
-            int delimiterPos = 0;
-            int len = delimiter.length();
-            while (undecodedChunk.isReadable() && delimiterPos < len) {
-                byte nextByte = undecodedChunk.readByte();
-                if (nextByte == delimiter.charAt(delimiterPos)) {
-                    delimiterPos++;
-                    sb.append((char) nextByte);
+            final int len = delimiter.length();
+            if (len + 2 > undecodedChunk.readableBytes()) {
+                // Not able to check if "--" is present
+                throw new NotEnoughDataDecoderException();
+            }
+            int newPositionDelimiter = findDelimiter(undecodedChunk, delimiter, 0);
+            if (newPositionDelimiter != 0) {
+                // Delimiter not fully found
+                throw new NotEnoughDataDecoderException();
+            }
+            byte nextByte = undecodedChunk.getByte(readerIndex + len);
+            // first check for opening delimiter
+            if (nextByte == HttpConstants.CR) {
+                nextByte = undecodedChunk.getByte(readerIndex + len + 1);
+                if (nextByte == HttpConstants.LF) {
+                    CharSequence line = undecodedChunk.readCharSequence(len, charset);
+                    undecodedChunk.skipBytes(2);
+                    return line.toString();
                 } else {
+                    // error since CR must be followed by LF
                     // delimiter not found so break here !
                     undecodedChunk.readerIndex(readerIndex);
                     throw new NotEnoughDataDecoderException();
                 }
-            }
-            // Now check if either opening delimiter or closing delimiter
-            if (undecodedChunk.isReadable()) {
-                byte nextByte = undecodedChunk.readByte();
-                // first check for opening delimiter
-                if (nextByte == HttpConstants.CR) {
-                    nextByte = undecodedChunk.readByte();
-                    if (nextByte == HttpConstants.LF) {
-                        return sb.toString();
-                    } else {
-                        // error since CR must be followed by LF
-                        // delimiter not found so break here !
-                        undecodedChunk.readerIndex(readerIndex);
-                        throw new NotEnoughDataDecoderException();
-                    }
-                } else if (nextByte == HttpConstants.LF) {
-                    return sb.toString();
-                } else if (nextByte == '-') {
-                    sb.append('-');
-                    // second check for closing delimiter
-                    nextByte = undecodedChunk.readByte();
-                    if (nextByte == '-') {
-                        sb.append('-');
-                        // now try to find if CRLF or LF there
-                        if (undecodedChunk.isReadable()) {
+            } else if (nextByte == HttpConstants.LF) {
+                CharSequence line = undecodedChunk.readCharSequence(len, charset);
+                undecodedChunk.skipBytes(1);
+                return line.toString();
+            } else if (nextByte == '-') {
+                // second check for closing delimiter
+                nextByte = undecodedChunk.getByte(readerIndex + len + 1);
+                if (nextByte == '-') {
+                    CharSequence line = undecodedChunk.readCharSequence(len + 2, charset);
+                    // now try to find if CRLF or LF there
+                    if (undecodedChunk.isReadable()) {
+                        nextByte = undecodedChunk.readByte();
+                        if (nextByte == HttpConstants.CR) {
                             nextByte = undecodedChunk.readByte();
-                            if (nextByte == HttpConstants.CR) {
-                                nextByte = undecodedChunk.readByte();
-                                if (nextByte == HttpConstants.LF) {
-                                    return sb.toString();
-                                } else {
-                                    // error CR without LF
-                                    // delimiter not found so break here !
-                                    undecodedChunk.readerIndex(readerIndex);
-                                    throw new NotEnoughDataDecoderException();
-                                }
-                            } else if (nextByte == HttpConstants.LF) {
-                                return sb.toString();
+                            if (nextByte == HttpConstants.LF) {
+                                return line.toString();
                             } else {
-                                // No CRLF but ok however (Adobe Flash uploader)
-                                // minus 1 since we read one char ahead but
-                                // should not
-                                undecodedChunk.readerIndex(undecodedChunk.readerIndex() - 1);
-                                return sb.toString();
+                                // error CR without LF
+                                // delimiter not found so break here !
+                                undecodedChunk.readerIndex(readerIndex);
+                                throw new NotEnoughDataDecoderException();
                             }
-                        }
-                        // FIXME what do we do here?
-                        // either considering it is fine, either waiting for
-                        // more data to come?
-                        // lets try considering it is fine...
-                        return sb.toString();
-                    }
-                    // only one '-' => not enough
-                    // whatever now => error since incomplete
-                }
-            }
-        } catch (IndexOutOfBoundsException e) {
-            undecodedChunk.readerIndex(readerIndex);
-            throw new NotEnoughDataDecoderException(e);
-        }
-        undecodedChunk.readerIndex(readerIndex);
-        throw new NotEnoughDataDecoderException();
-    }
-
-    /**
-     * Read one line up to --delimiter or --delimiter-- and if existing the CRLF
-     * or LF. Note that CRLF or LF are mandatory for opening delimiter
-     * (--delimiter) but not for closing delimiter (--delimiter--) since some
-     * clients does not include CRLF in this case.
-     *
-     * @param delimiter
-     *            of the form --string, such that '--' is already included
-     * @return the String from one line as the delimiter searched (opening or
-     *         closing)
-     * @throws NotEnoughDataDecoderException
-     *             Need more chunks and reset the readerInder to the previous
-     *             value
-     */
-    private static String readDelimiter(ByteBuf undecodedChunk, String delimiter) {
-        if (!undecodedChunk.hasArray()) {
-            return readDelimiterStandard(undecodedChunk, delimiter);
-        }
-        SeekAheadOptimize sao = new SeekAheadOptimize(undecodedChunk);
-        int readerIndex = undecodedChunk.readerIndex();
-        int delimiterPos = 0;
-        int len = delimiter.length();
-        try {
-            StringBuilder sb = new StringBuilder(64);
-            // check conformity with delimiter
-            while (sao.pos < sao.limit && delimiterPos < len) {
-                byte nextByte = sao.bytes[sao.pos++];
-                if (nextByte == delimiter.charAt(delimiterPos)) {
-                    delimiterPos++;
-                    sb.append((char) nextByte);
-                } else {
-                    // delimiter not found so break here !
-                    undecodedChunk.readerIndex(readerIndex);
-                    throw new NotEnoughDataDecoderException();
-                }
-            }
-            // Now check if either opening delimiter or closing delimiter
-            if (sao.pos < sao.limit) {
-                byte nextByte = sao.bytes[sao.pos++];
-                if (nextByte == HttpConstants.CR) {
-                    // first check for opening delimiter
-                    if (sao.pos < sao.limit) {
-                        nextByte = sao.bytes[sao.pos++];
-                        if (nextByte == HttpConstants.LF) {
-                            sao.setReadPosition(0);
-                            return sb.toString();
+                        } else if (nextByte == HttpConstants.LF) {
+                            return line.toString();
                         } else {
-                            // error CR without LF
-                            // delimiter not found so break here !
-                            undecodedChunk.readerIndex(readerIndex);
-                            throw new NotEnoughDataDecoderException();
+                            // No CRLF but ok however (Adobe Flash uploader)
+                            // minus 1 since we read one char ahead but
+                            // should not
+                            undecodedChunk.readerIndex(undecodedChunk.readerIndex() - 1);
+                            return line.toString();
                         }
-                    } else {
-                        // error since CR must be followed by LF
-                        // delimiter not found so break here !
-                        undecodedChunk.readerIndex(readerIndex);
-                        throw new NotEnoughDataDecoderException();
-                    }
-                } else if (nextByte == HttpConstants.LF) {
-                    // same first check for opening delimiter where LF used with
-                    // no CR
-                    sao.setReadPosition(0);
-                    return sb.toString();
-                } else if (nextByte == '-') {
-                    sb.append('-');
-                    // second check for closing delimiter
-                    if (sao.pos < sao.limit) {
-                        nextByte = sao.bytes[sao.pos++];
-                        if (nextByte == '-') {
-                            sb.append('-');
-                            // now try to find if CRLF or LF there
-                            if (sao.pos < sao.limit) {
-                                nextByte = sao.bytes[sao.pos++];
-                                if (nextByte == HttpConstants.CR) {
-                                    if (sao.pos < sao.limit) {
-                                        nextByte = sao.bytes[sao.pos++];
-                                        if (nextByte == HttpConstants.LF) {
-                                            sao.setReadPosition(0);
-                                            return sb.toString();
-                                        } else {
-                                            // error CR without LF
-                                            // delimiter not found so break here !
-                                            undecodedChunk.readerIndex(readerIndex);
-                                            throw new NotEnoughDataDecoderException();
-                                        }
-                                    } else {
-                                        // error CR without LF
-                                        // delimiter not found so break here !
-                                        undecodedChunk.readerIndex(readerIndex);
-                                        throw new NotEnoughDataDecoderException();
-                                    }
-                                } else if (nextByte == HttpConstants.LF) {
-                                    sao.setReadPosition(0);
-                                    return sb.toString();
-                                } else {
-                                    // No CRLF but ok however (Adobe Flash
-                                    // uploader)
-                                    // minus 1 since we read one char ahead but
-                                    // should not
-                                    sao.setReadPosition(1);
-                                    return sb.toString();
-                                }
-                            }
-                            // FIXME what do we do here?
-                            // either considering it is fine, either waiting for
-                            // more data to come?
-                            // lets try considering it is fine...
-                            sao.setReadPosition(0);
-                            return sb.toString();
-                        }
-                        // whatever now => error since incomplete
-                        // only one '-' => not enough or whatever not enough
-                        // element
                     }
+                    // FIXME what do we do here?
+                    // either considering it is fine, either waiting for
+                    // more data to come?
+                    // lets try considering it is fine...
+                    return line.toString();
                 }
+                // only one '-' => not enough
+                // whatever now => error since incomplete
             }
         } catch (IndexOutOfBoundsException e) {
             undecodedChunk.readerIndex(readerIndex);
@@ -1290,96 +1108,93 @@ private static String readDelimiter(ByteBuf undecodedChunk, String delimiter) {
     }
 
     /**
-     * Load the field value or file data from a Multipart request
+     * @param undecodedChunk the source where the delimiter is to be found
+     * @param delimiter the string to find out
+     * @param offset the offset from readerIndex within the undecodedChunk to
+     *     start from to find out the delimiter
      *
-     * @return {@code true} if the last chunk is loaded (boundary delimiter found), {@code false} if need more chunks
-     * @throws ErrorDataDecoderException
+     * @return a number >= 0 if found, else new offset with negative value
+     *     (to inverse), both from readerIndex
+     * @throws NotEnoughDataDecoderException
+     *             Need more chunks while relative position with readerIndex is 0
      */
-    private static boolean loadDataMultipartStandard(ByteBuf undecodedChunk, String delimiter, HttpData httpData) {
+    private static int findDelimiter(ByteBuf undecodedChunk, String delimiter, int offset) {
         final int startReaderIndex = undecodedChunk.readerIndex();
         final int delimeterLength = delimiter.length();
-        int index = 0;
-        int lastPosition = startReaderIndex;
-        byte prevByte = HttpConstants.LF;
-        boolean delimiterFound = false;
-        while (undecodedChunk.isReadable()) {
-            final byte nextByte = undecodedChunk.readByte();
-            // Check the delimiter
-            if (prevByte == HttpConstants.LF && nextByte == delimiter.codePointAt(index)) {
-                index++;
-                if (delimeterLength == index) {
-                    delimiterFound = true;
+        final int toRead = undecodedChunk.readableBytes();
+        int newOffset = offset;
+        boolean delimiterNotFound = true;
+        while (delimiterNotFound && newOffset + delimeterLength <= toRead) {
+            int posFirstChar = undecodedChunk
+                .bytesBefore(startReaderIndex + newOffset, toRead - newOffset,
+                             (byte) delimiter.codePointAt(0));
+            if (posFirstChar == -1) {
+                newOffset = toRead;
+                return -newOffset;
+            }
+            newOffset = posFirstChar + offset;
+            if (newOffset + delimeterLength > toRead) {
+                return -newOffset;
+            }
+            // assume will found it
+            delimiterNotFound = false;
+            for (int index = 1; index < delimeterLength; index++) {
+                if (undecodedChunk.getByte(startReaderIndex + newOffset + index) != delimiter.codePointAt(index)) {
+                    // ignore first found offset and redo search from next char
+                    newOffset++;
+                    delimiterNotFound = true;
                     break;
                 }
-                continue;
             }
-            lastPosition = undecodedChunk.readerIndex();
-            if (nextByte == HttpConstants.LF) {
-                index = 0;
-                lastPosition -= (prevByte == HttpConstants.CR)? 2 : 1;
-            }
-            prevByte = nextByte;
         }
-        if (prevByte == HttpConstants.CR) {
-            lastPosition--;
-        }
-        ByteBuf content = undecodedChunk.retainedSlice(startReaderIndex, lastPosition - startReaderIndex);
-        try {
-            httpData.addContent(content, delimiterFound);
-        } catch (IOException e) {
-            throw new ErrorDataDecoderException(e);
+        if (delimiterNotFound || newOffset + delimeterLength > toRead) {
+            if (newOffset == 0) {
+                throw new NotEnoughDataDecoderException();
+            }
+            return -newOffset;
         }
-        undecodedChunk.readerIndex(lastPosition);
-        return delimiterFound;
+        return newOffset;
     }
 
     /**
      * Load the field value from a Multipart request
      *
      * @return {@code true} if the last chunk is loaded (boundary delimiter found), {@code false} if need more chunks
+     *
      * @throws ErrorDataDecoderException
      */
-    private static boolean loadDataMultipart(ByteBuf undecodedChunk, String delimiter, HttpData httpData) {
-        if (!undecodedChunk.hasArray()) {
-            return loadDataMultipartStandard(undecodedChunk, delimiter, httpData);
-        }
-        final SeekAheadOptimize sao = new SeekAheadOptimize(undecodedChunk);
+    private boolean loadDataMultipart(ByteBuf undecodedChunk, String delimiter,
+                                      HttpData httpData) {
         final int startReaderIndex = undecodedChunk.readerIndex();
-        final int delimeterLength = delimiter.length();
-        int index = 0;
-        int lastRealPos = sao.pos;
-        byte prevByte = HttpConstants.LF;
-        boolean delimiterFound = false;
-        while (sao.pos < sao.limit) {
-            final byte nextByte = sao.bytes[sao.pos++];
-            // Check the delimiter
-            if (prevByte == HttpConstants.LF && nextByte == delimiter.codePointAt(index)) {
-                index++;
-                if (delimeterLength == index) {
-                    delimiterFound = true;
-                    break;
-                }
-                continue;
-            }
-            lastRealPos = sao.pos;
-            if (nextByte == HttpConstants.LF) {
-                index = 0;
-                lastRealPos -= (prevByte == HttpConstants.CR)? 2 : 1;
+        int newOffset;
+        try {
+            newOffset = findDelimiter(undecodedChunk, delimiter, lastDataPosition);
+            if (newOffset < 0) {
+                // delimiter not found
+                lastDataPosition = -newOffset;
+                return false;
             }
-            prevByte = nextByte;
+        } catch (NotEnoughDataDecoderException e) {
+            // Not enough data and no change to lastDataPosition
+            return false;
         }
-        if (prevByte == HttpConstants.CR) {
-            lastRealPos--;
+        // found delimiter but still need to check if CRLF before
+        int startDelimiter = newOffset;
+        if (undecodedChunk.getByte(startReaderIndex + startDelimiter - 1) == HttpConstants.LF) {
+            startDelimiter--;
+            if (undecodedChunk.getByte(startReaderIndex + startDelimiter - 1) == HttpConstants.CR) {
+                startDelimiter--;
+            }
         }
-        final int lastPosition = sao.getReadPosition(lastRealPos);
-        final ByteBuf content = undecodedChunk.retainedSlice(startReaderIndex, lastPosition - startReaderIndex);
+        ByteBuf content = undecodedChunk.retainedSlice(startReaderIndex, startDelimiter);
         try {
-            httpData.addContent(content, delimiterFound);
+            httpData.addContent(content, true);
         } catch (IOException e) {
             throw new ErrorDataDecoderException(e);
         }
-        undecodedChunk.readerIndex(lastPosition);
-        return delimiterFound;
+        lastDataPosition = 0;
+        undecodedChunk.readerIndex(startReaderIndex + startDelimiter);
+        return true;
     }
 
     /**
diff --git a/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostStandardRequestDecoder.java b/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostStandardRequestDecoder.java
index 37e41fa9176..89733794a96 100644
--- a/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostStandardRequestDecoder.java
+++ b/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostStandardRequestDecoder.java
@@ -299,12 +299,15 @@ public HttpPostStandardRequestDecoder offer(HttpContent content) {
                     // which is not really usable for us as we may exceed it once we add more bytes.
                     buf.alloc().buffer(buf.readableBytes()).writeBytes(buf);
         } else {
+            int readPos = undecodedChunk.readerIndex();
+            int writable = undecodedChunk.writableBytes();
+            int toWrite = buf.readableBytes();
+            if (undecodedChunk.refCnt() == 1 && writable < toWrite && readPos + writable >= toWrite) {
+                undecodedChunk.discardReadBytes();
+            }
             undecodedChunk.writeBytes(buf);
         }
         parseBody();
-        if (undecodedChunk != null && undecodedChunk.writerIndex() > discardThreshold) {
-            undecodedChunk.discardReadBytes();
-        }
         return this;
     }
 
diff --git a/microbench/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoderBenchmark.java b/microbench/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoderBenchmark.java
new file mode 100644
index 00000000000..c570cd67455
--- /dev/null
+++ b/microbench/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoderBenchmark.java
@@ -0,0 +1,202 @@
+/*
+ * Copyright 2020 The Netty Project
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ *   https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+package io.netty.handler.codec.http.multipart;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.handler.codec.http.DefaultHttpContent;
+import io.netty.handler.codec.http.DefaultHttpRequest;
+import io.netty.handler.codec.http.DefaultLastHttpContent;
+import io.netty.handler.codec.http.HttpHeaderNames;
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpVersion;
+import io.netty.microbench.util.AbstractMicrobenchmark;
+import io.netty.util.ResourceLeakDetector;
+import io.netty.util.ResourceLeakDetector.Level;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.concurrent.TimeUnit;
+
+
+@Threads(1)
+@Warmup(iterations = 2)
+@Measurement(iterations = 3)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+public class HttpPostMultipartRequestDecoderBenchmark
+        extends AbstractMicrobenchmark {
+
+    public double testHighNumberChunks(boolean big, boolean noDisk) {
+        String BOUNDARY = "01f136d9282f";
+        int size = 8 * 1024;
+        int chunkNumber = 64;
+        StringBuilder stringBuilder = new StringBuilder(size);
+        stringBuilder.setLength(size);
+        String data = stringBuilder.toString();
+
+        byte[] bodyStartBytes = ("--" + BOUNDARY + "\n" +
+                                 "Content-Disposition: form-data; name=\"msg_id\"\n\n15200\n--" +
+                                 BOUNDARY +
+                                 "\nContent-Disposition: form-data; name=\"msg1\"; filename=\"file1.txt\"\n\n" +
+                                 data).getBytes();
+        byte[] bodyPartBigBytes = data.getBytes();
+        byte[] intermediaryBytes = ("\n--" + BOUNDARY +
+                                    "\nContent-Disposition: form-data; name=\"msg2\"; filename=\"file2.txt\"\n\n" +
+                                    data).getBytes();
+        byte[] finalBigBytes = ("\n" + "--" + BOUNDARY + "--\n").getBytes();
+        ByteBuf firstBuf = Unpooled.wrappedBuffer(bodyStartBytes);
+        ByteBuf finalBuf = Unpooled.wrappedBuffer(finalBigBytes);
+        ByteBuf nextBuf;
+        if (big) {
+            nextBuf = Unpooled.wrappedBuffer(bodyPartBigBytes);
+        } else {
+            nextBuf = Unpooled.wrappedBuffer(intermediaryBytes);
+        }
+        DefaultHttpRequest req =
+                new DefaultHttpRequest(HttpVersion.HTTP_1_0, HttpMethod.POST, "/up");
+        req.headers().add(HttpHeaderNames.CONTENT_TYPE,
+                          "multipart/form-data; boundary=" + BOUNDARY);
+
+        long start = System.nanoTime();
+
+        DefaultHttpDataFactory defaultHttpDataFactory =
+                new DefaultHttpDataFactory(noDisk? 1024 * 1024 : 16 * 1024);
+        HttpPostRequestDecoder decoder =
+                new HttpPostRequestDecoder(defaultHttpDataFactory, req);
+        firstBuf.retain();
+        decoder.offer(new DefaultHttpContent(firstBuf));
+        firstBuf.release();
+        for (int i = 1; i < chunkNumber; i++) {
+            nextBuf.retain();
+            decoder.offer(new DefaultHttpContent(nextBuf));
+            nextBuf.release();
+            nextBuf.readerIndex(0);
+        }
+        finalBuf.retain();
+        decoder.offer(new DefaultLastHttpContent(finalBuf));
+        finalBuf.release();
+        while (decoder.hasNext()) {
+            InterfaceHttpData httpData = decoder.next();
+        }
+        while (finalBuf.refCnt() > 0) {
+            finalBuf.release();
+        }
+        while (nextBuf.refCnt() > 0) {
+            nextBuf.release();
+        }
+        while (finalBuf.refCnt() > 0) {
+            finalBuf.release();
+        }
+        long stop = System.nanoTime();
+        double time = (stop - start) / 1000000.0;
+        defaultHttpDataFactory.cleanAllHttpData();
+        defaultHttpDataFactory.cleanRequestHttpData(req);
+        decoder.destroy();
+        return time;
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderHighDisabledLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.DISABLED);
+            return testHighNumberChunks(false, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderBigDisabledLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.DISABLED);
+            return testHighNumberChunks(true, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderHighSimpleLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.SIMPLE);
+            return testHighNumberChunks(false, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderBigSimpleLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.SIMPLE);
+            return testHighNumberChunks(true, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderHighAdvancedLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.ADVANCED);
+            return testHighNumberChunks(false, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderBigAdvancedLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.ADVANCED);
+            return testHighNumberChunks(true, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderHighParanoidLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.PARANOID);
+            return testHighNumberChunks(false, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+    @Benchmark
+    public double multipartRequestDecoderBigParanoidLevel() {
+        final Level level = ResourceLeakDetector.getLevel();
+        try {
+            ResourceLeakDetector.setLevel(Level.PARANOID);
+            return testHighNumberChunks(true, true);
+        } finally {
+            ResourceLeakDetector.setLevel(level);
+        }
+    }
+
+}
diff --git a/microbench/src/main/java/io/netty/handler/codec/http/multipart/package-info.java b/microbench/src/main/java/io/netty/handler/codec/http/multipart/package-info.java
new file mode 100644
index 00000000000..1f4cc119dcb
--- /dev/null
+++ b/microbench/src/main/java/io/netty/handler/codec/http/multipart/package-info.java
@@ -0,0 +1,19 @@
+/*
+ * Copyright 2020 The Netty Project
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ *   https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+/**
+ * Benchmarks for {@link io.netty.handler.codec.http.multipart}.
+ */
+package io.netty.handler.codec.http.multipart;


Test patch:
diff --git a/codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java b/codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java
index ba01e4d240b..45b8461ce72 100644
--- a/codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java
+++ b/codec-http/src/test/java/io/netty/handler/codec/http/multipart/HttpPostRequestDecoderTest.java
@@ -132,6 +132,7 @@ public void testFullHttpRequestUpload() throws Exception {
         // Create decoder instance to test.
         final HttpPostRequestDecoder decoder = new HttpPostRequestDecoder(inMemoryFactory, req);
         assertFalse(decoder.getBodyHttpDatas().isEmpty());
+        req.release();
         decoder.destroy();
     }
 
@@ -178,6 +179,7 @@ public void testMultipartCodecWithCRasEndOfAttribute() throws Exception {
             assertNotNull(datar);
             assertEquals(datas[i].getBytes(CharsetUtil.UTF_8).length, datar.length);
 
+            req.release();
             decoder.destroy();
         }
     }
@@ -211,6 +213,7 @@ public void testQuotedBoundary() throws Exception {
         // Create decoder instance to test.
         final HttpPostRequestDecoder decoder = new HttpPostRequestDecoder(inMemoryFactory, req);
         assertFalse(decoder.getBodyHttpDatas().isEmpty());
+        req.release();
         decoder.destroy();
     }
 
@@ -368,6 +371,7 @@ public void testFilenameContainingSemicolon() throws Exception {
         // Create decoder instance to test.
         final HttpPostRequestDecoder decoder = new HttpPostRequestDecoder(inMemoryFactory, req);
         assertFalse(decoder.getBodyHttpDatas().isEmpty());
+        req.release();
         decoder.destroy();
     }
 
@@ -397,6 +401,7 @@ public void testFilenameContainingSemicolon2() throws Exception {
         assertTrue(part1 instanceof FileUpload);
         FileUpload fileUpload = (FileUpload) part1;
         assertEquals("tmp 0.txt", fileUpload.getFilename());
+        req.release();
         decoder.destroy();
     }
 
@@ -427,6 +432,7 @@ public void testMultipartRequestWithoutContentTypeBody() {
         // Create decoder instance to test without any exception.
         final HttpPostRequestDecoder decoder = new HttpPostRequestDecoder(inMemoryFactory, req);
         assertFalse(decoder.getBodyHttpDatas().isEmpty());
+        req.release();
         decoder.destroy();
     }
 
@@ -459,8 +465,8 @@ public void testDecodeOtherMimeHeaderFields() throws Exception {
         FileUpload fileUpload = (FileUpload) part1;
         byte[] fileBytes = fileUpload.get();
         assertTrue("the filecontent should not be decoded", filecontent.equals(new String(fileBytes)));
-        decoder.destroy();
         req.release();
+        decoder.destroy();
     }
 
     @Test
@@ -538,8 +544,8 @@ public void testFormEncodeIncorrect() throws Exception {
         } catch (HttpPostRequestDecoder.ErrorDataDecoderException e) {
             assertTrue(e.getCause() instanceof IllegalArgumentException);
         } finally {
-            decoder.destroy();
             content.release();
+            decoder.destroy();
         }
     }
 
@@ -573,8 +579,8 @@ public void testDecodeContentDispositionFieldParameters() throws Exception {
         assertTrue("the item should be a FileUpload", part1 instanceof FileUpload);
         FileUpload fileUpload = (FileUpload) part1;
         assertEquals("the filename should be decoded", filename, fileUpload.getFilename());
-        decoder.destroy();
         req.release();
+        decoder.destroy();
     }
 
     // https://github.com/netty/netty/pull/7265
@@ -609,8 +615,8 @@ public void testDecodeWithLanguageContentDispositionFieldParameters() throws Exc
         assertTrue("the item should be a FileUpload", part1 instanceof FileUpload);
         FileUpload fileUpload = (FileUpload) part1;
         assertEquals("the filename should be decoded", filename, fileUpload.getFilename());
-        decoder.destroy();
         req.release();
+        decoder.destroy();
     }
 
     // https://github.com/netty/netty/pull/7265
@@ -704,6 +710,7 @@ public void testDecodeMalformedEmptyContentTypeFieldParameters() throws Exceptio
         assertTrue(part1 instanceof FileUpload);
         FileUpload fileUpload = (FileUpload) part1;
         assertEquals("tmp-0.txt", fileUpload.getFilename());
+        req.release();
         decoder.destroy();
     }
 
@@ -752,7 +759,7 @@ public void testNotLeak() {
         FullHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/",
                 Unpooled.copiedBuffer("a=1&&b=2", CharsetUtil.US_ASCII));
         try {
-            new HttpPostStandardRequestDecoder(request);
+            new HttpPostStandardRequestDecoder(request).destroy();
         } finally {
             assertTrue(request.release());
         }
@@ -772,7 +779,7 @@ private static void testNotLeakWhenWrapIllegalArgumentException(ByteBuf buf) {
         buf.writeCharSequence("a=b&foo=%22bar%22&==", CharsetUtil.US_ASCII);
         FullHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/", buf);
         try {
-            new HttpPostStandardRequestDecoder(request);
+            new HttpPostStandardRequestDecoder(request).destroy();
         } finally {
             assertTrue(request.release());
         }
@@ -823,8 +830,8 @@ public void testDecodeWithLanguageContentDispositionFieldParametersForFix() thro
         FileUpload fileUpload = (FileUpload) part1;
         assertEquals("the filename should be decoded", filename, fileUpload.getFilename());
 
-        decoder.destroy();
         req.release();
+        decoder.destroy();
     }
 
     @Test
@@ -860,8 +867,8 @@ public void testDecodeFullHttpRequestWithUrlEncodedBody() throws Exception {
         assertTrue(attr.getByteBuf().isDirect());
         assertEquals("los angeles", attr.getValue());
 
-        decoder.destroy();
         req.release();
+        decoder.destroy();
     }
 
     @Test
@@ -877,7 +884,7 @@ public void testDecodeFullHttpRequestWithUrlEncodedBodyWithBrokenHexByte0() {
         } catch (HttpPostRequestDecoder.ErrorDataDecoderException e) {
             assertEquals("Invalid hex byte at index '0' in string: '%'", e.getMessage());
         } finally {
-            req.release();
+            assertTrue(req.release());
         }
     }
 
@@ -894,7 +901,7 @@ public void testDecodeFullHttpRequestWithUrlEncodedBodyWithBrokenHexByte1() {
         } catch (HttpPostRequestDecoder.ErrorDataDecoderException e) {
             assertEquals("Invalid hex byte at index '0' in string: '%2'", e.getMessage());
         } finally {
-            req.release();
+            assertTrue(req.release());
         }
     }
 
@@ -911,7 +918,7 @@ public void testDecodeFullHttpRequestWithUrlEncodedBodyWithInvalidHexNibbleHi()
         } catch (HttpPostRequestDecoder.ErrorDataDecoderException e) {
             assertEquals("Invalid hex byte at index '0' in string: '%Zc'", e.getMessage());
         } finally {
-            req.release();
+            assertTrue(req.release());
         }
     }
 
@@ -928,7 +935,7 @@ public void testDecodeFullHttpRequestWithUrlEncodedBodyWithInvalidHexNibbleLo()
         } catch (HttpPostRequestDecoder.ErrorDataDecoderException e) {
             assertEquals("Invalid hex byte at index '0' in string: '%2g'", e.getMessage());
         } finally {
-            req.release();
+            assertTrue(req.release());
         }
     }
 }