Repository: netty/netty
Issue: JdkZlibEncoder: heap allocated arrays for deflater input generate memory pressure under load

### Expected behavior

I am writing a high-throughput client for a custom protocol over TCP;  the protocol has the option of compressing contents using zlib encoding,  In order to reach performance targets I accumulate encoded frames into batches stored in off-heap pooled ByteBufs and then write the contents to the Netty Channel.  When running load tests with compression enabled I expected very little GC activity.

### Actual behavior

When running load tests with compression enabled I see a lot of GC activity.  This is due to byte arrays allocated on the heap to provide input to zlib Deflater.  When the incoming ByteBuf is direct, we allocate a byte array on the heap and copy the contents into it.

### Steps to reproduce

- Create a Netty client that uses a ZlibEncoder
- Write direct ByteBufs to the Channel in a loop
- Observe the increase in memory consumption (using system tools or a profiler)

### Netty version
```
4.1.70.Final
```

### JVM version (e.g. `java -version`)

```
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
```

### OS version (e.g. `uname -a`)
```Linux peso 5.10.0-1051-oem #53-Ubuntu SMP Thu Oct 28 08:11:53 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux```



Issue comments:

Pull request review comments:
```suggestion
                heapBuf = ctx.alloc().heapBuffer(len, len);
```
Please revert the formatting changes. 
please revert formatting changes
please revert formatting changes
make this final and assign in the `if / else ` blocks
Thank you; done.
@normanmaurer ,
> make this final and assign in the if / else blocks

I introduced a `try / finally` so that we would not leak the reference to the allocated heap buffer; this accounts for the formatting changes above and it also confuses the Java compiler when making `heapBuf` final; see attached screenshot.
![Screenshot from 2021-12-06 10-00-16](https://user-images.githubusercontent.com/13126037/144897921-5aff2f72-9d7c-40a9-9012-072bf18b20e4.png)



got it... 
@normanmaurer ,
Are you referring to the additional indentation?  If so, it is due to the addition of the `try / finally` block to ensure that the allocated heapBuf is released on method return.  Do do you want me to remove the `try / finally`?  
Check the checkstyle errors 
Got it; the checkstyle errors should now be addressed.
```suggestion
            for (;;) {
```
```suggestion
            for (;;) {
```

Patch that resolved the issue:
diff --git a/codec/src/main/java/io/netty/handler/codec/compression/JdkZlibEncoder.java b/codec/src/main/java/io/netty/handler/codec/compression/JdkZlibEncoder.java
index 08292ef9546..892f6ccf3d3 100644
--- a/codec/src/main/java/io/netty/handler/codec/compression/JdkZlibEncoder.java
+++ b/codec/src/main/java/io/netty/handler/codec/compression/JdkZlibEncoder.java
@@ -200,42 +200,50 @@ protected void encode(ChannelHandlerContext ctx, ByteBuf uncompressed, ByteBuf o
 
         int offset;
         byte[] inAry;
-        if (uncompressed.hasArray()) {
-            // if it is backed by an array we not need to to do a copy at all
-            inAry = uncompressed.array();
-            offset = uncompressed.arrayOffset() + uncompressed.readerIndex();
-            // skip all bytes as we will consume all of them
-            uncompressed.skipBytes(len);
-        } else {
-            inAry = new byte[len];
-            uncompressed.readBytes(inAry);
-            offset = 0;
-        }
+        ByteBuf heapBuf = null;
+        try {
+            if (uncompressed.hasArray()) {
+                // if it is backed by an array we not need to do a copy at all
+                inAry = uncompressed.array();
+                offset = uncompressed.arrayOffset() + uncompressed.readerIndex();
+                // skip all bytes as we will consume all of them
+                uncompressed.skipBytes(len);
+            } else {
+                heapBuf = ctx.alloc().heapBuffer(len, len);
+                uncompressed.readBytes(heapBuf, len);
+                inAry = heapBuf.array();
+                offset = heapBuf.arrayOffset() + heapBuf.readerIndex();
+            }
 
-        if (writeHeader) {
-            writeHeader = false;
-            if (wrapper == ZlibWrapper.GZIP) {
-                out.writeBytes(gzipHeader);
+            if (writeHeader) {
+                writeHeader = false;
+                if (wrapper == ZlibWrapper.GZIP) {
+                    out.writeBytes(gzipHeader);
+                }
             }
-        }
 
-        if (wrapper == ZlibWrapper.GZIP) {
-            crc.update(inAry, offset, len);
-        }
+            if (wrapper == ZlibWrapper.GZIP) {
+                crc.update(inAry, offset, len);
+            }
 
-        deflater.setInput(inAry, offset, len);
-        for (;;) {
-            deflate(out);
-            if (deflater.needsInput()) {
-                // Consumed everything
-                break;
-            } else {
-                if (!out.isWritable()) {
-                    // We did not consume everything but the buffer is not writable anymore. Increase the capacity to
-                    // make more room.
-                    out.ensureWritable(out.writerIndex());
+            deflater.setInput(inAry, offset, len);
+            for (;;) {
+                deflate(out);
+                if (deflater.needsInput()) {
+                    // Consumed everything
+                    break;
+                } else {
+                    if (!out.isWritable()) {
+                        // We did not consume everything but the buffer is not writable anymore. Increase the capacity
+                        // to make more room.
+                        out.ensureWritable(out.writerIndex());
+                    }
                 }
             }
+        } finally {
+            if (heapBuf != null) {
+                heapBuf.release();
+            }
         }
     }
 


Test patch: