Repository: netty/netty
Issue: 'io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1'

Hi,

We are sending GET requests to our application. We are encountering the issue mentioned below frequently.

Dependencies
Spring Cloud Gateway - 3.0.3
Spring Boot - 2.5.5
Spring webflux - 5.3.10
Reactor netty - 1.0.11
Netty - 4.1.63.Final
OS - Cent OS / Linux

```
{"instant":{"epochSecond":1651180720,"nanoOfSecond":289955175},"thread":"ingress-h2c-epoll-3","level":"WARN","loggerName":"io.netty.channel.DefaultChannelPipeline","message":"An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.","thrown":{"commonElementCount":0,"localizedMessage":"refCnt: 0, decrement: 1","message":"refCnt: 0, decrement: 1","name":"io.netty.util.IllegalReferenceCountException","extendedStackTrace":[{"class":"io.netty.util.internal.ReferenceCountUpdater","method":"toLiveRealRefCnt","file":"ReferenceCountUpdater.java","line":74,"exact":false,"location":"netty-common-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.util.internal.ReferenceCountUpdater","method":"release","file":"ReferenceCountUpdater.java","line":138,"exact":false,"location":"netty-common-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.buffer.AbstractReferenceCountedByteBuf","method":"release","file":"AbstractReferenceCountedByteBuf.java","line":100,"exact":false,"location":"netty-buffer-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.handler.codec.ByteToMessageDecoder","method":"channelRead","file":"ByteToMessageDecoder.java","line":285,"exact":false,"location":"netty-codec-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":379,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":365,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"fireChannelRead","file":"AbstractChannelHandlerContext.java","line":357,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.handler.logging.LoggingHandler","method":"channelRead","file":"LoggingHandler.java","line":271,"exact":true,"location":"netty-handler-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":379,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":365,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"fireChannelRead","file":"AbstractChannelHandlerContext.java","line":357,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.ChannelInboundHandlerAdapter","method":"channelRead","file":"ChannelInboundHandlerAdapter.java","line":93,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"ocpm.cne.gateway.util.ConnectionCountHandler","method":"channelRead","file":"ConnectionCountHandler.java","line":46,"exact":true,"location":"classes!/","version":"?"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":379,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":365,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"fireChannelRead","file":"AbstractChannelHandlerContext.java","line":357,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.DefaultChannelPipeline$HeadContext","method":"channelRead","file":"DefaultChannelPipeline.java","line":1410,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":379,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.AbstractChannelHandlerContext","method":"invokeChannelRead","file":"AbstractChannelHandlerContext.java","line":365,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.DefaultChannelPipeline","method":"fireChannelRead","file":"DefaultChannelPipeline.java","line":919,"exact":true,"location":"netty-transport-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe","method":"epollInReady","file":"AbstractEpollStreamChannel.java","line":795,"exact":true,"location":"netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.epoll.EpollEventLoop","method":"processReady","file":"EpollEventLoop.java","line":480,"exact":true,"location":"netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar!/","version":"4.1.63.Final"},{"class":"io.netty.channel.epoll.EpollEventLoop","method":"run","file":"EpollEventLoop.java","line":378,"exact":true,"location":"netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar!/","version":"4.1.63.Final"},{"class":"io.netty.util.concurrent.SingleThreadEventExecutor$4","method":"run","file":"SingleThreadEventExecutor.java","line":989,"exact":true,"location":"netty-common-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.util.internal.ThreadExecutorMap$2","method":"run","file":"ThreadExecutorMap.java","line":74,"exact":true,"location":"netty-common-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"io.netty.util.concurrent.FastThreadLocalRunnable","method":"run","file":"FastThreadLocalRunnable.java","line":30,"exact":true,"location":"netty-common-4.1.63.Final.jar!/","version":"4.1.63.Final"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":833,"exact":true,"location":"?","version":"?"}]},"endOfBatch":false,"loggerFqcn":"io.netty.util.internal.logging.LocationAwareSlf4JLogger","contextMap":{},"threadId":104,"threadPriority":5,"messageTimestamp":"2022-04-28T21:18:40.289+0000","ocLogId":"${ctx:ocLogId}","pod":"${ctx:hostname}","processId":"1","instanceType":"prod","ingressTxId":"${ctx:ingressTxId}"}
```

Please let us know how to resolve this issue?


Issue comments:
A `ByteBuf` was unexpectedly released while being worked on by a `ByteToMessageDecoder`.
Whatever class extends the `ByteToMessageDecoder` probably called `release()` on the `in` buffer argument, which it shouldn't do.
The input buffer is managed by the `ByteToMessageDecoder`, so sub-classes should just advance the `readerIndex`.
Hi @chrisvest ,
We have mix of GET & POST in the traffic. We are performing load testing on our application(1500 TPS. for 42 hours)
Things work fine for 42 hours(All requests are processed). Suddenly the application starts dropping the request with following exception
**[8c5489df, L:/10.233.114.202:8443 - R:/10.233.90.102:56316] EXCEPTION: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1**
Attaching the stacktrace for reference.
Need your assistance here to further understand this [behavior]
[StackTrace.txt](https://github.com/netty/netty/files/9242763/StackTrace.txt)


@joyfulnikhil Looks like the same thing. The bug is likely in the `ByteToMessageDecoder` subclass. I've put up a PR to make the error message more helpful.

Pull request review comments:
Can a check for **cumulation.refCount() >0** be placed here? This will ensure that only those bytebuf are released whose refCount>0 and if bytebuf already has a refcount of 0, it is already deallocated or can be returned to the object pool. This will prevent IllegalReferenceCount from occurring. 
Not sure about the implication on memory (Whether this check will cause a memory leak ?)

Patch that resolved the issue:
diff --git a/codec/src/main/java/io/netty/handler/codec/ByteToMessageDecoder.java b/codec/src/main/java/io/netty/handler/codec/ByteToMessageDecoder.java
index 39239ec0e3c..81ee3427ce5 100644
--- a/codec/src/main/java/io/netty/handler/codec/ByteToMessageDecoder.java
+++ b/codec/src/main/java/io/netty/handler/codec/ByteToMessageDecoder.java
@@ -23,6 +23,7 @@
 import io.netty.channel.ChannelHandlerContext;
 import io.netty.channel.ChannelInboundHandlerAdapter;
 import io.netty.channel.socket.ChannelInputShutdownEvent;
+import io.netty.util.IllegalReferenceCountException;
 import io.netty.util.internal.ObjectUtil;
 import io.netty.util.internal.StringUtil;
 
@@ -285,7 +286,15 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception
                 try {
                     if (cumulation != null && !cumulation.isReadable()) {
                         numReads = 0;
-                        cumulation.release();
+                        try {
+                            cumulation.release();
+                        } catch (IllegalReferenceCountException e) {
+                            //noinspection ThrowFromFinallyBlock
+                            throw new IllegalReferenceCountException(
+                                    getClass().getSimpleName() + "#decode() might have released its input buffer, " +
+                                            "or passed it down the pipeline without a retain() call, " +
+                                            "which is not allowed.", e);
+                        }
                         cumulation = null;
                     } else if (++numReads >= discardAfterReads) {
                         // We did enough reads already try to discard some bytes, so we not risk to see a OOME.


Test patch: