Repository: netty/netty
Issue: Event loop threads hang in SniHandler for SSL records with majorVersion != 3
Labels: defect

Under some unknown and rare circumstances 2 worker threads started to consume 100% CPU time of a single core each, permanently and indefinitely. It didn't prevent the app from its normal functioning, nevertheless excessive consumption of cpu is not a normal behaviour.
Noticed this behaviour once by now.

### Actual behavior
Epoll transport enabled. Two threads (out of four) burn 100% CPU time, each has the following stacktrace:

"epollEventLoopGroup-7-1" #20 prio=10 os_prio=0 cpu=120413943.05ms elapsed=889961.34s tid=0x00007fa110003000 nid=0x4381 runnable [0x00007fa124c2d000]
java.lang.Thread.State: RUNNABLE

"epollEventLoopGroup-7-3" #22 prio=10 os_prio=0 cpu=55102833.55ms elapsed=889961.34s tid=0x00007fa110008800 nid=0x4383 runnable [0x00007fa124a2b000]


```
"epollEventLoopGroup-7-3" #22 prio=10 os_prio=0 cpu=54941845.34ms elapsed=889798.92s tid=0x00007fa110008800 nid=0x4383 runnable  [0x00007fa124a2b000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1432)
        at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1419)
        at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1414)
        at io.netty.buffer.AbstractByteBuf.getByte(AbstractByteBuf.java:356)
        at io.netty.buffer.AbstractByteBuf.getUnsignedByte(AbstractByteBuf.java:369)
        at io.netty.handler.ssl.AbstractSniHandler.decode(AbstractSniHandler.java:62)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:493)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@12.0.2/Thread.java:835)
```

### Steps to reproduce
Haven't yet found the scenario.

### Netty version
4.1.44

### JVM version (e.g. `java -version`)
openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+9-Debian-1)
OpenJDK 64-Bit Server VM (build 12.0.2+9-Debian-1, mixed mode, sharing)

### OS version (e.g. `uname -a`)
Linux hostname 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3 (2019-09-02) x86_64 GNU/Linux


Issue comments:
Same problem repeated once again on the same server. 
```
"epollEventLoopGroup-7-3" #22 prio=10 os_prio=0 cpu=67508345.04ms elapsed=617139.89s tid=0x00007f5b68006000 nid=0x3de5 runnable  [0x00007f5b59674000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1432)
        at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1419)
        at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1414)
        at io.netty.buffer.AbstractByteBuf.getByte(AbstractByteBuf.java:356)
        at io.netty.buffer.AbstractByteBuf.getUnsignedByte(AbstractByteBuf.java:369)
        at io.netty.handler.ssl.AbstractSniHandler.decode(AbstractSniHandler.java:62)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:493)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@12.0.2/Thread.java:835)
```
Thanks @igor-lukyanov, I think I found the problem (introduced in 4.1.44.Final) - see #9933. I have renamed this issue to better reflect the bug, hope that's ok.

Pull request review comments:
add `fail();` above
@normanmaurer done, I actually just copied the existing `testFallbackToDefaultContext()` as a starting point, which had this `fail()` commented out for some reason. So I've uncommented it there too since an exception does also get thrown in that case.
@njhill thanks... Lets see if this works for all JDK versions that we use on the CI :) If so we can close the issue as well. 
@njhill after some more digging this actually depends on if `jdkCompatibilityMode` is true of not. Let me do a follow up to fix it and then investigate later on.

Patch that resolved the issue:
diff --git a/handler/src/main/java/io/netty/handler/ssl/AbstractSniHandler.java b/handler/src/main/java/io/netty/handler/ssl/AbstractSniHandler.java
index 8cd9bf0f861..fa7e9059b9d 100644
--- a/handler/src/main/java/io/netty/handler/ssl/AbstractSniHandler.java
+++ b/handler/src/main/java/io/netty/handler/ssl/AbstractSniHandler.java
@@ -153,8 +153,9 @@ protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) t
                                     select(ctx, extractSniHostname(handshakeBuffer, 0, handshakeLength));
                                     return;
                                 }
+                                break;
                             }
-                            break;
+                            // fall-through
                         default:
                             // not tls, ssl or application data, do not try sni
                             select(ctx, null);


Test patch:
diff --git a/handler/src/test/java/io/netty/handler/ssl/SniHandlerTest.java b/handler/src/test/java/io/netty/handler/ssl/SniHandlerTest.java
index 8cd3d766cda..3f468db0dd8 100644
--- a/handler/src/test/java/io/netty/handler/ssl/SniHandlerTest.java
+++ b/handler/src/test/java/io/netty/handler/ssl/SniHandlerTest.java
@@ -320,7 +320,7 @@ public void testFallbackToDefaultContext() throws Exception {
                 ch.writeInbound(Unpooled.wrappedBuffer(message));
                 // TODO(scott): This should fail because the engine should reject zero length records during handshake.
                 // See https://github.com/netty/netty/issues/6348.
-                // fail();
+                fail();
             } catch (Exception e) {
                 // expected
             }
@@ -344,6 +344,43 @@ public void testFallbackToDefaultContext() throws Exception {
         }
     }
 
+    @Test(timeout = 10000)
+    public void testMajorVersionNot3() throws Exception {
+        SslContext nettyContext = makeSslContext(provider, false);
+
+        try {
+            DomainNameMapping<SslContext> mapping = new DomainNameMappingBuilder<SslContext>(nettyContext).build();
+
+            SniHandler handler = new SniHandler(mapping);
+            EmbeddedChannel ch = new EmbeddedChannel(handler);
+
+            // invalid
+            byte[] message = {22, 2, 0, 0, 0};
+            try {
+                // Push the handshake message.
+                ch.writeInbound(Unpooled.wrappedBuffer(message));
+                fail();
+            } catch (Exception e) {
+                // expected
+            }
+
+            ch.close();
+
+            // When the channel is closed the SslHandler will write an empty buffer to the channel.
+            ByteBuf buf = ch.readOutbound();
+            if (buf != null) {
+                assertFalse(buf.isReadable());
+                buf.release();
+            }
+
+            assertThat(ch.finish(), is(false));
+            assertThat(handler.hostname(), nullValue());
+            assertThat(handler.sslContext(), is(nettyContext));
+        } finally {
+            releaseAll(nettyContext);
+        }
+    }
+
     @Test
     public void testSniWithApnHandler() throws Exception {
         SslContext nettyContext = makeSslContext(provider, true);