Repository: netty/netty
Issue: NetUtil can prevent using Netty due to SecurityManager denial
Labels: defect

@carl-mastrangelo and I recently encountered a regression when upgrading to 4.1.0-CR3 due to a51e2c87, with the same error as #3680 ("Unable to create Channel from class" caused by a `NoClassDefFoundError`). The exception was surprisingly unhelpful; we don't know why Java failed to include the additional cause.

After digging in deeper, we found the problem was due to a `SecurityException` being thrown from a File.exists call in NetUtil:
https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/NetUtil.java#L248

It seems like the previous fix swapped to `doPrivileged()`, which increases the cases that the code will work without an exception, but `SecurityManager`s are still permitted to throw `SecurityException`. Since the call is intended to be optional, if the `SecurityManager` denies the call then the code should probably catch the exception and treat it as if the file does not exist.

A custom `SecurityManager` was being used, but from my reading of `AccessController` and `doPrivileged()` the `SecurityManager` is behaving correctly:

> If that domain does not have the specified permission, an exception is thrown, as usual.

Once we got to this point we worked around the problem by whitelisting `/proc/sys/net/core/somaxconn` in the `SecurityManager`. So there's not an active need for a fix, but others may appreciate one.

The backtrace we saw:

```
Caused by: io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel
    at io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:40)
    at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:316)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:160)
    at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:142)
    at io.grpc.netty.NettyClientTransport.start(NettyClientTransport.java:131)
    at io.grpc.internal.TransportSet$1.run(TransportSet.java:200)
    at io.grpc.internal.TransportSet.scheduleConnection(TransportSet.java:241)
    at io.grpc.internal.TransportSet.obtainActiveTransport(TransportSet.java:166)
    at io.grpc.internal.ManagedChannelImpl$3.getTransport(ManagedChannelImpl.java:381)
    at io.grpc.SimpleLoadBalancerFactory$SimpleLoadBalancer.pickTransport(SimpleLoadBalancerFactory.java:97)
    at io.grpc.internal.ManagedChannelImpl$1.get(ManagedChannelImpl.java:135)
    at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:206)
    at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:101)
    at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:164)
    at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:245)
    at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:225)
    at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:186)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
    <snip>
    ... 95 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.DefaultChannelId
    at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:109)
    at io.netty.channel.AbstractChannel.<init>(AbstractChannel.java:81)
    at io.netty.channel.nio.AbstractNioChannel.<init>(AbstractNioChannel.java:82)
    at io.netty.channel.nio.AbstractNioByteChannel.<init>(AbstractNioByteChannel.java:52)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:97)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:87)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:80)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:73)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at java.lang.Class.newInstance(Class.java:443)
    at io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
    ... 116 more
```

Debugging found `Security policy violation: ("java.io.FilePermission" "/proc/sys/net/core/somaxconn" "read")` at:

```
    <snip>SecurityManager.checkRead(...)
    at java.io.File.exists(File.java:814)
    at io.netty.util.NetUtil$1.run(NetUtil.java:248)
    at io.netty.util.NetUtil$1.run(NetUtil.java:239)
    at java.security.AccessController.doPrivileged(Native Method)
    at io.netty.util.NetUtil.<clinit>(NetUtil.java:239)
    at io.netty.util.internal.MacAddressUtil.bestAvailableMac(MacAddressUtil.java:53)
    at io.netty.channel.DefaultChannelId.defaultMachineId(DefaultChannelId.java:124)
    at io.netty.channel.DefaultChannelId.<clinit>(DefaultChannelId.java:101)
    at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:109)
    at io.netty.channel.AbstractChannel.<init>(AbstractChannel.java:81)
    at io.netty.channel.nio.AbstractNioChannel.<init>(AbstractNioChannel.java:82)
    at io.netty.channel.nio.AbstractNioByteChannel.<init>(AbstractNioByteChannel.java:52)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:97)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:87)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:80)
    at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:73)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at java.lang.Class.newInstance(Class.java:443)
    at io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
```


Issue comments:
@ejona86 - Good find ... seems like a bug. You went through all the work to find it ... do you want to submit the patch and get credit for the fix too?

https://docs.oracle.com/javase/7/docs/api/java/security/AccessController.html#doPrivileged(java.security.PrivilegedAction)

> If the action's run method throws an (unchecked) exception, it will propagate through this method.

Sure. I'll send a PR

@ejona86 let me take this over from you...

@normanmaurer, fine with me. Enjoy :)

Fixed by https://github.com/netty/netty/pull/4977#issuecomment-196245202


Pull request review comments:

Patch that resolved the issue:
diff --git a/common/src/main/java/io/netty/util/NetUtil.java b/common/src/main/java/io/netty/util/NetUtil.java
index 5830435f7b7..fdadd53b0c7 100644
--- a/common/src/main/java/io/netty/util/NetUtil.java
+++ b/common/src/main/java/io/netty/util/NetUtil.java
@@ -245,28 +245,31 @@ public Integer run() {
                 // - Linux and Mac OS X: 128
                 int somaxconn = PlatformDependent.isWindows() ? 200 : 128;
                 File file = new File("/proc/sys/net/core/somaxconn");
-                if (file.exists()) {
-                    BufferedReader in = null;
-                    try {
+                BufferedReader in = null;
+                try {
+                    // file.exists() may throw a SecurityException if a SecurityManager is used, so execute it in the
+                    // try / catch block.
+                    // See https://github.com/netty/netty/issues/4936
+                    if (file.exists()) {
                         in = new BufferedReader(new FileReader(file));
                         somaxconn = Integer.parseInt(in.readLine());
                         if (logger.isDebugEnabled()) {
                             logger.debug("{}: {}", file, somaxconn);
                         }
-                    } catch (Exception e) {
-                        logger.debug("Failed to get SOMAXCONN from: {}", file, e);
-                    } finally {
-                        if (in != null) {
-                            try {
-                                in.close();
-                            } catch (Exception e) {
-                                // Ignored.
-                            }
+                    } else {
+                        if (logger.isDebugEnabled()) {
+                            logger.debug("{}: {} (non-existent)", file, somaxconn);
                         }
                     }
-                } else {
-                    if (logger.isDebugEnabled()) {
-                        logger.debug("{}: {} (non-existent)", file, somaxconn);
+                } catch (Exception e) {
+                    logger.debug("Failed to get SOMAXCONN from: {}", file, e);
+                } finally {
+                    if (in != null) {
+                        try {
+                            in.close();
+                        } catch (Exception e) {
+                            // Ignored.
+                        }
                     }
                 }
                 return somaxconn;


Test patch: