Repository: netty/netty
Issue: Throwing an exception in case glibc is missing instead of segfaulting the JVM

It seems Netty's native support depends on glibc (btw is this documented?).
When glibc is missing (for example using vanilla Alpine), the JVM could end up with a crash (`SIGSEGV`), see details in this issue: https://github.com/micrometer-metrics/micrometer/issues/2776

It also seems that `Epoll.isAvailable()` returns `true` even if glibc is missing.

### Expected behavior
Throwing an exception

### Actual behavior
JVM crash

### Steps to reproduce
Using Netty native with an environment that does not have glibc, e.g.: vanilla Alpine

### Minimal yet complete reproducer code (or URL to code)
This seems to be a known issue, please let me know if you really need a reproducer.

### Netty version
`4.1.68` (latest)

### JVM version (e.g. `java -version`)
Latest 11:
```bash
 ❯ docker run --rm 'azul/zulu-openjdk-alpine:11-jre' 'sh' '-c' 'java --version'
openjdk 11.0.12 2021-07-20 LTS
OpenJDK Runtime Environment Zulu11.50+19-CA (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM Zulu11.50+19-CA (build 11.0.12+7-LTS, mixed mode)
```
Or also latest 17:
```bash
 ❯ docker run --rm 'bellsoft/liberica-openjdk-alpine-musl:17' 'sh' '-c' 'java --version'
openjdk 17 2021-09-14 LTS
OpenJDK Runtime Environment (build 17+35-LTS)
OpenJDK 64-Bit Server VM (build 17+35-LTS, mixed mode)
```
I assume this is an issue in every JRE that is currently supported.

### OS version (e.g. `uname -a`)
Both images above have the same output:
```bash
 ❯ docker run --rm 'azul/zulu-openjdk-alpine:11-jre' 'sh' '-c' 'uname -a'
Linux 1ee8d1090f14 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 Linux
```
I assume this is an issue in every OS where glibc is missing.

Issue comments:
The binary compatibility baseline for our native code is the GLIBC version that's used on CentOS 6. The work-around is to use the Nio transport.

It's not clear from the error reports how this is really showing up. If it's an ABI problem, then I'm not sure what we can do to mitigate it. If we are calling glibc specific functions, then I wonder if we can somehow check what libc we are using at runtime.
>It's not clear from the error reports how this is really showing up.

I'm not sure what your question is. The error happens when Netty native is used without having glibc.

>If it's an ABI problem...

I don't think it is an ABI problem since glibc is missing in Alpine, there is nothing to be compatible with (and based on [this comment ](https://github.com/netty/netty/issues/6841#issuecomment-307716433), other libc implementations are out of question).

I think checking if glibc is available or not and/or fixing `Epoll.isAvailable()` would fix the issue (it seems it returns true even if glibc is missing).
I put together a little test for this: https://github.com/jonatan-ivanov/netty-gh-11701

I called `Epoll.isAvailable()` and `Epoll.ensureAvailability()` on different distros to see if they return false/fail in case glibc is missing. It seems `isAvailable` does not return false and `ensureAvailability` does not fail even if glibc is missing.

Image | `isAvailable` | `ensureAvailability` | Comment
------ | ------------------- |  --------------------------- | -------------
bellsoft_liberica-openjre-debian:17 | `true` | `OK` | works
bellsoft_liberica-openjre-alpine:17  | `true` | `OK` | works since it contains glibc
**bellsoft_liberica-openjre-alpine-musl:17**  | `true` | `OK` | **does not work as expected**
azul_zulu-openjdk:17 | `true` | `OK` | works as expected
**azul_zulu-openjdk-alpine:17-jre**  | `true` | `OK` | **does not work as expected**


There's no way to detect the libc vendor from within a process, as far as I can see. And calling out to `lld`, or something like that, is quite awkward and can be a security liability. I think the best we can do for the moment is to document that only the Nio transport is supported on musl.

What do you think, @normanmaurer ?
I agree with @chrisvest here 
I'm not very familiar with this domain but do you think is it at least possible to throw an exception instead of crashing a JVM if such a thing occur?

Also can you use the `__GLIBC__ ` from [`features.h`](https://sourceware.org/git/?p=glibc.git;a=blob;f=include/features.h;h=d974eabfafc24ffb9ff4cd066b650134df32048b;hb=refs/heads/master#l477) to detect if you are dealing with glibc or not or is this only possible compile time?

@jonatan-ivanov That's only at compile time. At runtime you can feature detect by inspecting linked symbols, except you cant do that without calling into libc which in this case would cause a crash. Alternatively we shell out to a program to inspect libc, but that is also bad for multiple reasons, and we don't want to do that.
I have updated the documentation on the native transports to note that musl is not officially supported: https://github.com/netty/netty/wiki/Native-transports

I think that's the best we can do for now.
musl generally tries to support glibc-linked binaries invoking POSIX and some BSD extensions. errors related to GNU-specific extensions (and glibc header bastardizations) should be caught at load time, as in #6841 "__strndup: symbol not found". that particular issue can be fixed by installing gcompat package in alpine linux. segfaults are, as you say, caused by some ABI incompatibility.

i briefly glanced through netty c code and found some suspicious items: https://github.com/netty/netty/blob/6b3ec62e6e4e19a85304aecd8cb5331a3fcc70be/transport-native-unix-common/src/main/c/netty_unix_errors.c#L41 is suspicious: application code should never check feature test macros, because it should be defining it themselves. https://github.com/netty/netty/blob/51ebcbd9aba7eac854716e45f07765744f8dcdaf/transport-native-epoll/src/main/c/netty_epoll_native.c#L140 is suspicious: sysctl values should always be accessed using `read`/`write`, never stdio. some sysctls will not be read correctly if using stdio. neither of these should cause segfault at run time though.
@Hello71 The netty_unit_errors code is because we wish to use a thread-safe strerror variant, which is not immediately portable. (I'm not familiar with the other one you point out)
If you know of a way to improve it, we'd love to review a PR!
Fix reverted because it negatively impacts musl-based systems that also use the glibc compatibility layer.
@chrisvest Would you consider reopening this issue so that we can track another fix (if any)?
reopened 
fwiw if you want to do this in the dumbest possible way, you can check for `gnu_get_libc_version`. if your program is segfaulting and not just failing with symbol not found, then most likely there is some more serious issue with your program, not related to musl.
@Hello71 The JVM itself might rely on signals and tie them to internal semantics, i.e. rare NullPointerExceptions sometimes rely on segfaults to optimise a non-null fast path. So I wouldn't feel very confident trying to trap signals in JNI library code.
I think this would be fixed by https://github.com/netty/netty/pull/12272 ... 

Pull request review comments:
either we should rename the variable or rename the property
nit: we could just move the whole `throw` in the if block that is contained in the while loop and so get rid of the boolean at all.

Patch that resolved the issue:
diff --git a/transport-native-epoll/src/main/c/netty_epoll_native.c b/transport-native-epoll/src/main/c/netty_epoll_native.c
index 5fed0889596..531d13879b8 100644
--- a/transport-native-epoll/src/main/c/netty_epoll_native.c
+++ b/transport-native-epoll/src/main/c/netty_epoll_native.c
@@ -326,6 +326,7 @@ static jint netty_epoll_native_epollCtlAdd0(JNIEnv* env, jclass clazz, jint efd,
     }
     return res;
 }
+
 static jint netty_epoll_native_epollCtlMod0(JNIEnv* env, jclass clazz, jint efd, jint fd, jint flags) {
     int res = epollCtl(env, efd, EPOLL_CTL_MOD, fd, flags);
     if (res < 0) {
@@ -540,6 +541,15 @@ static jstring netty_epoll_native_kernelVersion(JNIEnv* env, jclass clazz) {
     return NULL;
 }
 
+static jint netty_epoll_native_gnulibc(JNIEnv* env, jclass clazz) {
+#ifdef __GLIBC__
+    return 1;
+#else
+    // We are using an alternative libc, possibly musl but could be anything.
+    return 0;
+#endif // __GLIBC__
+}
+
 static jboolean netty_epoll_native_isSupportingSendmmsg(JNIEnv* env, jclass clazz) {
     if (SYS_sendmmsg == -1) {
         return JNI_FALSE;
@@ -658,7 +668,8 @@ static const JNINativeMethod statically_referenced_fixed_method_table[] = {
   { "isSupportingSendmmsg", "()Z", (void *) netty_epoll_native_isSupportingSendmmsg },
   { "isSupportingRecvmmsg", "()Z", (void *) netty_epoll_native_isSupportingRecvmmsg },
   { "tcpFastopenMode", "()I", (void *) netty_epoll_native_tcpFastopenMode },
-  { "kernelVersion", "()Ljava/lang/String;", (void *) netty_epoll_native_kernelVersion }
+  { "kernelVersion", "()Ljava/lang/String;", (void *) netty_epoll_native_kernelVersion },
+  { "gnulibc", "()I", (void *) netty_epoll_native_gnulibc }
 };
 static const jint statically_referenced_fixed_method_table_size = sizeof(statically_referenced_fixed_method_table) / sizeof(statically_referenced_fixed_method_table[0]);
 static const JNINativeMethod fixed_method_table[] = {
diff --git a/transport-native-epoll/src/main/java/io/netty/channel/epoll/Native.java b/transport-native-epoll/src/main/java/io/netty/channel/epoll/Native.java
index 8a409dd108f..775a037f50e 100644
--- a/transport-native-epoll/src/main/java/io/netty/channel/epoll/Native.java
+++ b/transport-native-epoll/src/main/java/io/netty/channel/epoll/Native.java
@@ -23,11 +23,15 @@
 import io.netty.util.internal.ClassInitializerUtil;
 import io.netty.util.internal.NativeLibraryLoader;
 import io.netty.util.internal.PlatformDependent;
+import io.netty.util.internal.SystemPropertyUtil;
 import io.netty.util.internal.ThrowableUtil;
 import io.netty.util.internal.logging.InternalLogger;
 import io.netty.util.internal.logging.InternalLoggerFactory;
 
+import java.io.BufferedReader;
+import java.io.FileInputStream;
 import java.io.IOException;
+import java.io.InputStreamReader;
 import java.nio.channels.FileChannel;
 import java.nio.channels.Selector;
 
@@ -36,6 +40,7 @@
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.epollin;
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.epollout;
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.epollrdhup;
+import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.gnulibc;
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.isSupportingRecvmmsg;
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.isSupportingSendmmsg;
 import static io.netty.channel.epoll.NativeStaticallyReferencedJniMethods.kernelVersion;
@@ -50,6 +55,7 @@
  * <p>Static members which call JNI methods must be defined in {@link NativeStaticallyReferencedJniMethods}.
  */
 public final class Native {
+    private static final boolean checkMusl = SystemPropertyUtil.getBoolean("io.netty.native.musl.check", true);
     private static final InternalLogger logger = InternalLoggerFactory.getInstance(Native.class);
 
     static {
@@ -77,9 +83,9 @@ public final class Native {
         );
 
         try {
-            // First, try calling a side-effect free JNI method to see if the library was already
+            // First, try calling a side effect free JNI method to see if the library was already
             // loaded by the application.
-            offsetofEpollData();
+            gnulibc();
         } catch (UnsatisfiedLinkError ignore) {
             // The library was not previously loaded, load it now.
             loadNativeLibrary();
@@ -92,6 +98,33 @@ public final class Native {
                 // Just ignore
             }
         }
+        if (checkMusl && gnulibc() == 1) {
+            // Our binary is compiled for linking with GLIBC.
+            // Let's check that we don't have anything that looks like Musl libc in our runtime.
+            try {
+                FileInputStream fis = new FileInputStream("/proc/self/maps");
+                try {
+                    BufferedReader reader = new BufferedReader(new InputStreamReader(fis));
+                    String line;
+                    while ((line = reader.readLine()) != null) {
+                        if (line.contains("-musl-")) {
+                            throw new LinkageError("Native library was compiled for linking with GLIBC, but GLIBC " +
+                                    "was not found among library mappings. This likely means the OS/JVM uses an " +
+                                    "alternative libc, such as musl. To fix, either use NIO transport, or build a " +
+                                    "native transport for your platform.");
+                        }
+                    }
+                } finally {
+                    try {
+                        fis.close();
+                    } catch (IOException e) {
+                        logger.debug("Failed to close /proc/self/maps file.", e);
+                    }
+                }
+            } catch (IOException e) {
+                logger.debug("Unable to check libc compatibility.", e);
+            }
+        }
         Unix.registerInternal(new Runnable() {
             @Override
             public void run() {
diff --git a/transport-native-epoll/src/main/java/io/netty/channel/epoll/NativeStaticallyReferencedJniMethods.java b/transport-native-epoll/src/main/java/io/netty/channel/epoll/NativeStaticallyReferencedJniMethods.java
index b52721b26f6..ba1df58e815 100644
--- a/transport-native-epoll/src/main/java/io/netty/channel/epoll/NativeStaticallyReferencedJniMethods.java
+++ b/transport-native-epoll/src/main/java/io/netty/channel/epoll/NativeStaticallyReferencedJniMethods.java
@@ -43,4 +43,5 @@ private NativeStaticallyReferencedJniMethods() { }
     static native boolean isSupportingRecvmmsg();
     static native int tcpFastopenMode();
     static native String kernelVersion();
+    static native int gnulibc();
 }


Test patch: