Repository: netty/netty
Issue: aarch64 native bits.

Hi,

It is valuable for me to have native aarch64 bits included in the release artifacts. I see there has been interest in a cross-compile previously: https://github.com/netty/netty/issues/7459 

It appears that the release is made on a RHEL 6 x86_64 compatible and I would like to explore making the necessary changes to the project to add in a cross-compile step for aarch64.

Assuming this is accurate, is the project open to a pull request to generate aarch64 bits during the release cycle?

best regards,
Richard

Issue comments:
@rhenwood-arm if we can easily cross-compile there we could do it I guess.
@rhenwood-arm btw if you do so you may also be interested in providing a pr for netty-tcnative.
Thanks for the prompt response and guidance @normanmaurer !

I'll spend some time looking into this in detail.
@rhenwood-arm any update ?
@rhenwood-arm ping
Hi @normanmaurer 
I am working with a colleague on this activity, so I've reached out to them to see where they are. Still waiting for an update.

NOTE: I am assuming that this is a 'nice to have' for the project today. Please let me know if you have observed other interest in this work which will help me prioritize it.

@rhenwood-arm you are right... nice to have and not urgent at all. Just wanted to see where we are here.
We are using netty on ARM64v8.   and waiting this very important feature.
Hi @rhenwood-arm do you have start some work around this topic ? I'm also interested to provide cross compilation for arm v7 32 bit architecture. 

Hi,

I don't have time to work on this currently, but I will ask to see if I can get some help...
It looks like the EPEL 6 repository would need to be added to the release requirement to at least provide the GCC cross compilers for AArch64 and Armv7.

@normanmaurer would you be amenable to that?

@rhenwood-arm sounds fair :)
@rhenwood-arm ping.. 
Thanks for the reminder @normanmaurer . At a high level, I think the plan to move forward is:

1. Add EPEL as a release dependency to this project. This will provide the cross compiler.
2. Update the pom here: https://github.com/netty/netty/blob/4.1/transport-native-epoll/pom.xml so that it uses the cross compiler to build for AArch64 and Armv7 as well.
3. Update these instructions: https://netty.io/wiki/releasing-new-version.html so the new native libraries are correctly packaged during release.

Can this plan be improved?

I do not currently have time to work on this. I see if I can get some help. @aygalinc do you have time to help?
We have the same requirement about netty on aarch64 as well. Any progress? Anything I can do to help with the ARM release?
@rhenwood-arm @normanmaurer Can I take over this issue? I'd like to do the work that @rhenwood-arm pointed.
Sure 🙏

> Am 13.11.2019 um 00:46 schrieb wangxiyuan <notifications@github.com>:
> 
> ﻿
> @rhenwood-arm @normanmaurer Can I take over this issue? I'd like to do the work that @rhenwood-arm pointed.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@wangxiyuan : please do! :)
I tested locally with java_1.8.0_222 and maven 3.6.0 on an ARM machine. The build and tests for 4.1.44 are passed at all. I'll test cross-compile next. I'll create the PR for it if everything works well.

@rhenwood-arm @normanmaurer I tested cross compile locally. The official yum package for `gcc-aarch64-linux-gnu` doesn't work. It missed lots of files. I used linaro ones instead. The minimum version is 4.9: https://releases.linaro.org/components/toolchain/binaries/4.9-2016.02/aarch64-linux-gnu/ and it relies on glibc 2.14 which CentOS 6.9 doesn't support by default. So I bumped glibc from `2.13` to `2.14` by hand. Then with the linaro gcc, the cross compile passed. 

As you can see, since the tools on CentOS 6 is a little old by default, we can't cross compile by default. How do you think about the step? Is it acceptable?
@wangxiyuan unfortunately bumping up GLIBC is not an option imho. So what we could do is to only support it on centos7 and newer for arm. 
@normanmaurer Using CentOS 7 is a better way of cause.

I'm not sure how `transport-native-epoll` jar is released to Maven Repo. But I find that it relies on CentOS 6.9 if I'm correct. https://github.com/netty/netty/blob/4.1/transport-native-epoll/pom.xml#L96
Is it possible to bump to 7?
Nope it is not possible to require centos7 in general. I wonder if we can just build on both and release

> Am 20.11.2019 um 10:16 schrieb wangxiyuan <notifications@github.com>:
> 
> ﻿
> @normanmaurer Using CentOS 7 is a better way of cause.
> 
> I'm not sure how transport-native-epoll jar is released to Maven Repo. But I find that it relies on CentOS 6.9 if I'm correct. https://github.com/netty/netty/blob/4.1/transport-native-epoll/pom.xml#L96
> Is it possible to bump to 7?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@normanmaurer make sense. Let me try 7 now.

BTW, I'm not sure how to support both 6 and 7 in netty. Will there be two jar, one for 6 and another for 7?
@wangxiyuan there would be one jar for arm and one for x86 I guess . And the one for arm would be centos7 ?
The glibc version seems to be the critical factor in providing architecture support. I suggest separate multiple releases of the same code-base are distinguished with that in-mind. One obvious approach is to include the glibc version in the release name. So far, I personally think will be the simplest way to avoid confusion and gives a good path to future-proofing new architectures.

I can also see labeling with centos6, centos7 etc (or el6, el7, el8) would be an OK alternative for me personally... but I do recognize there are other distributions out there that people might choose to run :)
@rhenwood-arm @normanmaurer I just added a new profile `linux-aarch64` for `transport-native-epoll` and its dependents `transport-native-unix-common` to build the aarch64 packge. I tested it locally already. Here is the step to cross compile:
1. Make sure the OS is CentOS 7.6 (which contains glibc>=2.14  by default )
2. Download aarch64-gcc toolchin  https://releases.linaro.org/components/toolchain/binaries/4.9-2016.02/aarch64-linux-gnu/gcc-linaro-4.9-2016.02-x86_64_aarch64-linux-gnu.tar.xz
3. Unpack it and add its `bin` folder into the user's default PATH( `maven-hawtjni-plugin` will use the default PATH to find aarch64-gcc, `export` doesn't work.)
4. go into `transport-native-unix-common`, try `mvn clean install -Plinux-aarch64`.
5. go into `transport-native-epoll`, try `mvn clean install -Plinux-aarch64`.

Is that OK?

And I don't know how netty releases jar to maven center. Should I add some release change to parent pom.xml?
@wangxiyuan left a comment on the PR. I think we should add a docker file for this and then we can make it happen as part of the release process.
sure, Let me add one. Thanks for your quick response.
Any progress on this?  This seems to build fine on an AWS a1 machine running RHEL8.  Getting the binaries for aarch64 released would be helpful for people wanting to run Hadoop/Spark etc on Arm machines.

Pull request review comments:
This complete plugin config is not needed anymore :) Just remove it
I don't know how to release the package. Any suggestion?
Done
@wangxiyuan can you make all the docker / docker-compose stuff consistent with what we have atm ? If you don't have time I can also do it tho.
I can but I don't know how to config the release management. So if you can take over this job. I'll be very appreciate. Thanks.
I will
Any progress, or anything elase I can do? I can update the dockerfile of cause.
s/Dockfile/Dockerfile/
Is there a https mirror available?
Actually we can drop this completely, Netty has mvnw (Maven wrapper) included in the netty repo.
./mvnw here instead of mvn
x86_64 this reads 32bit..
extract all occurences of 4.9-2016.02 into variable? eg. ENV TOOLCHAIN_VERSION=4.9-2016.02
why was this dependency included? shouldn't it be handled here? https://github.com/netty/netty/blob/a9731b43e0e86359319233416fa815ed3a79c564/transport-native-epoll/pom.xml#L128
This is github display problem. See L347. I added the `dependencies` there which will be handled by L275
Will do.
@wangxiyuan why are we skipping tests here?
Since this is cross compile, the test environment is still on X86. So when run test, it still require `libnetty_tcnative_linux_x86_64.so` which will lead the error like

```
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64_fedora, netty_tcnative_linux_x86_64, netty_tcnative_x86_64, netty_tcnative]
```

can you add a comment explaining that?
```suggestion
  # since we are cross compiling netty_tcnative as arch64 it cannot be loaded on x86_64 -> skipTests
  cross-compile-aarch64:
```
I added a suggested change.
Add it at L55. Please take a look
this should be removed... we should expect that will be present and mount it via our docker file just as we do for other docker-compose things. 
Done

Patch that resolved the issue:
diff --git a/all/pom.xml b/all/pom.xml
index afbdcb2a052..ebf7d21dc42 100644
--- a/all/pom.xml
+++ b/all/pom.xml
@@ -57,6 +57,14 @@
           <scope>compile</scope>
           <optional>true</optional>
         </dependency>
+        <dependency>
+          <groupId>${project.groupId}</groupId>
+          <artifactId>netty-transport-native-epoll</artifactId>
+          <version>${project.version}</version>
+          <classifier>linux-aarch_64</classifier>
+          <scope>compile</scope>
+          <optional>true</optional>
+        </dependency>
         <dependency>
           <groupId>${project.groupId}</groupId>
           <artifactId>netty-transport-native-kqueue</artifactId>
@@ -89,6 +97,14 @@
           <scope>compile</scope>
           <optional>true</optional>
         </dependency>
+        <dependency>
+          <groupId>${project.groupId}</groupId>
+          <artifactId>netty-transport-native-epoll</artifactId>
+          <version>${project.version}</version>
+          <classifier>linux-aarch_64</classifier>
+          <scope>compile</scope>
+          <optional>true</optional>
+        </dependency>
         <dependency>
           <groupId>${project.groupId}</groupId>
           <artifactId>netty-transport-native-kqueue</artifactId>
diff --git a/docker/Dockerfile.cross_compile_aarch64 b/docker/Dockerfile.cross_compile_aarch64
new file mode 100644
index 00000000000..077e46262a3
--- /dev/null
+++ b/docker/Dockerfile.cross_compile_aarch64
@@ -0,0 +1,18 @@
+FROM centos:7.6.1810
+
+ARG gcc_version=4.9-2016.02
+ENV GCC_VERSION $gcc_version
+
+# Install requirements
+RUN yum install -y wget tar git make redhat-lsb-core autoconf automake libtool glibc-devel libaio-devel openssl-devel apr-devel lksctp-tools
+
+# Install Java
+RUN yum install -y java-1.8.0-openjdk-devel
+
+# Install aarch64 gcc toolchain
+RUN set -x && \
+  wget https://releases.linaro.org/components/toolchain/binaries/$GCC_VERSION/aarch64-linux-gnu/gcc-linaro-$GCC_VERSION-x86_64_aarch64-linux-gnu.tar.xz && \
+  tar xvf gcc-linaro-$GCC_VERSION-x86_64_aarch64-linux-gnu.tar.xz
+
+ENV PATH="/gcc-linaro-$GCC_VERSION-x86_64_aarch64-linux-gnu/bin:${PATH}"
+ENV JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk/"
diff --git a/docker/README.md b/docker/README.md
index d96242b3212..378c673c844 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -16,4 +16,11 @@ docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.centos-6.1
 docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.centos-7.111.yaml run test
 ```
 
+## aarch64 cross compile for transport-native-epoll on X86_64
+
+```
+docker-compose -f docker/docker-compose.yaml run cross-compile-aarch64
+```
+The default version of aarch64 gcc is `4.9-2016.02`. Update the parameter `gcc_version` in `docker-compose.yaml` to use a version you want.
+
 etc, etc
diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml
index 5d9ec200ae5..f56cb53aaab 100644
--- a/docker/docker-compose.yaml
+++ b/docker/docker-compose.yaml
@@ -40,3 +40,23 @@ services:
       - ..:/code:delegated
       - ~/.m2:/root/.m2:delegated
     entrypoint: /bin/bash
+
+  cross-compile-aarch64-runtime-setup:
+    image: netty:cross_compile_aarch64
+    build:
+      context: .
+      dockerfile: Dockerfile.cross_compile_aarch64
+      args:
+        gcc_version : "4.9-2016.02"
+
+  cross-compile-aarch64:
+    image: netty:cross_compile_aarch64
+    depends_on: [cross-compile-aarch64-runtime-setup]
+    volumes:
+      - ~/.ssh:/root/.ssh:delegated
+      - ~/.gnupg:/root/.gnupg:delegated
+      - ..:/code:delegated
+      - ~/.m2:/root/.m2:delegated
+    # Since we are cross compiling netty-transport-native-epoll as aarch64 which cannot be loaded on x86_64, we add `skipTests` here to skip the test.
+    command: /bin/bash -cl "pushd ./transport-native-unix-common && ../mvnw clean install -Plinux-aarch64 && popd && pushd ./transport-native-epoll && ../mvnw clean install -Plinux-aarch64 -DskipTests && popd"
+    working_dir: /code
diff --git a/transport-native-epoll/pom.xml b/transport-native-epoll/pom.xml
index 5ae8bde4d50..5c94a8511cb 100644
--- a/transport-native-epoll/pom.xml
+++ b/transport-native-epoll/pom.xml
@@ -196,6 +196,154 @@
         </plugins>
       </build>
   
+      <dependencies>
+        <dependency>
+          <groupId>io.netty</groupId>
+          <artifactId>netty-transport-native-unix-common</artifactId>
+          <version>${project.version}</version>
+          <classifier>${jni.classifier}</classifier>
+          <!--
+            The unix-common with classifier dependency is optional because it is not a runtime dependency, but a build time
+            dependency to get the static library which is built directly into the shared library generated by this project.
+          -->
+          <optional>true</optional>
+        </dependency>
+      </dependencies>
+    </profile>
+    <profile>
+      <id>linux-aarch64</id>
+      <properties>
+        <jni.classifier>${os.detected.name}-aarch64</jni.classifier>
+      </properties>
+      <build>
+        <pluginManagement>
+          <plugins>
+            <plugin>
+              <artifactId>maven-enforcer-plugin</artifactId>
+              <version>1.4.1</version>
+              <dependencies>
+                <!-- Provides the 'requireFilesContent' enforcer rule. -->
+                <dependency>
+                  <groupId>com.ceilfors.maven.plugin</groupId>
+                  <artifactId>enforcer-rules</artifactId>
+                  <version>1.2.0</version>
+                </dependency>
+              </dependencies>
+            </plugin>
+          </plugins>
+        </pluginManagement>
+        <plugins>
+          <plugin>
+            <artifactId>maven-enforcer-plugin</artifactId>
+            <executions>
+              <execution>
+                <id>enforce-release-environment</id>
+                <goals>
+                  <goal>enforce</goal>
+                </goals>
+                <configuration>
+                  <rules>
+                    <requireProperty>
+                      <regexMessage>
+                        Cross compile and Release process must be performed on linux-x86_64.
+                      </regexMessage>
+                      <property>os.detected.classifier</property>
+                      <regex>^linux-x86_64.*</regex>
+                    </requireProperty>
+                    <requireFilesContent>
+                      <message>
+                        Cross compile and Release process must be performed on RHEL 7.6 or its derivatives.
+                      </message>
+                      <files>
+                        <file>/etc/redhat-release</file>
+                      </files>
+                      <content>release 7.6</content>
+                    </requireFilesContent>
+                  </rules>
+                </configuration>
+              </execution>
+            </executions>
+          </plugin>
+          <plugin>
+            <artifactId>maven-dependency-plugin</artifactId>
+            <executions>
+              <!-- unpack the unix-common static library and include files -->
+              <execution>
+                <id>unpack</id>
+                <phase>generate-sources</phase>
+                <goals>
+                  <goal>unpack-dependencies</goal>
+                </goals>
+                <configuration>
+                  <includeGroupIds>${project.groupId}</includeGroupIds>
+                  <includeArtifactIds>netty-transport-native-unix-common</includeArtifactIds>
+                  <classifier>${jni.classifier}</classifier>
+                  <outputDirectory>${unix.common.lib.dir}</outputDirectory>
+                  <includes>META-INF/native/**</includes>
+                  <overWriteReleases>false</overWriteReleases>
+                  <overWriteSnapshots>true</overWriteSnapshots>
+                </configuration>
+              </execution>
+            </executions>
+          </plugin>
+
+          <plugin>
+            <groupId>org.fusesource.hawtjni</groupId>
+            <artifactId>maven-hawtjni-plugin</artifactId>
+            <executions>
+              <execution>
+                <id>build-native-lib</id>
+                <configuration>
+                  <name>netty_transport_native_epoll_aarch_64</name>
+                  <nativeSourceDirectory>${nativeSourceDirectory}</nativeSourceDirectory>
+                  <libDirectory>${project.build.outputDirectory}</libDirectory>
+                  <!-- We use Maven's artifact classifier instead.
+                       This hack will make the hawtjni plugin to put the native library
+                       under 'META-INF/native' rather than 'META-INF/native/${platform}'. -->
+                  <platform>.</platform>
+                  <configureArgs>
+                    <arg>${jni.compiler.args.ldflags}</arg>
+                    <arg>${jni.compiler.args.cflags}</arg>
+                    <configureArg>--libdir=${project.build.directory}/native-build/target/lib</configureArg>
+                    <configureArg>--host=aarch64-linux-gnu</configureArg>
+                  </configureArgs>
+                </configuration>
+                <goals>
+                  <goal>generate</goal>
+                  <goal>build</goal>
+                </goals>
+              </execution>
+            </executions>
+          </plugin>
+          <plugin>
+            <artifactId>maven-jar-plugin</artifactId>
+            <executions>
+              <!-- Generate the JAR that contains the native library in it. -->
+              <execution>
+                <id>native-jar</id>
+                <goals>
+                  <goal>jar</goal>
+                </goals>
+                <configuration>
+                  <archive>
+                    <manifest>
+                      <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
+                    </manifest>
+                    <manifestEntries>
+                      <Bundle-NativeCode>META-INF/native/libnetty_transport_native_epoll_aarch_64.so; osname=Linux; processor=aarch_64,*</Bundle-NativeCode>
+                      <Automatic-Module-Name>${javaModuleName}</Automatic-Module-Name>
+                    </manifestEntries>
+                    <index>true</index>
+                    <manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile>
+                  </archive>
+                  <classifier>${jni.classifier}</classifier>
+                </configuration>
+              </execution>
+            </executions>
+          </plugin>
+        </plugins>
+      </build>
+
       <dependencies>
         <dependency>
           <groupId>io.netty</groupId>
diff --git a/transport-native-unix-common/pom.xml b/transport-native-unix-common/pom.xml
index 6e65bffe1b8..e228266bd87 100644
--- a/transport-native-unix-common/pom.xml
+++ b/transport-native-unix-common/pom.xml
@@ -207,6 +207,71 @@
         </plugins>
       </build>
     </profile>
+    <profile>
+      <id>linux-aarch64</id>
+      <properties>
+        <jni.classifier>${os.detected.name}-aarch64</jni.classifier>
+        <jni.platform>linux</jni.platform>
+        <exe.compiler>aarch64-linux-gnu-gcc</exe.compiler>
+        <exe.archiver>aarch64-linux-gnu-ar</exe.archiver>
+      </properties>
+      <build>
+        <plugins>
+          <plugin>
+            <artifactId>maven-antrun-plugin</artifactId>
+            <executions>
+              <!-- Build the additional JAR that contains the native library. -->
+              <execution>
+                <id>native-jar</id>
+                <phase>package</phase>
+                <goals>
+                  <goal>run</goal>
+                </goals>
+                <configuration>
+                  <target>
+                    <copy todir="${nativeJarWorkdir}">
+                      <zipfileset src="${defaultJarFile}" />
+                    </copy>
+                    <copy todir="${nativeJarWorkdir}" includeEmptyDirs="false">
+                      <zipfileset dir="${nativeLibOnlyDir}" />
+                      <regexpmapper handledirsep="yes" from="^(?:[^/]+/)*([^/]+)$" to="META-INF/native/lib/\1" />
+                    </copy>
+                    <copy todir="${nativeJarWorkdir}" includeEmptyDirs="false">
+                      <zipfileset dir="${nativeIncludeDir}" />
+                      <regexpmapper handledirsep="yes" from="^(?:[^/]+/)*([^/]+).h$" to="META-INF/native/include/\1.h" />
+                    </copy>
+                    <jar destfile="${nativeJarFile}" manifest="${nativeJarWorkdir}/META-INF/MANIFEST.MF" basedir="${nativeJarWorkdir}" index="true" excludes="META-INF/MANIFEST.MF,META-INF/INDEX.LIST" />
+                    <attachartifact file="${nativeJarFile}" classifier="${jni.classifier}" type="jar" />
+                  </target>
+                </configuration>
+              </execution>
+              <!-- invoke the make file to build a static library -->
+              <execution>
+                <id>build-native-lib</id>
+                <phase>generate-sources</phase>
+                <goals>
+                  <goal>run</goal>
+                </goals>
+                <configuration>
+                  <target>
+                    <exec executable="${exe.make}" failonerror="true" resolveexecutable="true">
+                      <env key="CC" value="${exe.compiler}" />
+                      <env key="AR" value="${exe.archiver}" />
+                      <env key="LIB_DIR" value="${nativeLibOnlyDir}" />
+                      <env key="OBJ_DIR" value="${nativeObjsOnlyDir}" />
+                      <env key="JNI_PLATFORM" value="${jni.platform}" />
+                      <env key="CFLAGS" value="-O3 -Werror -Wno-attributes -fPIC -fno-omit-frame-pointer -Wunused-variable -fvisibility=hidden" />
+                      <env key="LDFLAGS" value="-Wl,--no-as-needed -lrt" />
+                      <env key="LIB_NAME" value="${nativeLibName}" />
+                    </exec>
+                  </target>
+                </configuration>
+              </execution>
+            </executions>
+          </plugin>
+        </plugins>
+      </build>
+    </profile>
     <profile>
       <id>freebsd</id>
       <activation>


Test patch: