Repository: jenkinsci/aws-java-sdk2-plugin
Pull request #67: Add netty-nio-client
Author: login: rsandell
Labels: name: enhancement
description: For changelog: Minor enhancement. use `major-rfe` for changes to be highlighted
Created: 2025-05-14T15:34:59Z | Merged: 2025-05-26T08:14:18Z

<!-- Please describe your pull request here. -->
For https://github.com/jenkinsci/s3-plugin/pull/327
### Testing done

<!-- Comment:
Provide a clear description of how this change was tested.
At minimum this should include proof that a computer has executed the changed lines.
Ideally this should include an automated test or an explanation as to why this change has no tests.
Note that automated test coverage is less than complete, so a successful PR build does not necessarily imply that a computer has executed the changed lines.
If automated test coverage does not exist for the lines you are changing, you must describe the scenario(s) in which you manually tested the change.
For frontend changes, include screenshots of the relevant page(s) before and after the change.
For refactoring and code cleanup changes, exercise the code before and after the change and verify the behavior remains the same.
-->

### Submitter checklist
- [ ] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [ ] Ensure that the pull request title represents the desired changelog entry
- [ ] Please describe what you did
- [ ] Link to relevant issues in GitHub or Jira
- [ ] Link to relevant pull requests, esp. upstream and downstream changes
- [ ] Ensure you have provided tests - that demonstrates feature works or fixes the issue

<!--
Put an `x` into the [ ] to show you have filled the information.
The template comes from https://github.com/jenkinsci/.github/blob/master/.github/pull_request_template.md 
You can override it by creating .github/pull_request_template.md in your own repository 
-->



Diff:
diff --git a/aws-java-sdk2-netty-nio-client/pom.xml b/aws-java-sdk2-netty-nio-client/pom.xml
new file mode 100644
index 0000000..879d59b
--- /dev/null
+++ b/aws-java-sdk2-netty-nio-client/pom.xml
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+    <artifactId>aws-java-sdk2-parent</artifactId>
+    <version>${revision}-${changelist}</version>
+  </parent>
+
+  <artifactId>aws-java-sdk2-netty-nio-client</artifactId>
+  <packaging>hpi</packaging>
+  <name>AWS Java SDK :: HTTP Clients :: Netty Non-Blocking I/O</name>
+  <url>https://github.com/jenkinsci/aws-java-sdk2-plugin</url>
+
+  <dependencies>
+    <dependency>
+      <!--
+      TODO keep in sync with what software.amazon.awssdk:netty-nio-client:${revision} needs
+        see around https://github.com/aws/aws-sdk-java-v2/blob/${revision}/pom.xml#L118
+      -->
+      <groupId>io.jenkins.plugins</groupId>
+      <artifactId>netty-api</artifactId>
+      <version>4.1.118.Final-9.v776038d601a_7</version>
+    </dependency>
+    <dependency>
+      <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+      <artifactId>aws-java-sdk2-core</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>software.amazon.awssdk</groupId>
+      <artifactId>netty-nio-client</artifactId>
+      <exclusions>
+        <exclusion>
+          <groupId>io.netty</groupId>
+          <artifactId>*</artifactId>
+        </exclusion>
+        <!-- Provided by Jenkins core -->
+        <exclusion>
+          <groupId>org.slf4j</groupId>
+          <artifactId>slf4j-api</artifactId>
+        </exclusion>
+      </exclusions>
+    </dependency>
+  </dependencies>
+</project>
diff --git a/aws-java-sdk2-netty-nio-client/src/main/resources/index.jelly b/aws-java-sdk2-netty-nio-client/src/main/resources/index.jelly
new file mode 100644
index 0000000..950a4ac
--- /dev/null
+++ b/aws-java-sdk2-netty-nio-client/src/main/resources/index.jelly
@@ -0,0 +1,4 @@
+<?jelly escape-by-default='true'?>
+<div>
+  HTTP Clients :: Netty Non-Blocking I/O module for the <a href="https://aws.amazon.com/sdk-for-java/">AWS SDK for Java</a>.
+</div>
diff --git a/pom.xml b/pom.xml
index 750eb0e..5a9a799 100644
--- a/pom.xml
+++ b/pom.xml
@@ -42,6 +42,7 @@
     <module>aws-java-sdk2-iam</module>
     <module>aws-java-sdk2-kinesis</module>
     <module>aws-java-sdk2-lambda</module>
+    <module>aws-java-sdk2-netty-nio-client</module>
     <module>aws-java-sdk2-organizations</module>
     <module>aws-java-sdk2-s3</module>
     <module>aws-java-sdk2-secretsmanager</module>


Reviews:
user: login: basil
body: A great start! This is looking good so far. The only problem with it is that it fully embeds Netty:

```
08:39:11  [INFO] Assembling webapp aws-java-sdk2-netty-nio-client in /home/jenkins/agent/workspace/ugins_aws-java-sdk2-plugin_PR-67/aws-java-sdk2-netty-nio-client/target/aws-java-sdk2-netty-nio-client
08:39:11  [WARNING] Bundling transitive dependency netty-buffer-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-codec-http2-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-codec-http-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-codec-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-common-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-handler-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-resolver-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-transport-classes-epoll-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-transport-native-unix-common-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency netty-transport-4.1.118.Final.jar (via netty-nio-client)
08:39:11  [WARNING] Bundling transitive dependency reactive-streams-1.0.4.jar (via netty-nio-client)
08:39:11  [INFO] Bundling direct dependency netty-nio-client-2.31.26.jar
```


Fully embedding Netty would be fine if this were the only plugin on the system that used Netty. However, there are some others:

```
./work/amazon-inspector-image-scanner/492.v4996d9230272/amazon-inspector-image-scanner.hpi
WEB-INF/lib/netty-buffer-4.1.115.Final.jar
WEB-INF/lib/netty-codec-4.1.115.Final.jar
WEB-INF/lib/netty-codec-http-4.1.115.Final.jar
WEB-INF/lib/netty-codec-http2-4.1.115.Final.jar
WEB-INF/lib/netty-common-4.1.115.Final.jar
WEB-INF/lib/netty-handler-4.1.115.Final.jar
WEB-INF/lib/netty-nio-client-2.29.43.jar
WEB-INF/lib/netty-resolver-4.1.115.Final.jar
WEB-INF/lib/netty-transport-4.1.115.Final.jar
WEB-INF/lib/netty-transport-classes-epoll-4.1.115.Final.jar
WEB-INF/lib/netty-transport-native-unix-common-4.1.115.Final.jar

./work/artifactory/4.0.8/artifactory.hpi
WEB-INF/lib/netty-transport-4.1.79.Final.jar
WEB-INF/lib/netty-handler-4.1.46.Final.jar
WEB-INF/lib/netty-resolver-4.1.79.Final.jar
WEB-INF/lib/netty-codec-http-4.1.79.Final.jar
WEB-INF/lib/netty-codec-4.1.79.Final.jar
WEB-INF/lib/netty-transport-native-unix-common-4.1.46.Final.jar
WEB-INF/lib/netty-transport-native-epoll-4.1.46.Final.jar
WEB-INF/lib/netty-buffer-4.1.79.Final.jar
WEB-INF/lib/netty-common-4.1.79.Final.jar
WEB-INF/lib/netty-codec-socks-4.1.79.Final.jar
WEB-INF/lib/netty-transport-native-kqueue-4.1.46.Final.jar
WEB-INF/lib/netty-handler-proxy-4.1.79.Final.jar
WEB-INF/lib/docker-java-transport-netty-3.3.3.jar

./work/azure-sdk/211.vd3b_6c5504331/azure-sdk.hpi
WEB-INF/lib/azure-core-http-netty-1.15.11.jar
WEB-INF/lib/netty-buffer-4.1.118.Final.jar
WEB-INF/lib/netty-codec-4.1.118.Final.jar
WEB-INF/lib/netty-codec-dns-4.1.112.Final.jar
WEB-INF/lib/netty-codec-http-4.1.118.Final.jar
WEB-INF/lib/netty-codec-http2-4.1.118.Final.jar
WEB-INF/lib/netty-codec-socks-4.1.118.Final.jar
WEB-INF/lib/netty-common-4.1.118.Final.jar
WEB-INF/lib/netty-handler-4.1.118.Final.jar
WEB-INF/lib/netty-handler-proxy-4.1.118.Final.jar
WEB-INF/lib/netty-resolver-4.1.118.Final.jar
WEB-INF/lib/netty-resolver-dns-4.1.112.Final.jar
WEB-INF/lib/netty-resolver-dns-classes-macos-4.1.112.Final.jar
WEB-INF/lib/netty-resolver-dns-native-macos-4.1.112.Final-osx-x86_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final-linux-aarch_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final-linux-x86_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final-osx-aarch_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final-osx-x86_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final-windows-x86_64.jar
WEB-INF/lib/netty-tcnative-boringssl-static-2.0.70.Final.jar
WEB-INF/lib/netty-tcnative-classes-2.0.70.Final.jar
WEB-INF/lib/netty-transport-4.1.118.Final.jar
WEB-INF/lib/netty-transport-classes-epoll-4.1.118.Final.jar
WEB-INF/lib/netty-transport-classes-kqueue-4.1.118.Final.jar
WEB-INF/lib/netty-transport-native-epoll-4.1.118.Final-linux-x86_64.jar
WEB-INF/lib/netty-transport-native-kqueue-4.1.118.Final-osx-x86_64.jar
WEB-INF/lib/netty-transport-native-unix-common-4.1.118.Final.jar
WEB-INF/lib/reactor-netty-core-1.0.48.jar
WEB-INF/lib/reactor-netty-http-1.0.48.jar
```

Whenever two different plugins provide the same library, this has historically caused subtle class loading problems. The uberclassloader used in rendering web views can pick the wrong loader, and then linkage errors can ensue. The status quo mentioned above is already fragile/dangerous, and I fear we would be making the situation more fragile/dangerous by adding yet another full copy of Netty.

A similar situation was present a few years ago, when I naively added a fully copy of Jersey to the Jackson plugin. That exposed the same scenario mentioned above, not in initial testing, but over several weeks as users began to complain about linkage errors.

The solution, then and now, was to extract the libary into its own library plugin, as in https://github.com/jenkinsci/jackson2-api-plugin/pull/122. The situation here is exactly the same, but instead of Jersey it’s Netty. So we would want to create a new Netty library plugin that contains only Netty. Once that is done, I can help refactor at least `azure-sdk` to use it. And then in this PR you would consume the Netty library plugin rather than bundling a full copy of Netty. Not only does this save disk space, but it eliminates the risk of linkage errors mentioned above, since the uberclassloader only has one loader to choose from, and instances of Netty classes can be freely passed around between plugins without fear of linkage errors.

state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T21:03:13Z
user: login: raul-arabaolaza
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T10:57:17Z
user: login: vwagh-dev
body: LGTM
state: APPROVED
author_association: NONE
submitted_at: 2025-05-23T11:27:08Z
user: login: BorisYaoA
state: APPROVED
author_association: NONE
submitted_at: 2025-05-23T12:17:01Z
user: login: basil
body: ```
02:57:28  [INFO] Copy webapp webResources to /home/jenkins/agent/workspace/ugins_aws-java-sdk2-plugin_PR-67/aws-java-sdk2-netty-nio-client/target/aws-java-sdk2-netty-nio-client
02:57:28  [INFO] Assembling webapp aws-java-sdk2-netty-nio-client in /home/jenkins/agent/workspace/ugins_aws-java-sdk2-plugin_PR-67/aws-java-sdk2-netty-nio-client/target/aws-java-sdk2-netty-nio-client
02:57:29  [WARNING] Bundling transitive dependency reactive-streams-1.0.4.jar (via netty-nio-client)
02:57:29  [INFO] Bundling direct dependency netty-nio-client-2.31.26.jar
02:57:29  [INFO] Generating hpi /home/jenkins/agent/workspace/ugins_aws-java-sdk2-plugin_PR-67/aws-java-sdk2-netty-nio-client/target/aws-java-sdk2-netty-nio-client.hpi
02:57:29  [INFO] Building jar: /home/jenkins/agent/workspace/ugins_aws-java-sdk2-plugin_PR-67/aws-java-sdk2-netty-nio-client/target/aws-java-sdk2-netty-nio-client.hpi
```

`reactive-streams-1.0.4.jar` is already part of `aws-java-sdk2-core`, so we should depend on that rather than bundling it again here.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-23T17:56:13Z
user: login: olamy
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-26T00:41:33Z
user: login: olamy
body: all good
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-26T08:12:32Z

Review comments:
diff_hunk: @@ -0,0 +1,82 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+    <artifactId>aws-java-sdk2-parent</artifactId>
+    <version>${revision}-${changelist}</version>
+  </parent>
+
+  <artifactId>aws-java-sdk2-netty-nio-client</artifactId>
+  <packaging>hpi</packaging>
+  <name>AWS Java SDK :: HTTP Clients :: Netty Non-Blocking I/O</name>
+  <url>https://github.com/jenkinsci/aws-java-sdk2-plugin</url>
+
+  <dependencies>
+    <dependency>
+      <!--
+      TODO keep in sync with what software.amazon.awssdk:netty-nio-client:${revision} needs
+        see around https://github.com/aws/aws-sdk-java-v2/blob/${revision}/pom.xml#L118
+      -->
+      <groupId>io.jenkins.plugins</groupId>
+      <artifactId>netty-api</artifactId>
+      <version>4.1.118.Final-9.v776038d601a_7</version>
+    </dependency>
+    <dependency>
+      <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+      <artifactId>aws-java-sdk2-core</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>software.amazon.awssdk</groupId>
+      <artifactId>netty-nio-client</artifactId>
+      <exclusions>
path: aws-java-sdk2-netty-nio-client/pom.xml
user: login: raul-arabaolaza
body: ```suggestion
      <!-- Already provided by the netty-api plugin to avoid funny classloading issues -->
      <exclusions> 
```
created_at: 2025-05-23T10:56:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,82 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+    <artifactId>aws-java-sdk2-parent</artifactId>
+    <version>${revision}-${changelist}</version>
+  </parent>
+
+  <artifactId>aws-java-sdk2-netty-nio-client</artifactId>
+  <packaging>hpi</packaging>
+  <name>AWS Java SDK :: HTTP Clients :: Netty Non-Blocking I/O</name>
+  <url>https://github.com/jenkinsci/aws-java-sdk2-plugin</url>
+
+  <dependencies>
+    <dependency>
+      <!--
+      TODO keep in sync with what software.amazon.awssdk:netty-nio-client:${revision} needs
+        see around https://github.com/aws/aws-sdk-java-v2/blob/${revision}/pom.xml#L118
+      -->
+      <groupId>io.jenkins.plugins</groupId>
+      <artifactId>netty-api</artifactId>
+      <version>4.1.118.Final-9.v776038d601a_7</version>
+    </dependency>
+    <dependency>
+      <groupId>io.jenkins.plugins.aws-java-sdk2</groupId>
+      <artifactId>aws-java-sdk2-core</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>software.amazon.awssdk</groupId>
+      <artifactId>netty-nio-client</artifactId>
+      <exclusions>
path: aws-java-sdk2-netty-nio-client/pom.xml
user: login: olamy
body: can be simplified using a single 
```
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>*</artifactId>
        </exclusion>
```
created_at: 2025-05-26T00:41:19Z
author_association: MEMBER

PR comments:
user: login: rsandell
created_at: 2025-05-19T08:15:44Z
body: I don't think the classloading problems will be as severe as for example in jackson-api as you suspect. Most issues comes between plugins that are in the same dependency chain and the netty classes should never be exposed to views. There is a slight risk with the remoting classloader, and for the s3 plugin that I am working on it has a mode where it will do uploads from the agents, so the netty classes will be loaded over remoting in that case.
author_association: MEMBER
user: login: rsandell
created_at: 2025-05-19T14:53:11Z
body: https://github.com/jenkins-infra/repository-permissions-updater/issues/4447
author_association: MEMBER
user: login: rsandell
created_at: 2025-05-22T15:16:34Z
body: @basil netty-api-plugin is in!
author_association: MEMBER
user: login: basil
created_at: 2025-05-26T16:22:32Z
body: @raul-arabaolaza, looks like my comment from https://github.com/jenkinsci/aws-java-sdk2-plugin/pull/67#pullrequestreview-2865299044 was not addressed?
author_association: MEMBER
user: login: rsandell
created_at: 2025-05-27T08:43:27Z
body: https://github.com/jenkinsci/aws-java-sdk2-plugin/pull/74
author_association: MEMBER
user: login: basil
created_at: 2025-05-27T15:51:00Z
body: Thanks!

> Once that is done, I can help refactor at least `azure-sdk` to use it.

⇒ https://github.com/jenkinsci/azure-sdk-plugin/pull/299
author_association: MEMBER