Repository: elastic/elasticsearch
Pull request #124404: [Entitlements] Add an option to perform bytecode verification during instrumentation
Author: login: ldematte
Labels: name: >non-issue
name: Team:Core/Infra
description: Meta label for core/infra team
name: auto-backport
description: Automatically create backport pull requests when merged
name: v8.18.1
name: v8.19.0
name: v9.0.1
name: v9.1.0
name: :Core/Infra/Entitlements
description: Entitlements infrastructure
Created: 2025-03-08T10:53:57Z | Merged: 2025-03-19T14:02:02Z

Using ASM CheckClassAdapter was key to diagnose the issue we had with incorrect signatures for some check methods. 
In this PR I polished up the code I used to pinpoint the issue, and made it available via a system property so it can be turned on if we need it (and it's always on for Entitlements IT tests too).

It is also turned on in case we get VerifyErrors during retransformClasses early in the Entitlement agent bootstrap: `retransformClasses` runs in the native part of the JVM, so the `VerifyError` it produces is not so readable (e.g. it lacks a full stack trace and a description); in case this happens, we re-apply the transformation with verification turned on to get a meaningful error before dying.



Diff:
diff --git a/libs/entitlement/asm-provider/build.gradle b/libs/entitlement/asm-provider/build.gradle
index 7bfd3e1f1f553..d992792cd96d8 100644
--- a/libs/entitlement/asm-provider/build.gradle
+++ b/libs/entitlement/asm-provider/build.gradle
@@ -14,9 +14,15 @@ dependencies {
   compileOnly project(':libs:core')
   compileOnly project(':libs:logging')
   implementation 'org.ow2.asm:asm:9.7.1'
+  implementation 'org.ow2.asm:asm-util:9.7.1'
+  implementation 'org.ow2.asm:asm-tree:9.7.1'
+  implementation 'org.ow2.asm:asm-analysis:9.7.1'
   testImplementation project(":test:framework")
   testImplementation project(":libs:entitlement:bridge")
-  testImplementation 'org.ow2.asm:asm-util:9.7.1'
+}
+
+tasks.named("dependencyLicenses").configure {
+  mapping from: /asm-.*/, to: 'asm'
 }
 
 tasks.named('test').configure {
diff --git a/libs/entitlement/asm-provider/src/main/java/module-info.java b/libs/entitlement/asm-provider/src/main/java/module-info.java
index ddd8cdf08da57..ed75bc2136f34 100644
--- a/libs/entitlement/asm-provider/src/main/java/module-info.java
+++ b/libs/entitlement/asm-provider/src/main/java/module-info.java
@@ -12,6 +12,7 @@
 
 module org.elasticsearch.entitlement.instrumentation {
     requires org.objectweb.asm;
+    requires org.objectweb.asm.util;
     requires org.elasticsearch.entitlement;
 
     requires static org.elasticsearch.base; // for SuppressForbidden
diff --git a/libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java b/libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
index c9b1bb6fe00db..ed9a5b2cc9c6b 100644
--- a/libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
+++ b/libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
@@ -9,6 +9,7 @@
 
 package org.elasticsearch.entitlement.instrumentation.impl;
 
+import org.elasticsearch.core.Strings;
 import org.elasticsearch.entitlement.instrumentation.CheckMethod;
 import org.elasticsearch.entitlement.instrumentation.EntitlementInstrumented;
 import org.elasticsearch.entitlement.instrumentation.Instrumenter;
@@ -24,9 +25,12 @@
 import org.objectweb.asm.Opcodes;
 import org.objectweb.asm.RecordComponentVisitor;
 import org.objectweb.asm.Type;
+import org.objectweb.asm.util.CheckClassAdapter;
 
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.PrintWriter;
+import java.io.StringWriter;
 import java.util.Map;
 import java.util.stream.Stream;
 
@@ -63,6 +67,7 @@ public class InstrumenterImpl implements Instrumenter {
     }
 
     public static InstrumenterImpl create(Class<?> checkerClass, Map<MethodKey, CheckMethod> checkMethods) {
+
         Type checkerClassType = Type.getType(checkerClass);
         String handleClass = checkerClassType.getInternalName() + "Handle";
         String getCheckerClassMethodDescriptor = Type.getMethodDescriptor(checkerClassType);
@@ -82,13 +87,54 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        try {
+            String result = verify(classfileBuffer);
+            if (result.isEmpty() == false) {
+                logger.error(Strings.format("Bytecode verification (%s) for class [%s] failed: %s", phase, className, result));
+            } else {
+                logger.info("Bytecode verification ({}) for class [{}] passed", phase, className);
+            }
+        } catch (ClassCircularityError e) {
+            // Apparently, verification during instrumentation is challenging for class resolution and loading
+            // Treat this not as an error, but as "inconclusive"
+            logger.warn(Strings.format("Cannot perform bytecode verification (%s) for class [%s]", phase, className), e);
+        } catch (IllegalArgumentException e) {
+            // The ASM CheckClassAdapter in some cases throws this instead of printing the error
+            logger.error(Strings.format("Bytecode verification (%s) for class [%s] failed", phase, className), e);
+        }
+    }
+
     @Override
-    public byte[] instrumentClass(String className, byte[] classfileBuffer) {
+    public byte[] instrumentClass(String className, byte[] classfileBuffer, boolean verify) {
+        if (verify) {
+            verifyAndLog(classfileBuffer, className, VerificationPhase.BEFORE_INSTRUMENTATION);
+        }
+
         ClassReader reader = new ClassReader(classfileBuffer);
         ClassWriter writer = new ClassWriter(reader, COMPUTE_FRAMES | COMPUTE_MAXS);
         ClassVisitor visitor = new EntitlementClassVisitor(Opcodes.ASM9, writer, className);
         reader.accept(visitor, 0);
-        return writer.toByteArray();
+        var outBytes = writer.toByteArray();
+
+        if (verify) {
+            verifyAndLog(outBytes, className, VerificationPhase.AFTER_INSTRUMENTATION);
+        }
+
+        return outBytes;
     }
 
     class EntitlementClassVisitor extends ClassVisitor {
diff --git a/libs/entitlement/asm-provider/src/test/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterTests.java b/libs/entitlement/asm-provider/src/test/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterTests.java
index 5a6bf409f1ac4..35251ec9c7d0e 100644
--- a/libs/entitlement/asm-provider/src/test/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterTests.java
+++ b/libs/entitlement/asm-provider/src/test/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterTests.java
@@ -227,9 +227,9 @@ public void testNotInstrumentedTwice() throws Exception {
         var instrumenter = createInstrumenter(Map.of("checkSomeStaticMethod", targetMethod));
 
         var loader1 = instrumentTestClass(instrumenter);
-        byte[] instrumentedTwiceBytecode = instrumenter.instrumentClass(TestClassToInstrument.class.getName(), loader1.testClassBytes);
-        logger.trace(() -> Strings.format("Bytecode after 2nd instrumentation:\n%s", bytecode2text(instrumentedTwiceBytecode)));
-        var loader2 = new TestLoader(TestClassToInstrument.class.getName(), instrumentedTwiceBytecode);
+        byte[] instrumentedTwiceBytes = instrumenter.instrumentClass(TestClassToInstrument.class.getName(), loader1.testClassBytes, true);
+        logger.trace(() -> Strings.format("Bytecode after 2nd instrumentation:\n%s", bytecode2text(instrumentedTwiceBytes)));
+        var loader2 = new TestLoader(TestClassToInstrument.class.getName(), instrumentedTwiceBytes);
 
         assertStaticMethodThrows(loader2, targetMethod, 123);
         assertEquals(1, TestEntitlementCheckerHolder.checkerInstance.checkSomeStaticMethodIntCallCount);
@@ -307,7 +307,7 @@ private static InstrumenterImpl createInstrumenter(Map<String, Executable> metho
     private static TestLoader instrumentTestClass(InstrumenterImpl instrumenter) throws IOException {
         var clazz = TestClassToInstrument.class;
         ClassFileInfo initial = getClassFileInfo(clazz);
-        byte[] newBytecode = instrumenter.instrumentClass(Type.getInternalName(clazz), initial.bytecodes());
+        byte[] newBytecode = instrumenter.instrumentClass(Type.getInternalName(clazz), initial.bytecodes(), true);
         if (logger.isTraceEnabled()) {
             logger.trace("Bytecode after instrumentation:\n{}", bytecode2text(newBytecode));
         }
diff --git a/libs/entitlement/qa/src/javaRestTest/java/org/elasticsearch/entitlement/qa/EntitlementsTestRule.java b/libs/entitlement/qa/src/javaRestTest/java/org/elasticsearch/entitlement/qa/EntitlementsTestRule.java
index 15d470646ea08..9cad8b710ae11 100644
--- a/libs/entitlement/qa/src/javaRestTest/java/org/elasticsearch/entitlement/qa/EntitlementsTestRule.java
+++ b/libs/entitlement/qa/src/javaRestTest/java/org/elasticsearch/entitlement/qa/EntitlementsTestRule.java
@@ -84,6 +84,7 @@ protected void before() throws Throwable {
             .module("entitled", spec -> buildEntitlements(spec, "org.elasticsearch.entitlement.qa.entitled", ENTITLED_POLICY))
             .module(ENTITLEMENT_TEST_PLUGIN_NAME, spec -> setupEntitlements(spec, modular, policyBuilder))
             .systemProperty("es.entitlements.enabled", "true")
+            .systemProperty("es.entitlements.verify_bytecode", "true")
             .systemProperty("es.entitlements.testdir", () -> testDir.getRoot().getAbsolutePath())
             .systemProperties(spec -> tempDirSystemPropertyProvider.get(testDir.getRoot().toPath()))
             .setting("xpack.security.enabled", "false")
diff --git a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
index 682bdad3d7c0b..ba972262c96d4 100644
--- a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
+++ b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
@@ -102,6 +102,11 @@ public static void initialize(Instrumentation inst) throws Exception {
         manager = initChecker();
 
         var latestCheckerInterface = getVersionSpecificCheckerClass(EntitlementChecker.class);
+        var verifyBytecode = Booleans.parseBoolean(System.getProperty("es.entitlements.verify_bytecode", "false"));
+
+        if (verifyBytecode) {
+            ensureClassesSensitiveToVerificationAreInitialized();
+        }
 
         Map<MethodKey, CheckMethod> checkMethods = new HashMap<>(INSTRUMENTATION_SERVICE.lookupMethods(latestCheckerInterface));
         Stream.of(
@@ -124,8 +129,23 @@ public static void initialize(Instrumentation inst) throws Exception {
         var classesToTransform = checkMethods.keySet().stream().map(MethodKey::className).collect(Collectors.toSet());
 
         Instrumenter instrumenter = INSTRUMENTATION_SERVICE.newInstrumenter(latestCheckerInterface, checkMethods);
-        inst.addTransformer(new Transformer(instrumenter, classesToTransform), true);
-        inst.retransformClasses(findClassesToRetransform(inst.getAllLoadedClasses(), classesToTransform));
+        var transformer = new Transformer(instrumenter, classesToTransform, verifyBytecode);
+        inst.addTransformer(transformer, true);
+
+        var classesToRetransform = findClassesToRetransform(inst.getAllLoadedClasses(), classesToTransform);
+        try {
+            inst.retransformClasses(classesToRetransform);
+        } catch (VerifyError e) {
+            // Turn on verification and try to retransform one class at the time to get detailed diagnostic
+            transformer.enableClassVerification();
+
+            for (var classToRetransform : classesToRetransform) {
+                inst.retransformClasses(classToRetransform);
+            }
+
+            // We should have failed already in the loop above, but just in case we did not, rethrow.
+            throw e;
+        }
     }
 
     private static Class<?>[] findClassesToRetransform(Class<?>[] loadedClasses, Set<String> classesToTransform) {
@@ -431,6 +451,24 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    /**
+     * If bytecode verification is enabled, ensure these classes get loaded before transforming/retransforming them.
+     * For these classes, the order in which we transform and verify them matters. Verification during class transformation is at least an
+     * unforeseen (if not unsupported) scenario: we are loading a class, and while we are still loading it (during transformation) we try
+     * to verify it. This in turn leads to more classes loading (for verification purposes), which could turn into those classes to be
+     * transformed and undergo verification. In order to avoid circularity errors as much as possible, we force a partial order.
+     */
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
+        var classesToInitialize = Set.of("sun.net.www.protocol.http.HttpURLConnection");
+        for (String className : classesToInitialize) {
+            try {
+                Class.forName(className);
+            } catch (ClassNotFoundException unexpected) {
+                throw new AssertionError(unexpected);
+            }
+        }
+    }
+
     /**
      * Returns the "most recent" checker class compatible with the current runtime Java version.
      * For checkers, we have (optionally) version specific classes, each with a prefix (e.g. Java23).
diff --git a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Instrumenter.java b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Instrumenter.java
index 9f39cbbbd0df0..c94dc70ae6262 100644
--- a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Instrumenter.java
+++ b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Instrumenter.java
@@ -10,5 +10,5 @@
 package org.elasticsearch.entitlement.instrumentation;
 
 public interface Instrumenter {
-    byte[] instrumentClass(String className, byte[] classfileBuffer);
+    byte[] instrumentClass(String className, byte[] classfileBuffer, boolean verify);
 }
diff --git a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Transformer.java b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Transformer.java
index c6512ee975dbf..1977f767014d0 100644
--- a/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Transformer.java
+++ b/libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Transformer.java
@@ -20,12 +20,19 @@ public class Transformer implements ClassFileTransformer {
     private final Instrumenter instrumenter;
     private final Set<String> classesToTransform;
 
-    public Transformer(Instrumenter instrumenter, Set<String> classesToTransform) {
+    private boolean verifyClasses;
+
+    public Transformer(Instrumenter instrumenter, Set<String> classesToTransform, boolean verifyClasses) {
         this.instrumenter = instrumenter;
         this.classesToTransform = classesToTransform;
+        this.verifyClasses = verifyClasses;
         // TODO: Should warn if any MethodKey doesn't match any methods
     }
 
+    public void enableClassVerification() {
+        this.verifyClasses = true;
+    }
+
     @Override
     public byte[] transform(
         ClassLoader loader,
@@ -36,7 +43,7 @@ public byte[] transform(
     ) {
         if (classesToTransform.contains(className)) {
             // System.out.println("Transforming " + className);
-            return instrumenter.instrumentClass(className, classfileBuffer);
+            return instrumenter.instrumentClass(className, classfileBuffer, verifyClasses);
         } else {
             // System.out.println("Not transforming " + className);
             return classfileBuffer;


Reviews:
user: login: rjernst
body: A couple questions
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-08T14:59:09Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T15:07:07Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T15:09:45Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T15:11:50Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T15:13:35Z
user: login: prdoyle
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T15:21:56Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T15:59:49Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T16:01:42Z
user: login: prdoyle
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T17:33:28Z
user: login: rjernst
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T20:45:35Z
user: login: rjernst
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T20:48:13Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T08:31:41Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T08:33:23Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T09:08:13Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T09:10:49Z
user: login: rjernst
body: Looks better, a couple more nits
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-15T14:09:06Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T07:45:38Z
user: login: rjernst
body: LGTM, one final nit
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-18T16:41:46Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T09:47:20Z
user: login: rjernst
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T13:02:04Z
user: login: ldematte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T14:25:52Z

Review comments:
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: Can we do this as an assert so we don't need a different system property? Then it will automatically be on during tests.
created_at: 2025-03-08T14:57:10Z
author_association: MEMBER
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: Why the different log levels?
created_at: 2025-03-08T14:57:33Z
author_association: MEMBER
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
+        try {
+            String result = verify(classfileBuffer);
+            if (result.isEmpty() == false) {
+                logger.log(
+                    failureLogLevel,
+                    Strings.format("Bytecode verification (%s) for class [%s] failed: %s", phase, className, result)
+                );
+            } else {
+                logger.info("Bytecode verification ({}) for class [{}] passed", phase, className);
+            }
+        } catch (Throwable e) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: This is a broad catch, is there a specific exception we can catch that the adapter throws?
created_at: 2025-03-08T14:58:52Z
author_association: MEMBER
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
+        try {
+            String result = verify(classfileBuffer);
+            if (result.isEmpty() == false) {
+                logger.log(
+                    failureLogLevel,
+                    Strings.format("Bytecode verification (%s) for class [%s] failed: %s", phase, className, result)
+                );
+            } else {
+                logger.info("Bytecode verification ({}) for class [{}] passed", phase, className);
+            }
+        } catch (Throwable e) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: So far I have seen IllegalArgumentException and ClassCircularityError (yes, some JDK classes in Network are... complicated). I expect there could be more thing that are not handled by the adapter.
created_at: 2025-03-08T15:07:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: I assumed we want to highlight if something fails verification after our instrumentation, it is something we want to pay immediate attention. But it's also true that something that fails verification before will inevitably fail after too.
I thought about making this compare the error before/after, and emit a WARN if they are identical (not our fault), and ERROR if different (our fault)
created_at: 2025-03-08T15:09:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: I would avoid it: I think it is OK to have this turned on for our tests, but there are JDK classes that fail verification regularly, and verification is not lightweight/may have side effects (change class loading order). 
ASM is not as bad as ClassFile (I had to revert that, as it was breaking some classes by loading them too agressively/in the wrong loader).
created_at: 2025-03-08T15:11:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: I think about this as an instrument for our diagnostics; it could even have been some commented out code  that we could un-comment if we need to do again the kind of investigation we had to do (but I really dislike commented out code, and the ability to turn it on on already compiled binaries is appealing)
created_at: 2025-03-08T15:13:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,9 +14,11 @@ dependencies {
   compileOnly project(':libs:core')
   compileOnly project(':libs:logging')
   implementation 'org.ow2.asm:asm:9.7.1'
+  implementation 'org.ow2.asm:asm-util:9.7.1'
+  implementation 'org.ow2.asm:asm-tree:9.7.1'
+  implementation 'org.ow2.asm:asm-analysis:9.7.1'
   testImplementation project(":test:framework")
   testImplementation project(":libs:entitlement:bridge")
-  testImplementation 'org.ow2.asm:asm-util:9.7.1'
path: libs/entitlement/asm-provider/build.gradle
user: login: prdoyle
body: Does it work if you remove this? I was under the impression that, in ElasticSearch, `implementation` does not imply `testImplementation` so you need both.
created_at: 2025-03-10T15:15:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,8 +40,8 @@ public class InstrumentationServiceImpl implements InstrumentationService {
     private static final String OBJECT_INTERNAL_NAME = Type.getInternalName(Object.class);
 
     @Override
-    public Instrumenter newInstrumenter(Class<?> clazz, Map<MethodKey, CheckMethod> methods) {
-        return InstrumenterImpl.create(clazz, methods);
+    public Instrumenter newInstrumenter(Class<?> clazz, Map<MethodKey, CheckMethod> methods, boolean verifyBytecode) {
+        return InstrumenterImpl.create(clazz, methods, verifyBytecode);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumentationServiceImpl.java
user: login: prdoyle
body: This `verifyBytecode` argument always comes from the system property. Even tests don't pass a different value in here.

I wonder if this factory method itself ought to check the system property?
created_at: 2025-03-10T15:18:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: prdoyle
body: If some JDK classes fail verification, does that mean we can't enable this in production? If that's true, we probably shouldn't bother with the system property in the first place.

created_at: 2025-03-10T15:20:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,9 +14,11 @@ dependencies {
   compileOnly project(':libs:core')
   compileOnly project(':libs:logging')
   implementation 'org.ow2.asm:asm:9.7.1'
+  implementation 'org.ow2.asm:asm-util:9.7.1'
+  implementation 'org.ow2.asm:asm-tree:9.7.1'
+  implementation 'org.ow2.asm:asm-analysis:9.7.1'
   testImplementation project(":test:framework")
   testImplementation project(":libs:entitlement:bridge")
-  testImplementation 'org.ow2.asm:asm-util:9.7.1'
path: libs/entitlement/asm-provider/build.gradle
user: login: ldematte
body: Good question, I assumed it did. 
Tests run though... in CI too.
created_at: 2025-03-10T15:59:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: > If some JDK classes fail verification

Some JDK classes fail verification as they are (_before_ instrumentation), that's why I don't think we can have this as an assertion, but only in specific tests and only logging.
created_at: 2025-03-10T16:01:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,9 +14,11 @@ dependencies {
   compileOnly project(':libs:core')
   compileOnly project(':libs:logging')
   implementation 'org.ow2.asm:asm:9.7.1'
+  implementation 'org.ow2.asm:asm-util:9.7.1'
+  implementation 'org.ow2.asm:asm-tree:9.7.1'
+  implementation 'org.ow2.asm:asm-analysis:9.7.1'
   testImplementation project(":test:framework")
   testImplementation project(":libs:entitlement:bridge")
-  testImplementation 'org.ow2.asm:asm-util:9.7.1'
path: libs/entitlement/asm-provider/build.gradle
user: login: prdoyle
body: Works for me. 🤷 
created_at: 2025-03-10T17:33:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: If the incoming bytecode fails verification, we should just bail early with an AssertionError, we shouldn't continue on, so the log levels still don't make sense to me. In either case, it should be an error (not just logged).
created_at: 2025-03-10T20:45:35Z
author_association: MEMBER
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: > Some JDK classes fail verification as they are

That seems like something is very wrong then. JDK classes shouldn't be failing verification? @ChrisHegarty ?
created_at: 2025-03-10T20:48:13Z
author_association: MEMBER
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: > Some JDK classes fail verification as they are

It's all linked to this: we can log, but we cannot fail.
created_at: 2025-03-11T08:31:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: > does that mean we can't enable this in production?

If we keep the current implementation, which just logs, we can. And the sysprop is to enable it on demand (e.g. on our IT tests, or in cases like the other day where we need to do some deeper investigation)
created_at: 2025-03-11T08:33:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: I investigated this a bit more: the JDK classes that fail verification fail _only_ when we try to verify them during transformation. The failure is a pretty specific `ClassCircularityError`. We probably hit a corner case where verification try to load classes (for verification purposes) while we are transforming them.
This means we cannot enable this "by default", via assert or similar. 
We agreed to keep the sysprop to enable it "on demand" in case we need to investigate, but also enable it if we hit a VerifyError: we enable verification and try to transform again to collect details on what fails and why.
I'll push a change shortly.
created_at: 2025-03-13T09:08:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -82,13 +91,63 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        var failureLogLevel = (phase == VerificationPhase.BEFORE_INSTRUMENTATION ? Level.WARN : Level.ERROR);
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: ldematte
body: See above: we will just log here, and fail during retransform if we hit a VerifyError
created_at: 2025-03-13T09:10:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,9 +14,11 @@ dependencies {
   compileOnly project(':libs:core')
   compileOnly project(':libs:logging')
   implementation 'org.ow2.asm:asm:9.7.1'
+  implementation 'org.ow2.asm:asm-util:9.7.1'
+  implementation 'org.ow2.asm:asm-tree:9.7.1'
+  implementation 'org.ow2.asm:asm-analysis:9.7.1'
   testImplementation project(":test:framework")
   testImplementation project(":libs:entitlement:bridge")
-  testImplementation 'org.ow2.asm:asm-util:9.7.1'
 }
 
path: libs/entitlement/asm-provider/build.gradle
user: login: rjernst
body: nit: You can avoid needing additional license/notice files with the following gradle configuration:

```
tasks.named("dependencyLicenses").configure {
  mapping from: /asm-.*/, to: 'asm'
}
```
created_at: 2025-03-15T14:01:08Z
author_association: MEMBER
diff_hunk: @@ -82,13 +87,60 @@ static ClassFileInfo getClassFileInfo(Class<?> clazz) throws IOException {
         return new ClassFileInfo(fileName, originalBytecodes);
     }
 
+    private enum VerificationPhase {
+        BEFORE_INSTRUMENTATION,
+        AFTER_INSTRUMENTATION
+    }
+
+    private static String verify(byte[] classfileBuffer) {
+        ClassReader reader = new ClassReader(classfileBuffer);
+        StringWriter stringWriter = new StringWriter();
+        PrintWriter printWriter = new PrintWriter(stringWriter);
+        CheckClassAdapter.verify(reader, false, printWriter);
+        return stringWriter.toString();
+    }
+
+    private static void verifyAndLog(byte[] classfileBuffer, String className, VerificationPhase phase) {
+        try {
+            String result = verify(classfileBuffer);
+            if (result.isEmpty() == false) {
+                logger.error(Strings.format("Bytecode verification (%s) for class [%s] failed: %s", phase, className, result));
+            } else {
+                logger.info("Bytecode verification ({}) for class [{}] passed", phase, className);
+            }
+        } catch (ClassCircularityError e) {
+            // Apparently, verification during instrumentation is challenging for class resolution and loading
+            // Treat this not as an error, but as "inconclusive"
+            logger.warn(
+                "Cannot perform bytecode verification ({}) for class [{}]: {}: {}",
+                phase,
+                className,
+                e.getClass().getName(),
+                e.getMessage()
path: libs/entitlement/asm-provider/src/main/java/org/elasticsearch/entitlement/instrumentation/impl/InstrumenterImpl.java
user: login: rjernst
body: Can we not swallow the exception stacktrace?
created_at: 2025-03-15T14:04:08Z
author_association: MEMBER
diff_hunk: @@ -424,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: rjernst
body: This seems odd. The class is only relevant because we had a bug, right? There's nothing inherently special about this class, so I don't see why we would need to force load it, especially in the case we've already got a VerifyError.
created_at: 2025-03-15T14:08:48Z
author_association: MEMBER
diff_hunk: @@ -424,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: ldematte
body: > The class is only relevant because we had a bug, right? 

Nope, the fact that this is related to the bug is tangential; this needs to be initialized first if we do verification, otherwise verification of other classes fails - verification change the order in which we load classes.
If we don't initialize this first, we get additional false positives in verification errors.
I'll remove this from the VerifyError case - I agree it's not needed in that case.
created_at: 2025-03-17T07:45:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -425,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
+        var classesToInitialize = Set.of("sun.net.www.protocol.http.HttpURLConnection");
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: rjernst
body: This needs a comment. As I understand it, this is because verification fails if a certain load order occurs? This still seems like a JDK bug, so it should at least be documented this is a workaround for that.
created_at: 2025-03-18T16:41:35Z
author_association: MEMBER
diff_hunk: @@ -425,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
+        var classesToInitialize = Set.of("sun.net.www.protocol.http.HttpURLConnection");
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: ldematte
body: I don't think it is; the verification fails/has issues depending on the order in which we _transform_ and _verify_ things; I think we are at least in an unforeseen (if not unsupported) scenario here: we are loading a class, and while loading it we try to verify it, which in order leads to more classes loading, which _could_ turn into verification...
Same reason why we can't have this always on, not even in snapshot builds. It's a good instrument if we need to get more info in case of a VerifyError, but it's not without side effects.

There is a comment already at the call site, I'll move that here and expand.
created_at: 2025-03-19T09:47:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -425,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
+        var classesToInitialize = Set.of("sun.net.www.protocol.http.HttpURLConnection");
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: rjernst
body: LogThe reason a class is being transformed (normally) is it is about to be reloaded. In this case, there should be no difference since the class will be reloaded anyways.

I wonder if we need to be smarter about the order we pass the classes to be retransformed. Normally the jdk would load these in the appropriate order based on hierarchy? Or perhaps calling retransform one at a time is actually the safer approach?
created_at: 2025-03-19T13:02:04Z
author_association: MEMBER
diff_hunk: @@ -425,6 +447,17 @@ private static Stream<InstrumentationService.InstrumentationInfo> pathChecks() {
         });
     }
 
+    private static void ensureClassesSensitiveToVerificationAreInitialized() {
+        var classesToInitialize = Set.of("sun.net.www.protocol.http.HttpURLConnection");
path: libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java
user: login: ldematte
body: > The reason a class is being transformed (normally) is it is about to be reloaded. In this case, there should be no difference since the class will be reloaded anyways.

++ loaded or reloaded.
It might be that all of this only matters for retransform in conjunction with verify; in that case ordering how we retransform classes could solve the issue.
created_at: 2025-03-19T14:25:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: ldematte
created_at: 2025-03-08T13:05:36Z
body: I've tried to use the ClassFile API (https://github.com/elastic/elasticsearch/pull/124404/commits/82207acc43a8e5d7b96037fd3d4fe435a21dadb8) but that is too invasive: it forces classes to load, they get loaded into the wrong loader, and this causes troubles down the road.
author_association: CONTRIBUTOR
user: login: elasticsearchmachine
created_at: 2025-03-08T13:06:35Z
body: Pinging @elastic/es-core-infra (Team:Core/Infra)
author_association: COLLABORATOR
user: login: ldematte
created_at: 2025-03-11T13:57:09Z
body: Let me add some more information on what I have found so far about failing verification in the "before" case (so for classes that should be as they are in the JDK).
I've tried both the ASM verifier ([CheckClassAdapter](https://asm.ow2.io/javadoc/org/objectweb/asm/util/CheckClassAdapter.html)) and the JDK [ClassFile#verify](https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/classfile/ClassFile.html#verify(byte%5B%5D))

So far, from all the classes we transform, they both fail for a single cause (in a different way, but the error is the same): `ClassCircularityError` on some java.net classes (I'll have a list).
This happens while they are trying to verify if a class is assignable from another, for which they both load classes (with `Class.forName`, IIRC).

One thing I'm not sure is if this is a problem with the verification, with the verification done at that stage (during transformation), or with the classes themselves. The last one seems a bit unlikely, but possible: I think that if they are verified one by one, without "context", they would not be problematic, they would have valid structure and bytecode in isolation.
author_association: CONTRIBUTOR
user: login: elasticsearchmachine
created_at: 2025-03-19T14:04:03Z
body: ## 💔 Backport failed
| Status | Branch | Result |
|:------:|:------:|:------:|
| ❌ |  8.18  | Commit could not be cherrypicked due to conflicts |
| ❌ |  8.x  | Commit could not be cherrypicked due to conflicts |
| ✅ |  [9.0](https://github.com/elastic/elasticsearch/pull/125218)  | [<img src="https://img.shields.io/github/pulls/detail/state/elastic/elasticsearch/125218">](https://github.com/elastic/elasticsearch/pull/125218) |

You can use [sqren/backport](https://github.com/sqren/backport) to manually backport by running `backport --upstream elastic/elasticsearch --pr 124404`
author_association: COLLABORATOR