Repository: DataDog/datadog-agent
Pull request #35045: Add `admission.datadoghq.com/apm-inject.debug`
Author: login: keisku
Labels: name: qa/done
description: QA done before merge and regressions are covered by tests
name: medium review
description: PR review might take time
name: team/injection-platform
Created: 2025-03-12T09:18:27Z | Merged: 2025-03-18T08:24:25Z

<!--
* Contributors are encouraged to read our [CONTRIBUTING](/CONTRIBUTING.md) documentation.
* Both Contributor and Reviewer Checklists are available at https://datadoghq.dev/datadog-agent/guidelines/contributing/#pull-requests.
* The pull request:
  * Should only fix one issue or add one feature at a time.
  * Must update the test suite for the relevant functionality.
  * Should pass all status checks before being reviewed or merged.
* Commit titles should be prefixed with general area of pull request's change.
* Please fill the below sections if possible with relevant information or links.
-->
### What does this PR do?

Add new annotation, `admission.datadoghq.com/apm-inject.debug`.

### Motivation

To debug an injected container easier.

### Describe how you validated your changes
<!--
Validate your changes before merge, ensuring that:
* Your PR is tested by static / unit / integrations / e2e tests
* Your PR description details which e2e tests cover your changes, if any
* The PR description contains details of how you validated your changes. If you validated changes manually and not through automated tests, add context on why automated tests did not fit your changes validation.

If you want additional validation by a second person, you can ask reviewers to do it. Describe how to set up an environment for manual tests in the PR description. Manual validation is expected to happen on every commit before merge.

Any manual validation step should then map to an automated test. Manual validation should not substitute automation, minus exceptions not supported by test tooling yet.
-->

Ensured expected env vars were injected to a container in a Pod having `admission.datadoghq.com/apm-inject.debug: "true"`.

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    admission.datadoghq.com/apm-inject.debug: "true"
    admission.datadoghq.com/apm-inject.version: v0
  labels:
    admission.datadoghq.com/enabled: "true"
spec:
  automountServiceAccountToken: false
  containers:
  - env:
    # ....skip.......
    - name: DD_INJECT_START_TIME
      value: "1741770582"
    - name: DD_APM_INSTRUMENTATION_DEBUG
      value: "true"
    - name: DD_TRACE_STARTUP_LOGS
      value: "true"
    - name: DD_TRACE_DEBUG
      value: "true"
```

### Possible Drawbacks / Trade-offs

### Additional Notes
<!--
* Anything else we should know when reviewing?
* Include benchmarking information here whenever possible.
* Include info about alternatives that were considered and why the proposed
  version was chosen.
-->


Diff:
diff --git a/pkg/clusteragent/admission/mutate/autoinstrumentation/auto_instrumentation_test.go b/pkg/clusteragent/admission/mutate/autoinstrumentation/auto_instrumentation_test.go
index 854cb6f01c2a..8452dddb7e2b 100644
--- a/pkg/clusteragent/admission/mutate/autoinstrumentation/auto_instrumentation_test.go
+++ b/pkg/clusteragent/admission/mutate/autoinstrumentation/auto_instrumentation_test.go
@@ -140,6 +140,43 @@ func TestInjectAutoInstruConfigV2(t *testing.T) {
 				},
 			},
 		},
+		{
+			name: "java + debug enabled",
+			pod: common.FakePodSpec{
+				Annotations: map[string]string{
+					"admission.datadoghq.com/apm-inject.version": "v0",
+					"admission.datadoghq.com/apm-inject.debug":   "true",
+				},
+			}.Create(),
+			expectedInjectorImage:   commonRegistry + "/apm-inject:v0",
+			expectedSecurityContext: &corev1.SecurityContext{},
+			expectedLibConfigEnvs: map[string]string{
+				"DD_APM_INSTRUMENTATION_DEBUG": "true",
+				"DD_TRACE_STARTUP_LOGS":        "true",
+				"DD_TRACE_DEBUG":               "true",
+			},
+			libInfo: extractedPodLibInfo{
+				libs: []libInfo{
+					java.libInfo("", "gcr.io/datadoghq/dd-lib-java-init:v1"),
+				},
+			},
+		},
+		{
+			name: "java + debug disabled",
+			pod: common.FakePodSpec{
+				Annotations: map[string]string{
+					"admission.datadoghq.com/apm-inject.version": "v0",
+					"admission.datadoghq.com/apm-inject.debug":   "false",
+				},
+			}.Create(),
+			expectedInjectorImage:   commonRegistry + "/apm-inject:v0",
+			expectedSecurityContext: &corev1.SecurityContext{},
+			libInfo: extractedPodLibInfo{
+				libs: []libInfo{
+					java.libInfo("", "gcr.io/datadoghq/dd-lib-java-init:v1"),
+				},
+			},
+		},
 		{
 			name:                    "config injector-image-override",
 			pod:                     common.FakePod("java-pod"),
diff --git a/pkg/clusteragent/admission/mutate/autoinstrumentation/env_vars.go b/pkg/clusteragent/admission/mutate/autoinstrumentation/env_vars.go
index 33d8567ec1c1..6305488dfb23 100644
--- a/pkg/clusteragent/admission/mutate/autoinstrumentation/env_vars.go
+++ b/pkg/clusteragent/admission/mutate/autoinstrumentation/env_vars.go
@@ -96,6 +96,10 @@ func identityValFunc(s string) envValFunc {
 	return func(string) string { return s }
 }
 
+func trueValFunc() envValFunc {
+	return identityValFunc("true")
+}
+
 func joinValFunc(value string, separator string) envValFunc {
 	return func(predefinedVal string) string {
 		if predefinedVal == "" {
diff --git a/pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go b/pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go
index bfea083f5cc1..97ec2317841e 100644
--- a/pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go
+++ b/pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go
@@ -12,6 +12,7 @@ import (
 	"strconv"
 	"time"
 
+	"github.com/DataDog/datadog-agent/pkg/util/log"
 	corev1 "k8s.io/api/core/v1"
 )
 
@@ -73,6 +74,7 @@ var volumeMountETCDPreloadAppContainer = etcVolume.mount(corev1.VolumeMount{
 type injector struct {
 	image      string
 	registry   string
+	debug      bool
 	injected   bool
 	injectTime time.Time
 	opts       libRequirementOptions
@@ -125,20 +127,23 @@ func (i *injector) requirements() libRequirement {
 			volumeMountETCDPreloadAppContainer.prepended(),
 			v2VolumeMountInjector.prepended(),
 		},
-		envVars: []envVar{
-			{
-				key:     "LD_PRELOAD",
-				valFunc: joinValFunc(asAbs(injectorFilePath("launcher.preload.so")), ":"),
+		envVars: append(
+			[]envVar{
+				{
+					key:     "LD_PRELOAD",
+					valFunc: joinValFunc(asAbs(injectorFilePath("launcher.preload.so")), ":"),
+				},
+				{
+					key:     "DD_INJECT_SENDER_TYPE",
+					valFunc: identityValFunc("k8s"),
+				},
+				{
+					key:     "DD_INJECT_START_TIME",
+					valFunc: identityValFunc(strconv.FormatInt(i.injectTime.Unix(), 10)),
+				},
 			},
-			{
-				key:     "DD_INJECT_SENDER_TYPE",
-				valFunc: identityValFunc("k8s"),
-			},
-			{
-				key:     "DD_INJECT_START_TIME",
-				valFunc: identityValFunc(strconv.FormatInt(i.injectTime.Unix(), 10)),
-			},
-		},
+			i.debugEnvVars()...,
+		),
 	}
 }
 
@@ -154,6 +159,11 @@ var injectorImageAnnotationExtractor = annotationExtractor[injectorOption]{
 	do:  infallibleFn(injectorWithImageName),
 }
 
+var injectorDebugAnnotationExtractor = annotationExtractor[injectorOption]{
+	key: "admission.datadoghq.com/apm-inject.debug",
+	do:  infallibleFn(injectorDebug),
+}
+
 func injectorWithLibRequirementOptions(opts libRequirementOptions) injectorOption {
 	return func(i *injector) {
 		i.opts = opts
@@ -172,6 +182,20 @@ func injectorWithImageTag(tag string) injectorOption {
 	}
 }
 
+func injectorDebug(boolean string) injectorOption {
+	var debug bool
+	var err error
+	if boolean != "" {
+		debug, err = strconv.ParseBool(boolean)
+		if err != nil {
+			log.Errorf("parse admission.datadoghq.com/apm-inject.debug: %s", err)
+		}
+	}
+	return func(i *injector) {
+		i.debug = debug
+	}
+}
+
 func newInjector(startTime time.Time, registry, imageTag string, opts ...injectorOption) *injector {
 	i := &injector{
 		registry:   registry,
@@ -208,3 +232,25 @@ func (i *injector) podMutator(v version) podMutator {
 		return nil
 	})
 }
+
+var debugEnvVars = []envVar{
+	{
+		key:     "DD_APM_INSTRUMENTATION_DEBUG",
+		valFunc: trueValFunc(),
+	},
+	{
+		key:     "DD_TRACE_STARTUP_LOGS",
+		valFunc: trueValFunc(),
+	},
+	{
+		key:     "DD_TRACE_DEBUG",
+		valFunc: trueValFunc(),
+	},
+}
+
+func (i *injector) debugEnvVars() []envVar {
+	if i.debug {
+		return debugEnvVars
+	}
+	return nil
+}
diff --git a/pkg/clusteragent/admission/mutate/autoinstrumentation/namespace_mutator.go b/pkg/clusteragent/admission/mutate/autoinstrumentation/namespace_mutator.go
index adf5ad2e9ce7..4dc31ca8cf01 100644
--- a/pkg/clusteragent/admission/mutate/autoinstrumentation/namespace_mutator.go
+++ b/pkg/clusteragent/admission/mutate/autoinstrumentation/namespace_mutator.go
@@ -216,6 +216,7 @@ func (m *mutatorCore) newInjector(startTime time.Time, pod *corev1.Pod, opts ...
 	for _, e := range []annotationExtractor[injectorOption]{
 		injectorVersionAnnotationExtractor,
 		injectorImageAnnotationExtractor,
+		injectorDebugAnnotationExtractor,
 	} {
 		opt, err := e.extract(pod)
 		if err != nil {
diff --git a/releasenotes/notes/add-admission-controller-annotation-apm-inject-debug-dfdd9f8689c091b0.yaml b/releasenotes/notes/add-admission-controller-annotation-apm-inject-debug-dfdd9f8689c091b0.yaml
new file mode 100644
index 000000000000..e0749490cb55
--- /dev/null
+++ b/releasenotes/notes/add-admission-controller-annotation-apm-inject-debug-dfdd9f8689c091b0.yaml
@@ -0,0 +1,15 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    Add ``admission.datadoghq.com/apm-inject.debug: "true"`` annotation to inject
+    ``DD_APM_INSTRUMENTATION_DEBUG=true``, ``DD_TRACE_STARTUP_LOGS=true`` and
+    ``DD_TRACE_DEBUG=true`` to an application container.
+    This is useful for debugging Single Step Instrumentation or tracer issues
+    in Kubernetes environments.


Reviews:
user: login: buraizu
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T18:51:15Z
user: login: stanistan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T19:21:55Z
user: login: stanistan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T19:25:16Z
user: login: stanistan
body: Thanks for this patch!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-17T13:48:44Z
user: login: buraizu
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T00:19:11Z

Review comments:
diff_hunk: @@ -0,0 +1,15 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    Add ``admission.datadoghq.com/apm-inject.debug: "true"`` annotation to inject
+    ``DD_APM_INSTRUMENTATION_DEBUG=true``, ``DD_TRACE_STARTUP_LOGS=true`` and
+    ``DD_TRACE_DEBUG=true`` to an application container.
+    This is useful for debuuging Single Step Instrumentation or tracer issues
+    in Kubernetes environments.
path: releasenotes/notes/add-admission-controller-annotation-apm-inject-debug-dfdd9f8689c091b0.yaml
user: login: buraizu
body: ```suggestion
    Add ``admission.datadoghq.com/apm-inject.debug: "true"`` annotation to inject
    ``DD_APM_INSTRUMENTATION_DEBUG=true``, ``DD_TRACE_STARTUP_LOGS=true``, and
    ``DD_TRACE_DEBUG=true`` to an application container.
    This is useful for debugging Single Step Instrumentation or tracer issues
    in Kubernetes environments.
```
created_at: 2025-03-12T18:51:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -114,6 +116,36 @@ func (i *injector) initContainer() initContainer {
 }
 
 func (i *injector) requirements() libRequirement {
+	envVars := []envVar{
+		{
+			key:     "LD_PRELOAD",
+			valFunc: joinValFunc(asAbs(injectorFilePath("launcher.preload.so")), ":"),
+		},
+		{
+			key:     "DD_INJECT_SENDER_TYPE",
+			valFunc: identityValFunc("k8s"),
+		},
+		{
+			key:     "DD_INJECT_START_TIME",
+			valFunc: identityValFunc(strconv.FormatInt(i.injectTime.Unix(), 10)),
+		},
+	}
+	if i.debug {
+		envVars = append(envVars, []envVar{
path: pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go
user: login: stanistan
body: You don't need to do an intermediate slice and `...` here, you could do:

```go
trueVal := identityValFunc("true")
envVars = append(
    envVars, 
    { key: "DD_APM_INSTRUMENTATION_DEBUG", valFunc: trueVal },
    { key: "DD_TRACE_STARTUP_LOGS", valFunc: trueVal },
    { key: "DD_TRACE_DEBUG", valFunc: trueVal },
)
```
created_at: 2025-03-13T19:21:55Z
author_association: MEMBER
diff_hunk: @@ -114,6 +116,36 @@ func (i *injector) initContainer() initContainer {
 }
 
 func (i *injector) requirements() libRequirement {
+	envVars := []envVar{
+		{
+			key:     "LD_PRELOAD",
+			valFunc: joinValFunc(asAbs(injectorFilePath("launcher.preload.so")), ":"),
+		},
+		{
+			key:     "DD_INJECT_SENDER_TYPE",
+			valFunc: identityValFunc("k8s"),
+		},
+		{
+			key:     "DD_INJECT_START_TIME",
+			valFunc: identityValFunc(strconv.FormatInt(i.injectTime.Unix(), 10)),
+		},
+	}
+	if i.debug {
+		envVars = append(envVars, []envVar{
path: pkg/clusteragent/admission/mutate/autoinstrumentation/injector.go
user: login: stanistan
body: The other option is to declare the `debugEnvVars` outside of this function and to then  have a helper do:

```go
var debugEnvVars = []envVar{...}
func (i *injector) debugEnvVars() []envVar {
    if i.debug {
        return debugEnvVars
    }
    
    return nil
}
```

And in `requirements()` this would work: 

```go
envVars: append($EXISTING_CODE, i.debugEnvVars()...),
```

created_at: 2025-03-13T19:25:16Z
author_association: MEMBER

PR comments:
user: login: agent-platform-auto-pr[bot]
created_at: 2025-03-12T09:52:45Z
body: ## Test changes on VM

Use this command from [test-infra-definitions](https://github.com/DataDog/test-infra-definitions) to manually test this PR changes on a VM:
```sh
dda inv aws.create-vm --pipeline-id=58800633 --os-family=ubuntu
```
Note: This applies to commit **4ecc3a5b**
author_association: CONTRIBUTOR
user: login: agent-platform-auto-pr[bot]
created_at: 2025-03-12T09:55:29Z
body: ## Uncompressed package size comparison

Comparison with [ancestor](https://github.com/DataDog/datadog-agent/commit/e99528340055aa9a06665e56125b2485186c4017) `e99528340055aa9a06665e56125b2485186c4017`


<details>
  <summary> Diff per package </summary>

|package|diff|status|size|ancestor|threshold|
|--|--|--|--|--|--|
|datadog-agent-amd64-deb|0.00MB|✅|809.18MB|809.18MB|0.50MB|
|datadog-agent-x86_64-rpm|0.00MB|✅|818.98MB|818.98MB|0.50MB|
|datadog-agent-x86_64-suse|0.00MB|✅|818.98MB|818.98MB|0.50MB|
|datadog-agent-aarch64-rpm|0.00MB|✅|809.96MB|809.96MB|0.50MB|
|datadog-agent-arm64-deb|0.00MB|✅|800.18MB|800.18MB|0.50MB|
|datadog-dogstatsd-amd64-deb|0.00MB|✅|39.33MB|39.33MB|0.50MB|
|datadog-dogstatsd-x86_64-rpm|0.00MB|✅|39.41MB|39.41MB|0.50MB|
|datadog-dogstatsd-x86_64-suse|0.00MB|✅|39.41MB|39.41MB|0.50MB|
|datadog-dogstatsd-arm64-deb|0.00MB|✅|37.85MB|37.85MB|0.50MB|
|datadog-heroku-agent-amd64-deb|0.00MB|✅|440.62MB|440.62MB|0.50MB|
|datadog-iot-agent-amd64-deb|0.00MB|✅|62.17MB|62.17MB|0.50MB|
|datadog-iot-agent-x86_64-rpm|0.00MB|✅|62.24MB|62.24MB|0.50MB|
|datadog-iot-agent-x86_64-suse|0.00MB|✅|62.24MB|62.24MB|0.50MB|
|datadog-iot-agent-arm64-deb|0.00MB|✅|59.40MB|59.40MB|0.50MB|
|datadog-iot-agent-aarch64-rpm|0.00MB|✅|59.47MB|59.47MB|0.50MB|

</details>

## Decision
✅ Passed



author_association: CONTRIBUTOR
user: login: agent-platform-auto-pr[bot]
created_at: 2025-03-12T10:04:45Z
body: ## Static quality checks ✅

Please find below the results from static quality gates


<details>
<summary>Successful checks</summary>

### Info

|Result|Quality gate|On disk size|On disk size limit|On wire size|On wire size limit|
|----|----|----|----|----|----|
|✅|static_quality_gate_agent_deb_amd64|783.1 MiB|801.8 MiB|190.85 MiB|202.62 MiB|
|✅|static_quality_gate_agent_deb_arm64|774.58 MiB|793.14 MiB|171.64 MiB|184.51 MiB|
|✅|static_quality_gate_agent_rpm_amd64|783.09 MiB|801.79 MiB|193.08 MiB|205.03 MiB|
|✅|static_quality_gate_agent_rpm_arm64|774.41 MiB|793.09 MiB|175.23 MiB|186.44 MiB|
|✅|static_quality_gate_agent_suse_amd64|783.08 MiB|801.81 MiB|193.08 MiB|205.03 MiB|
|✅|static_quality_gate_agent_suse_arm64|774.51 MiB|793.14 MiB|175.23 MiB|186.44 MiB|
|✅|static_quality_gate_dogstatsd_deb_amd64|37.58 MiB|47.67 MiB|9.74 MiB|19.78 MiB|
|✅|static_quality_gate_dogstatsd_deb_arm64|36.18 MiB|46.27 MiB|8.45 MiB|18.49 MiB|
|✅|static_quality_gate_dogstatsd_rpm_amd64|37.58 MiB|47.67 MiB|9.75 MiB|19.79 MiB|
|✅|static_quality_gate_dogstatsd_suse_amd64|37.58 MiB|47.67 MiB|9.75 MiB|19.79 MiB|
|✅|static_quality_gate_iot_agent_deb_amd64|59.37 MiB|69.0 MiB|14.91 MiB|24.8 MiB|
|✅|static_quality_gate_iot_agent_deb_arm64|56.72 MiB|66.4 MiB|12.85 MiB|22.8 MiB|
|✅|static_quality_gate_iot_agent_rpm_amd64|59.37 MiB|69.0 MiB|14.93 MiB|24.8 MiB|
|✅|static_quality_gate_iot_agent_rpm_arm64|56.72 MiB|66.4 MiB|12.88 MiB|22.8 MiB|
|✅|static_quality_gate_iot_agent_suse_amd64|59.37 MiB|69.0 MiB|14.93 MiB|24.8 MiB|
|✅|static_quality_gate_docker_agent_amd64|867.85 MiB|886.12 MiB|291.97 MiB|304.21 MiB|
|✅|static_quality_gate_docker_agent_arm64|882.53 MiB|900.79 MiB|278.32 MiB|290.47 MiB|
|✅|static_quality_gate_docker_agent_jmx_amd64|1.04 GiB|1.06 GiB|367.06 MiB|379.33 MiB|
|✅|static_quality_gate_docker_agent_jmx_arm64|1.04 GiB|1.06 GiB|349.39 MiB|361.55 MiB|
|✅|static_quality_gate_docker_dogstatsd_amd64|45.73 MiB|55.78 MiB|17.24 MiB|27.28 MiB|
|✅|static_quality_gate_docker_dogstatsd_arm64|44.36 MiB|54.45 MiB|16.12 MiB|26.16 MiB|
|✅|static_quality_gate_docker_cluster_agent_amd64|264.88 MiB|274.78 MiB|106.3 MiB|116.28 MiB|
|✅|static_quality_gate_docker_cluster_agent_arm64|280.8 MiB|290.82 MiB|101.13 MiB|111.12 MiB|

</details>

author_association: CONTRIBUTOR
user: login: agent-platform-auto-pr[bot]
created_at: 2025-03-12T10:15:07Z
body: 
[Fast Unit Tests Report]

On pipeline [58800633](https://gitlab.ddbuild.io/DataDog/datadog-agent/-/pipelines/58800633) ([CI Visibility](https://app.datadoghq.com/ci/pipeline-executions?query=ci_level%3Apipeline%20%40ci.pipeline.name%3ADataDog%2Fdatadog-agent%20%40ci.pipeline.id%3A58800633&fromUser=false&index=cipipeline)). The following jobs did not run any unit tests:

<details>
<summary>Jobs:</summary>

  - tests_flavor_dogstatsd_deb-x64
  - tests_flavor_heroku_deb-x64
  - tests_flavor_iot_deb-x64
</details>

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help
author_association: CONTRIBUTOR
user: login: cit-pr-commenter[bot]
created_at: 2025-03-12T10:15:36Z
body: ## Regression Detector

# Regression Detector Results

[Metrics dashboard](https://app.datadoghq.com/dashboard/ykh-ua8-vcu/SMP-Regression-Detector-Metrics?fromUser=true&refresh_mode=paused&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false)  
[Target profiles](https://app.datadoghq.com/profiling/explorer?query=env%3Asingle-machine-performance%20service%3Adatadog-agent%20job_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&viz=stream&start=1741909635000&end=1741921035000&paused=true)  
Run ID: eaab5db3-8813-434b-a0cc-e45921157e66

Baseline: e99528340055aa9a06665e56125b2485186c4017
Comparison: 4ecc3a5b5e9d31d828d3f25045276963adcaf284
[Diff](e99528340055aa9a06665e56125b2485186c4017...4ecc3a5b5e9d31d828d3f25045276963adcaf284)


## Optimization Goals: ✅ No significant changes detected


<details>
<summary><h2>
Fine details of change detection per experiment
</h2></summary>

| perf | experiment                                   | goal               | Δ mean % | Δ mean % CI    | trials | links                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|------|----------------------------------------------|--------------------|----------|----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ➖   | uds_dogstatsd_to_api_cpu                     | % cpu utilization  | +1.11    | [+0.28, +1.94] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api_cpu%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                   |
| ➖   | quality_gate_logs                            | % cpu utilization  | +1.11    | [-1.67, +3.89] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_logs%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                          |
| ➖   | quality_gate_idle                            | memory utilization | +0.78    | [+0.70, +0.86] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false)                           |
| ➖   | file_to_blackhole_1000ms_latency_linear_load | egress throughput  | +0.21    | [-0.25, +0.67] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency_linear_load%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                               |
| ➖   | uds_dogstatsd_20mb_12k_contexts_20_senders   | memory utilization | +0.13    | [+0.06, +0.20] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_20mb_12k_contexts_20_senders%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                 |
| ➖   | file_to_blackhole_0ms_latency_http1          | egress throughput  | +0.02    | [-0.81, +0.84] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http1%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                        |
| ➖   | file_to_blackhole_0ms_latency                | egress throughput  | +0.02    | [-0.86, +0.89] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                              |
| ➖   | file_to_blackhole_0ms_latency_http2          | egress throughput  | +0.01    | [-0.85, +0.87] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http2%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                        |
| ➖   | tcp_dd_logs_filter_exclude                   | ingress throughput | +0.01    | [-0.01, +0.03] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_dd_logs_filter_exclude%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                 |
| ➖   | file_to_blackhole_100ms_latency              | egress throughput  | +0.01    | [-0.69, +0.71] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_100ms_latency%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | file_to_blackhole_500ms_latency              | egress throughput  | +0.01    | [-0.77, +0.79] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_500ms_latency%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | file_to_blackhole_300ms_latency              | egress throughput  | -0.01    | [-0.64, +0.63] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_300ms_latency%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | uds_dogstatsd_to_api                         | ingress throughput | -0.01    | [-0.29, +0.27] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                       |
| ➖   | file_to_blackhole_1000ms_latency             | egress throughput  | -0.05    | [-0.83, +0.72] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                           |
| ➖   | file_tree                                    | memory utilization | -0.15    | [-0.29, -0.01] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_tree%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                                  |
| ➖   | quality_gate_idle_all_features               | memory utilization | -0.24    | [-0.30, -0.17] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle_all_features%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false) |
| ➖   | tcp_syslog_to_blackhole                      | ingress throughput | -1.29    | [-1.34, -1.24] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_syslog_to_blackhole%20run_id%3Aeaab5db3-8813-434b-a0cc-e45921157e66&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741909635000&to_ts=1741921035000&live=false)                                                                                                                                                                                                                                                                                                                                                                    |
</details>


<details>
<summary><h2>
Bounds Checks: ✅ Passed
</h2></summary>

| perf | experiment                                   | bounds_check_name  | replicates_passed | links                                                                                                                                                                                                                                                                                                                                                       |
|------|----------------------------------------------|--------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ✅   | file_to_blackhole_0ms_latency                | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency                | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http1          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http1          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http2          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http2          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_1000ms_latency             | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_1000ms_latency_linear_load | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_100ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_100ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_300ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_300ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_500ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_500ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_idle                            | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false)              |
| ✅   | quality_gate_idle                            | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false)              |
| ✅   | quality_gate_idle_all_features               | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false) |
| ✅   | quality_gate_idle_all_features               | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&tpl_var_run-id%5B0%5D=eaab5db3-8813-434b-a0cc-e45921157e66&view=spans&from_ts=1741916835000&to_ts=1741917435000&live=false) |
| ✅   | quality_gate_logs                            | intake_connections | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_logs                            | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_logs                            | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
</details>


<details>
<summary><h2>
Explanation
</h2></summary>

**Confidence level:** 90.00%
**Effect size tolerance:** |Δ mean %| ≥ 5.00%

Performance changes are noted in the **perf** column of each table:

* ✅ = significantly better comparison variant performance
* ❌ = significantly worse comparison variant performance
* ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".
</details>

## CI Pass/Fail Decision

✅ **Passed.** All Quality Gates passed.

- **quality_gate_logs**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **lost_bytes**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
author_association: NONE
user: login: keisku
created_at: 2025-03-18T07:26:02Z
body: /merge
author_association: CONTRIBUTOR
user: login: dd-devflow[bot]
created_at: 2025-03-18T07:26:06Z
body: <!-- "FeedbackWorkflowId":"f56a89f1-6670-4184-a006-9291891e5dee_36" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/f56a89f1-6670-4184-a006-9291891e5dee_36)
`2025-03-18 07:26:06 UTC` :information_source: Start processing command `/merge`

---
`2025-03-18 07:26:12 UTC` :information_source: **MergeQueue:** pull request added to the queue

The expected merge time in `main` is approximately 41m (p90).

---
`2025-03-18 07:46:17 UTC` :information_source: **MergeQueue:** Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.

---
`2025-03-18 08:24:28 UTC` :information_source: **MergeQueue:** This merge request was merged

author_association: NONE