Repository: DataDog/datadog-agent
Pull request #32611: Lazy load rtloader when scheduling a python check
Author: login: pgimalac
Labels: name: changelog/no-changelog
description: No changelog entry needed
name: qa/done
description: QA done before merge and regressions are covered by tests
name: long review
description: PR is complex, plan time to review it
name: ask-review
description: Ask required teams to review this PR
name: team/agent-runtimes
name: team/agent-metric-pipelines
Created: 2025-01-02T13:14:10Z | Merged: 2025-03-12T14:55:25Z

<!--
* Contributors are encouraged to read our [CONTRIBUTING](/CONTRIBUTING.md) documentation.
* Both Contributor and Reviewer Checklists are available at https://datadoghq.dev/datadog-agent/guidelines/contributing/#pull-requests.
* The pull request:
  * Should only fix one issue or add one feature at a time.
  * Must update the test suite for the relevant functionality.
  * Should pass all status checks before being reviewed or merged.
* Commit titles should be prefixed with general area of pull request's change.
* Please fill the below sections if possible with relevant information or links.

https://datadoghq.atlassian.net/browse/ASCII-2645
-->
### What does this PR do?
Initialize Python and rtloader when a first Python check is scheduled, rather than when starting the agent.

### Motivation
Eventually this will allow not loading Python if no Python check is scheduled.
Right now this doesn't change anything (in particular no change in resource usage), since we need to load Python before loading any check anyway (including Go checks).
 
### Describe how you validated your changes
<!--
Validate your changes before merge, ensuring that:
* Your PR is tested by static / unit / integrations / e2e tests
* Your PR description details which e2e tests cover your changes, if any
* The PR description contains details of how you validated your changes. If you validated changes manually and not through automated tests, add context on why automated tests did not fit your changes validation.

If you want additional validation by a second person, you can ask reviewers to do it. Describe how to set up an environment for manual tests in the PR description. Manual validation is expected to happen on every commit before merge.

Any manual validation step should then map to an automated test. Manual validation should not substitute automation, minus exceptions not supported by test tooling yet.
-->
We have some e2e tests running custom Python checks and ensuring they behave as expected, so if they pass then this change works properly.

### Possible Drawbacks / Trade-offs

### Additional Notes
<!--
* Anything else we should know when reviewing?
* Include benchmarking information here whenever possible.
* Include info about alternatives that were considered and why the proposed
  version was chosen.
-->
This only makes sense if we can avoid loading Python for many users. 
In particular next steps are:
- moving default Python checks to Go (eg. https://github.com/DataDog/datadog-agent/pull/33116 for `disk`) 
- avoid needing to load Python to load a Go check (which is the case now) 



Diff:
diff --git a/comp/collector/collector/collectorimpl/collector.go b/comp/collector/collector/collectorimpl/collector.go
index 59f16e2be354..d669f7ce2d3a 100644
--- a/comp/collector/collector/collectorimpl/collector.go
+++ b/comp/collector/collector/collectorimpl/collector.go
@@ -27,9 +27,9 @@ import (
 	haagent "github.com/DataDog/datadog-agent/comp/haagent/def"
 	metadata "github.com/DataDog/datadog-agent/comp/metadata/runner/runnerimpl"
 	"github.com/DataDog/datadog-agent/pkg/aggregator/sender"
-	pkgCollector "github.com/DataDog/datadog-agent/pkg/collector"
 	"github.com/DataDog/datadog-agent/pkg/collector/check"
 	checkid "github.com/DataDog/datadog-agent/pkg/collector/check/id"
+	"github.com/DataDog/datadog-agent/pkg/collector/python"
 	"github.com/DataDog/datadog-agent/pkg/collector/runner"
 	"github.com/DataDog/datadog-agent/pkg/collector/runner/expvars"
 	"github.com/DataDog/datadog-agent/pkg/collector/scheduler"
@@ -132,7 +132,9 @@ func newCollector(deps dependencies) *collectorImpl {
 		createdAt:          time.Now(),
 	}
 
-	pkgCollector.InitPython(common.GetPythonPaths()...)
+	if !deps.Config.GetBool("python_lazy_loading") {
+		python.InitPython(common.GetPythonPaths()...)
+	}
 
 	deps.Lc.Append(fx.Hook{
 		OnStart: c.start,
diff --git a/pkg/cli/subcommands/check/command.go b/pkg/cli/subcommands/check/command.go
index 0ecd87a5b119..0fde23ae8dbe 100644
--- a/pkg/cli/subcommands/check/command.go
+++ b/pkg/cli/subcommands/check/command.go
@@ -290,10 +290,11 @@ func run(
 		cliParams.cmd.Help() //nolint:errcheck
 		return nil
 	}
-
 	// TODO: (components) - Until the checks are components we set there context so they can depends on components.
 	check.InitializeInventoryChecksContext(invChecks)
-	pkgcollector.InitPython(common.GetPythonPaths()...)
+	if !config.GetBool("python_lazy_loading") {
+		python.InitPython(common.GetPythonPaths()...)
+	}
 	// TODO Ideally we would support RC in the check subcommand,
 	//  but at the moment this is not possible - only one process can access the RC database at a time,
 	//  so the subcommand can't read the RC database if the agent is also running.
diff --git a/pkg/collector/embed_nopy.go b/pkg/collector/python/embed_nopy.go
similarity index 95%
rename from pkg/collector/embed_nopy.go
rename to pkg/collector/python/embed_nopy.go
index b4195f3d8a02..4e31dc74cb40 100644
--- a/pkg/collector/embed_nopy.go
+++ b/pkg/collector/python/embed_nopy.go
@@ -5,7 +5,7 @@
 
 //go:build !python
 
-package collector
+package python
 
 // InitPython is a no-op when the build tag is not set
 func InitPython(_ ...string) {}
diff --git a/pkg/collector/embed_python.go b/pkg/collector/python/embed_python.go
similarity index 83%
rename from pkg/collector/embed_python.go
rename to pkg/collector/python/embed_python.go
index c20edcf3b7ae..d4b038222cf9 100644
--- a/pkg/collector/embed_python.go
+++ b/pkg/collector/python/embed_python.go
@@ -5,10 +5,9 @@
 
 //go:build python
 
-package collector
+package python
 
 import (
-	"github.com/DataDog/datadog-agent/pkg/collector/python"
 	pkgconfigsetup "github.com/DataDog/datadog-agent/pkg/config/setup"
 	"github.com/DataDog/datadog-agent/pkg/util/log"
 )
@@ -31,16 +30,16 @@ func InitPython(paths ...string) {
 }
 
 func pySetup(paths ...string) (pythonVersion, pythonHome, pythonPath string) {
-	if err := python.Initialize(paths...); err != nil {
+	if err := Initialize(paths...); err != nil {
 		log.Errorf("Could not initialize Python: %s", err)
 	}
-	return python.PythonVersion, python.PythonHome, python.PythonPath
+	return PythonVersion, PythonHome, PythonPath
 }
 
 func pyPrepareEnv() error {
 	if pkgconfigsetup.Datadog().IsSet("procfs_path") {
 		procfsPath := pkgconfigsetup.Datadog().GetString("procfs_path")
-		return python.SetPythonPsutilProcPath(procfsPath)
+		return SetPythonPsutilProcPath(procfsPath)
 	}
 	return nil
 }
diff --git a/pkg/collector/python/loader.go b/pkg/collector/python/loader.go
index 0146a3840337..238330517c65 100644
--- a/pkg/collector/python/loader.go
+++ b/pkg/collector/python/loader.go
@@ -17,6 +17,7 @@ import (
 
 	"github.com/mohae/deepcopy"
 
+	"github.com/DataDog/datadog-agent/cmd/agent/common"
 	"github.com/DataDog/datadog-agent/comp/core/autodiscovery/integration"
 	tagger "github.com/DataDog/datadog-agent/comp/core/tagger/def"
 	integrations "github.com/DataDog/datadog-agent/comp/logs/integrations/def"
@@ -49,6 +50,7 @@ var (
 	py3LintedLock    sync.Mutex
 	linterLock       sync.Mutex
 	agentVersionTags []string
+	pythonOnce       sync.Once
 )
 
 const (
@@ -116,6 +118,12 @@ func (*PythonCheckLoader) Name() string {
 // Load tries to import a Python module with the same name found in config.Name, searches for
 // subclasses of the AgentCheck class and returns the corresponding Check
 func (cl *PythonCheckLoader) Load(senderManager sender.SenderManager, config integration.Config, instance integration.Data) (check.Check, error) {
+	if pkgconfigsetup.Datadog().GetBool("python_lazy_loading") {
+		pythonOnce.Do(func() {
+			InitPython(common.GetPythonPaths()...)
+		})
+	}
+
 	if rtloader == nil {
 		return nil, fmt.Errorf("python is not initialized")
 	}
diff --git a/pkg/collector/python/test_check.go b/pkg/collector/python/test_check.go
index 1334467214be..dcba49f53d21 100644
--- a/pkg/collector/python/test_check.go
+++ b/pkg/collector/python/test_check.go
@@ -202,8 +202,7 @@ void reset_check_mock() {
 import "C"
 
 func testRunCheck(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
 		return
@@ -235,8 +234,7 @@ func testRunCheck(t *testing.T) {
 }
 
 func testRunCheckWithRuntimeNotInitializedError(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
 		return
@@ -282,8 +280,7 @@ func testInitiCheckWithRuntimeNotInitialized(t *testing.T) {
 }
 
 func testCheckCancel(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
 		return
@@ -318,8 +315,7 @@ func testCheckCancel(t *testing.T) {
 }
 
 func testCheckCancelWhenRuntimeUnloaded(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
@@ -353,13 +349,7 @@ func testCheckCancelWhenRuntimeUnloaded(t *testing.T) {
 }
 
 func testFinalizer(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() {
-		// We have to wrap this in locks otherwise the race detector complains
-		pyDestroyLock.Lock()
-		rtloader = nil
-		pyDestroyLock.Unlock()
-	}()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
@@ -397,13 +387,7 @@ func testFinalizer(t *testing.T) {
 }
 
 func testFinalizerWhenRuntimeUnloaded(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() {
-		// We have to wrap this in locks otherwise the race detector complains
-		pyDestroyLock.Lock()
-		rtloader = nil
-		pyDestroyLock.Unlock()
-	}()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
@@ -442,8 +426,7 @@ func testFinalizerWhenRuntimeUnloaded(t *testing.T) {
 }
 
 func testRunErrorNil(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
@@ -470,8 +453,7 @@ func testRunErrorNil(t *testing.T) {
 }
 
 func testRunErrorReturn(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
@@ -499,8 +481,7 @@ func testRun(t *testing.T) {
 	sender := mocksender.NewMockSender(checkid.ID("testID"))
 	sender.SetupAcceptAll()
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	c, err := NewPythonFakeCheck(sender.GetSenderManager())
 	if !assert.Nil(t, err) {
@@ -532,8 +513,7 @@ func testRunSimple(t *testing.T) {
 	sender := mocksender.NewMockSender(checkid.ID("testID"))
 	sender.SetupAcceptAll()
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	c, err := NewPythonFakeCheck(sender.GetSenderManager())
 	if !assert.Nil(t, err) {
@@ -562,8 +542,7 @@ func testRunSimple(t *testing.T) {
 }
 
 func testConfigure(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	senderManager := mocksender.CreateDefaultDemultiplexer()
 	c, err := NewPythonFakeCheck(senderManager)
@@ -597,8 +576,7 @@ func testConfigure(t *testing.T) {
 }
 
 func testConfigureDeprecated(t *testing.T) {
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	senderManager := mocksender.CreateDefaultDemultiplexer()
 	c, err := NewPythonFakeCheck(senderManager)
@@ -636,8 +614,7 @@ func testConfigureDeprecated(t *testing.T) {
 func testGetDiagnoses(t *testing.T) {
 	C.reset_check_mock()
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	check, err := NewPythonFakeCheck(aggregator.NewNoOpSenderManager())
 	if !assert.Nil(t, err) {
diff --git a/pkg/collector/python/test_loader.go b/pkg/collector/python/test_loader.go
index f8a4051c0c2f..4fd9ead370f5 100644
--- a/pkg/collector/python/test_loader.go
+++ b/pkg/collector/python/test_loader.go
@@ -11,13 +11,14 @@ import (
 	"runtime"
 	"testing"
 
+	"github.com/stretchr/testify/assert"
+
 	"github.com/DataDog/datadog-agent/comp/core/autodiscovery/integration"
 	nooptagger "github.com/DataDog/datadog-agent/comp/core/tagger/impl-noop"
 	integrations "github.com/DataDog/datadog-agent/comp/logs/integrations/def"
 	"github.com/DataDog/datadog-agent/pkg/aggregator/mocksender"
 	pkgconfigsetup "github.com/DataDog/datadog-agent/pkg/config/setup"
 	"github.com/DataDog/datadog-agent/pkg/util/option"
-	"github.com/stretchr/testify/assert"
 )
 
 /*
@@ -157,8 +158,8 @@ func testLoadCustomCheck(t *testing.T) {
 		InitConfig: integration.Data("{}"),
 	}
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
+
 	senderManager := mocksender.CreateDefaultDemultiplexer()
 	logReceiver := option.None[integrations.Component]()
 	tagger := nooptagger.NewComponent()
@@ -195,8 +196,7 @@ func testLoadWheelCheck(t *testing.T) {
 		InitConfig: integration.Data("{}"),
 	}
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
 
 	senderManager := mocksender.CreateDefaultDemultiplexer()
 	logReceiver := option.None[integrations.Component]()
@@ -233,8 +233,8 @@ func testLoadHACheck(t *testing.T) {
 		InitConfig: integration.Data("{}"),
 	}
 
-	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	mockRtloader(t)
+
 	senderManager := mocksender.CreateDefaultDemultiplexer()
 	logReceiver := option.None[integrations.Component]()
 	tagger := nooptagger.NewComponent()
diff --git a/pkg/collector/python/test_util.go b/pkg/collector/python/test_util.go
index 02ffca458721..6081bffe5192 100644
--- a/pkg/collector/python/test_util.go
+++ b/pkg/collector/python/test_util.go
@@ -8,6 +8,7 @@
 package python
 
 import (
+	"sync"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -76,3 +77,15 @@ func testGetSubprocessOutputError(t *testing.T) {
 	assert.NotEqual(t, C.int(0), cRetCode)
 	assert.Nil(t, exception)
 }
+
+func mockRtloader(t *testing.T) {
+	rtloader = newMockRtLoaderPtr()
+	pythonOnce.Do(func() {})
+	t.Cleanup(func() {
+		// We have to wrap this in locks otherwise the race detector complains
+		pyDestroyLock.Lock()
+		rtloader = nil
+		pythonOnce = sync.Once{}
+		pyDestroyLock.Unlock()
+	})
+}
diff --git a/pkg/config/setup/config.go b/pkg/config/setup/config.go
index 1e1443763b6b..4617cc52418d 100644
--- a/pkg/config/setup/config.go
+++ b/pkg/config/setup/config.go
@@ -328,6 +328,10 @@ func InitConfig(config pkgconfigmodel.Setup) {
 	// library support will not work reliably in those environments)
 	config.BindEnvAndSetDefault("allow_python_path_heuristics_failure", false)
 
+	// If true, Python is loaded when the first Python check is loaded.
+	// Otherwise, Python is loaded when the collector is initialized.
+	config.BindEnvAndSetDefault("python_lazy_loading", true)
+
 	// if/when the default is changed to true, make the default platform
 	// dependent; default should remain false on Windows to maintain backward
 	// compatibility with Agent5 behavior/win
diff --git a/releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml b/releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
new file mode 100644
index 000000000000..84fca932805d
--- /dev/null
+++ b/releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
@@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually avoid having to load Python when no Python check is enabled.
+    This change can be reverted by setting ``python_lazy_loading: false`` in your configuration.


Reviews:
user: login: GustavoCaso
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:50:07Z
user: login: pgimalac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T17:00:27Z
user: login: GustavoCaso
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-25T09:59:23Z
user: login: pgimalac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-25T13:52:02Z
user: login: GustavoCaso
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-25T14:24:42Z
user: login: rahulkaukuntla
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-25T18:08:12Z
user: login: vickenty
body: This is a significant change and definitely needs to be called out in the changelog, and point out the fallback configuration.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T16:17:25Z
user: login: vickenty
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T16:20:02Z
user: login: pgimalac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T16:59:09Z
user: login: michaelcretzman
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T22:09:25Z
user: login: pgimalac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:41:05Z
user: login: michaelcretzman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:03:06Z
user: login: michaelcretzman
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T18:04:01Z
user: login: nubtron
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T13:18:07Z
user: login: iliakur
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T14:07:43Z
user: login: vickenty
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T15:27:00Z

Review comments:
diff_hunk: @@ -293,7 +293,7 @@ func run(
 
 	// TODO: (components) - Until the checks are components we set there context so they can depends on components.
 	check.InitializeInventoryChecksContext(invChecks)
-	pkgcollector.InitPython(common.GetPythonPaths()...)
+	python.InitPython(common.GetPythonPaths()...)
path: pkg/cli/subcommands/check/command.go
user: login: GustavoCaso
body: I want to challenge the need to initialize python here. I know it was here before your change, still it might have being here for historical reasons, but is not required 😄 
created_at: 2025-02-24T16:50:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -293,7 +293,7 @@ func run(
 
 	// TODO: (components) - Until the checks are components we set there context so they can depends on components.
 	check.InitializeInventoryChecksContext(invChecks)
-	pkgcollector.InitPython(common.GetPythonPaths()...)
+	python.InitPython(common.GetPythonPaths()...)
path: pkg/cli/subcommands/check/command.go
user: login: pgimalac
body: Yeah I agree this line can be removed ! Good catch !

Before my change I think this is required though, because the check command doesn't initialize the collector (so Python might not be initialized) https://github.com/DataDog/datadog-agent/pull/22699
created_at: 2025-02-24T17:00:27Z
author_association: MEMBER
diff_hunk: @@ -139,7 +140,11 @@ func testLoadCustomCheck(t *testing.T) {
 	}
 
 	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	pythonOnce.Do(func() {})
path: pkg/collector/python/test_loader.go
user: login: GustavoCaso
body: Could we use `mockRtloader` utility test function on this tests as well? 
created_at: 2025-02-25T09:59:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,7 +140,11 @@ func testLoadCustomCheck(t *testing.T) {
 	}
 
 	rtloader = newMockRtLoaderPtr()
-	defer func() { rtloader = nil }()
+	pythonOnce.Do(func() {})
path: pkg/collector/python/test_loader.go
user: login: pgimalac
body: Yes good idea ! Done
created_at: 2025-02-25T13:52:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
+    This change can be reverted by setting `python_lazy_loading: false` in your configuration.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: vickenty
body: Changelog is in RST format which uses double backticks for verbatim text.
created_at: 2025-03-04T16:18:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
+    This change can be reverted by setting `python_lazy_loading: false` in your configuration.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: pgimalac
body: Ah yes I forgot about that 👍 
created_at: 2025-03-04T16:59:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: michaelcretzman
body: ```suggestion
    This will eventually prevent the loading of Python when no Python check is enabled.
```
created_at: 2025-03-04T22:08:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: michaelcretzman
body: I think this is clearer, unless I'm misunderstand what you're saying here.
created_at: 2025-03-04T22:09:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: pgimalac
body: @michaelcretzman I'm not sure that's quite equivalent to what I mean 🤔 
It's not that we are accidentally loading Python, we're doing it on purpose, but we'd like to be able not to do it

What do you think of the following ?
```suggestion
    This will eventually avoid having to load Python when no Python check is enabled.
```
created_at: 2025-03-05T09:41:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+enhancements:
+  - |
+    The Python runtime is now loaded when the first check is loaded, rather than when the agent starts.
+    This will eventually allow not loading Python when no Python check is enabled.
path: releasenotes/notes/rtloader-python-lazy-loading-f16eed252c3176e7.yaml
user: login: michaelcretzman
body: that's clearer. thanks!
created_at: 2025-03-05T18:03:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: agent-platform-auto-pr[bot]
created_at: 2025-01-02T13:35:32Z
body: ## Uncompressed package size comparison

Comparison with [ancestor](https://github.com/DataDog/datadog-agent/commit/c5aa0b05a5665af92f6a84b2c577c1800f996ade) `c5aa0b05a5665af92f6a84b2c577c1800f996ade`


<details>
  <summary> Diff per package </summary>

|package|diff|status|size|ancestor|threshold|
|--|--|--|--|--|--|
|datadog-agent-x86_64-rpm|0.00MB|✅|819.78MB|819.77MB|0.50MB|
|datadog-agent-x86_64-suse|0.00MB|✅|819.78MB|819.77MB|0.50MB|
|datadog-agent-amd64-deb|0.00MB|✅|809.98MB|809.98MB|0.50MB|
|datadog-dogstatsd-arm64-deb|0.00MB|✅|37.86MB|37.85MB|0.50MB|
|datadog-agent-aarch64-rpm|0.00MB|✅|810.70MB|810.70MB|0.50MB|
|datadog-agent-arm64-deb|0.00MB|✅|800.93MB|800.92MB|0.50MB|
|datadog-dogstatsd-amd64-deb|0.00MB|✅|39.33MB|39.33MB|0.50MB|
|datadog-dogstatsd-x86_64-rpm|0.00MB|✅|39.41MB|39.41MB|0.50MB|
|datadog-dogstatsd-x86_64-suse|0.00MB|✅|39.41MB|39.41MB|0.50MB|
|datadog-heroku-agent-amd64-deb|0.00MB|✅|441.20MB|441.20MB|0.50MB|
|datadog-iot-agent-amd64-deb|0.00MB|✅|62.17MB|62.17MB|0.50MB|
|datadog-iot-agent-x86_64-rpm|0.00MB|✅|62.24MB|62.24MB|0.50MB|
|datadog-iot-agent-x86_64-suse|0.00MB|✅|62.24MB|62.24MB|0.50MB|
|datadog-iot-agent-arm64-deb|0.00MB|✅|59.40MB|59.40MB|0.50MB|
|datadog-iot-agent-aarch64-rpm|0.00MB|✅|59.47MB|59.47MB|0.50MB|

</details>

## Decision
✅ Passed



author_association: CONTRIBUTOR
user: login: agent-platform-auto-pr[bot]
created_at: 2025-01-02T13:39:47Z
body: ## Test changes on VM

Use this command from [test-infra-definitions](https://github.com/DataDog/test-infra-definitions) to manually test this PR changes on a VM:
```sh
dda inv aws.create-vm --pipeline-id=58548026 --os-family=ubuntu
```
Note: This applies to commit **939842d7**
author_association: CONTRIBUTOR
user: login: cit-pr-commenter[bot]
created_at: 2025-01-02T14:40:27Z
body: ## Regression Detector

# Regression Detector Results

[Metrics dashboard](https://app.datadoghq.com/dashboard/ykh-ua8-vcu/SMP-Regression-Detector-Metrics?fromUser=true&refresh_mode=paused&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false)  
[Target profiles](https://app.datadoghq.com/profiling/explorer?query=env%3Asingle-machine-performance%20service%3Adatadog-agent%20job_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&viz=stream&start=1741776907000&end=1741788307000&paused=true)  
Run ID: 279ad9f2-22fd-4cf2-9739-1e9fd3a14106

Baseline: c5aa0b05a5665af92f6a84b2c577c1800f996ade
Comparison: 939842d70581830aaab74fef51dbacfb26966f3b
[Diff](c5aa0b05a5665af92f6a84b2c577c1800f996ade...939842d70581830aaab74fef51dbacfb26966f3b)


## Optimization Goals: ✅ No significant changes detected


<details>
<summary><h2>
Fine details of change detection per experiment
</h2></summary>

| perf | experiment                                   | goal               | Δ mean % | Δ mean % CI    | trials | links                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|------|----------------------------------------------|--------------------|----------|----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ➖   | uds_dogstatsd_to_api_cpu                     | % cpu utilization  | +0.85    | [+0.03, +1.66] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api_cpu%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                   |
| ➖   | file_to_blackhole_1000ms_latency             | egress throughput  | +0.71    | [-0.07, +1.49] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                           |
| ➖   | file_to_blackhole_500ms_latency              | egress throughput  | +0.07    | [-0.73, +0.86] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_500ms_latency%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | file_tree                                    | memory utilization | +0.06    | [-0.06, +0.18] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_tree%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                                  |
| ➖   | file_to_blackhole_0ms_latency_http1          | egress throughput  | +0.01    | [-0.81, +0.83] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http1%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                        |
| ➖   | file_to_blackhole_300ms_latency              | egress throughput  | +0.01    | [-0.62, +0.64] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_300ms_latency%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | file_to_blackhole_100ms_latency              | egress throughput  | +0.00    | [-0.69, +0.69] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_100ms_latency%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                            |
| ➖   | tcp_dd_logs_filter_exclude                   | ingress throughput | -0.00    | [-0.02, +0.02] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_dd_logs_filter_exclude%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                 |
| ➖   | uds_dogstatsd_to_api                         | ingress throughput | -0.02    | [-0.28, +0.25] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                       |
| ➖   | file_to_blackhole_0ms_latency                | egress throughput  | -0.05    | [-0.87, +0.77] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                              |
| ➖   | file_to_blackhole_0ms_latency_http2          | egress throughput  | -0.07    | [-0.94, +0.81] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http2%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                        |
| ➖   | file_to_blackhole_1000ms_latency_linear_load | egress throughput  | -0.08    | [-0.55, +0.39] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency_linear_load%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                               |
| ➖   | quality_gate_idle_all_features               | memory utilization | -0.39    | [-0.45, -0.32] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle_all_features%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false) |
| ➖   | uds_dogstatsd_20mb_12k_contexts_20_senders   | memory utilization | -0.95    | [-1.02, -0.87] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_20mb_12k_contexts_20_senders%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                 |
| ➖   | tcp_syslog_to_blackhole                      | ingress throughput | -1.01    | [-1.07, -0.95] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_syslog_to_blackhole%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                    |
| ➖   | quality_gate_idle                            | memory utilization | -1.35    | [-1.41, -1.28] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false)                           |
| ➖   | quality_gate_logs                            | % cpu utilization  | -2.20    | [-4.93, +0.53] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_logs%20run_id%3A279ad9f2-22fd-4cf2-9739-1e9fd3a14106&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1741776907000&to_ts=1741788307000&live=false)                                                                                                                                                                                                                                                                                                                                                                          |
</details>


<details>
<summary><h2>
Bounds Checks: ✅ Passed
</h2></summary>

| perf | experiment                                   | bounds_check_name  | replicates_passed | links                                                                                                                                                                                                                                                                                                                                                       |
|------|----------------------------------------------|--------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ✅   | file_to_blackhole_0ms_latency                | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency                | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http1          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http1          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http2          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_0ms_latency_http2          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_1000ms_latency             | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_1000ms_latency_linear_load | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_100ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_100ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_300ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_300ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_500ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | file_to_blackhole_500ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_idle                            | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false)              |
| ✅   | quality_gate_idle                            | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false)              |
| ✅   | quality_gate_idle_all_features               | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false) |
| ✅   | quality_gate_idle_all_features               | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&tpl_var_run-id%5B0%5D=279ad9f2-22fd-4cf2-9739-1e9fd3a14106&view=spans&from_ts=1741784107000&to_ts=1741784707000&live=false) |
| ✅   | quality_gate_logs                            | intake_connections | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_logs                            | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
| ✅   | quality_gate_logs                            | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                                                                             |
</details>


<details>
<summary><h2>
Explanation
</h2></summary>

**Confidence level:** 90.00%
**Effect size tolerance:** |Δ mean %| ≥ 5.00%

Performance changes are noted in the **perf** column of each table:

* ✅ = significantly better comparison variant performance
* ❌ = significantly worse comparison variant performance
* ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".
</details>

## CI Pass/Fail Decision

✅ **Passed.** All Quality Gates passed.

- **quality_gate_logs**, bounds check **lost_bytes**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
author_association: NONE
user: login: agent-platform-auto-pr[bot]
created_at: 2025-02-24T16:45:10Z
body: ## Static quality checks ✅

Please find below the results from static quality gates


<details>
<summary>Successful checks</summary>

### Info

|Result|Quality gate|On disk size|On disk size limit|On wire size|On wire size limit|
|----|----|----|----|----|----|
|✅|static_quality_gate_agent_deb_amd64|784.0 MiB|801.8 MiB|191.12 MiB|202.62 MiB|
|✅|static_quality_gate_agent_deb_arm64|775.32 MiB|793.14 MiB|172.92 MiB|184.51 MiB|
|✅|static_quality_gate_agent_rpm_amd64|783.99 MiB|801.79 MiB|193.14 MiB|205.03 MiB|
|✅|static_quality_gate_agent_rpm_arm64|775.37 MiB|793.09 MiB|175.43 MiB|186.44 MiB|
|✅|static_quality_gate_agent_suse_amd64|784.04 MiB|801.81 MiB|193.14 MiB|205.03 MiB|
|✅|static_quality_gate_agent_suse_arm64|775.37 MiB|793.14 MiB|175.43 MiB|186.44 MiB|
|✅|static_quality_gate_dogstatsd_deb_amd64|37.58 MiB|47.67 MiB|9.74 MiB|19.78 MiB|
|✅|static_quality_gate_dogstatsd_deb_arm64|36.18 MiB|46.27 MiB|8.45 MiB|18.49 MiB|
|✅|static_quality_gate_dogstatsd_rpm_amd64|37.58 MiB|47.67 MiB|9.75 MiB|19.79 MiB|
|✅|static_quality_gate_dogstatsd_suse_amd64|37.58 MiB|47.67 MiB|9.75 MiB|19.79 MiB|
|✅|static_quality_gate_iot_agent_deb_amd64|59.37 MiB|69.0 MiB|14.91 MiB|24.8 MiB|
|✅|static_quality_gate_iot_agent_deb_arm64|56.72 MiB|66.4 MiB|12.88 MiB|22.8 MiB|
|✅|static_quality_gate_iot_agent_rpm_amd64|59.37 MiB|69.0 MiB|14.94 MiB|24.8 MiB|
|✅|static_quality_gate_iot_agent_rpm_arm64|56.72 MiB|66.4 MiB|12.87 MiB|22.8 MiB|
|✅|static_quality_gate_iot_agent_suse_amd64|59.37 MiB|69.0 MiB|14.94 MiB|24.8 MiB|
|✅|static_quality_gate_docker_agent_amd64|868.61 MiB|886.12 MiB|292.18 MiB|304.21 MiB|
|✅|static_quality_gate_docker_agent_arm64|883.24 MiB|900.79 MiB|278.53 MiB|290.47 MiB|
|✅|static_quality_gate_docker_agent_jmx_amd64|1.04 GiB|1.06 GiB|367.28 MiB|379.33 MiB|
|✅|static_quality_gate_docker_agent_jmx_arm64|1.04 GiB|1.06 GiB|349.6 MiB|361.55 MiB|
|✅|static_quality_gate_docker_dogstatsd_amd64|45.73 MiB|55.78 MiB|17.25 MiB|27.28 MiB|
|✅|static_quality_gate_docker_dogstatsd_arm64|44.36 MiB|54.45 MiB|16.12 MiB|26.16 MiB|
|✅|static_quality_gate_docker_cluster_agent_amd64|264.87 MiB|274.78 MiB|106.3 MiB|116.28 MiB|
|✅|static_quality_gate_docker_cluster_agent_arm64|280.8 MiB|290.82 MiB|101.12 MiB|111.12 MiB|

</details>

author_association: CONTRIBUTOR
user: login: pgimalac
created_at: 2025-03-11T10:04:54Z
body: /merge
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-11T10:05:00Z
body: <!-- "FeedbackWorkflowId":"1978b3d9-ce36-4c27-a176-b5e93173c665_36" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/1978b3d9-ce36-4c27-a176-b5e93173c665_36)
`2025-03-11 10:04:59 UTC` :information_source: Start processing command `/merge`

---
`2025-03-11 10:05:14 UTC` :information_source: **MergeQueue:** pull request added to the queue

The median merge time in `main` is 32m.

---
`2025-03-11 10:05:34 UTC` :warning: **MergeQueue:** This merge request was unqueued

pierre.gimalac@datadoghq.com unqueued this merge request

author_association: NONE
user: login: pgimalac
created_at: 2025-03-11T10:05:25Z
body: /merge -c
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-11T10:05:30Z
body: <!-- "FeedbackWorkflowId":"fc5378d2-4075-42aa-96fb-f7b386d005d5_36" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/fc5378d2-4075-42aa-96fb-f7b386d005d5_36)
`2025-03-11 10:05:30 UTC` :information_source: Start processing command `/merge -c`

author_association: NONE
user: login: pgimalac
created_at: 2025-03-11T10:06:20Z
body: /trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-11T10:06:24Z
body: <!-- "FeedbackWorkflowId":"08c28dee-ccdd-434b-9c3c-2bd1530fa9d0_21" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/08c28dee-ccdd-434b-9c3c-2bd1530fa9d0_21)
`2025-03-11 10:06:23 UTC` :information_source: Start processing command `/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on`

---
`2025-03-11 10:07:16 UTC` :information_source: Gitlab pipeline started

Started pipeline [#58387981](https://gitlab.ddbuild.io/DataDog/datadog-agent/-/pipelines/58387981)

author_association: NONE
user: login: pgimalac
created_at: 2025-03-12T10:07:42Z
body: /trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-12T10:07:45Z
body: <!-- "FeedbackWorkflowId":"ece19c01-0ea7-40f6-ab6c-9331b7ebc067_21" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/ece19c01-0ea7-40f6-ab6c-9331b7ebc067_21)
`2025-03-12 10:07:44 UTC` :information_source: Start processing command `/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on`

---
`2025-03-12 10:08:28 UTC` :information_source: Gitlab pipeline started

Started pipeline [#58536823](https://gitlab.ddbuild.io/DataDog/datadog-agent/-/pipelines/58536823)

author_association: NONE
user: login: pgimalac
created_at: 2025-03-12T12:28:36Z
body: /trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-12T12:28:39Z
body: <!-- "FeedbackWorkflowId":"6014eb38-64d0-4ca0-9327-fe59c55992f2_21" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/6014eb38-64d0-4ca0-9327-fe59c55992f2_21)
`2025-03-12 12:28:39 UTC` :information_source: Start processing command `/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on`

---
`2025-03-12 12:29:21 UTC` :information_source: Gitlab pipeline started

Started pipeline [#58548026](https://gitlab.ddbuild.io/DataDog/datadog-agent/-/pipelines/58548026)

author_association: NONE
user: login: pgimalac
created_at: 2025-03-12T14:17:45Z
body: /merge
author_association: MEMBER
user: login: dd-devflow[bot]
created_at: 2025-03-12T14:17:49Z
body: <!-- "FeedbackWorkflowId":"15c6b8da-28ef-416f-924d-e0cb7952787b_36" -->
[View all feedbacks in Devflow UI.](https://devflow.static-app.us1.prod.dog/logs/15c6b8da-28ef-416f-924d-e0cb7952787b_36)
`2025-03-12 14:17:49 UTC` :information_source: Start processing command `/merge`

---
`2025-03-12 14:17:58 UTC` :information_source: **MergeQueue:** pull request added to the queue

The expected merge time in `main` is approximately 46m (p90).

---
`2025-03-12 14:55:31 UTC` :information_source: **MergeQueue:** This merge request was merged

author_association: NONE