Repository: DataDog/datadog-agent
Pull request #36879: [AGTMETRICS-234]Update sanitizer when API keys change
Author: login: StephenWakely
Labels: name: qa/done
description: QA done before merge and regressions are covered by tests
name: medium review
description: PR review might take time
name: ask-review
description: Ask required teams to review this PR
name: team/agent-metric-pipelines
Created: 2025-05-12T15:57:05Z | Merged: 2025-05-14T11:05:46Z

### What does this PR do?

Ensures the api key replacer replaces the API key after an API key has been refreshed.

### Motivation

After the API is updated, the serializer needs to update it's replacers so that it can replace the new API keys with a placeholder.

Previously the replacers were sorting the API keys and restoring based on position. I have removed the sort, since the order would change if the API key changed, thus restoring to the wrong API key. We now rely only on the position the key is in the configuration.

## Caveats

- An additional complexity is that API keys can be deduped. Theoretically you can have a configuration like this:

```
api_key: ENC[mainkee]
additional_endpoints:
  "http://localhost:8080":
  - ENC[kee1]
```

`mainkee` and `kee1` could return the same API key. The forwarders dedup this to ensure only one transaction is sent to any given endpoint/api key pair. At runtime, the keys could be changed so they are no longer the same. If this happens, the number of API keys will change and thus be in different positions. So we need to make sure we store the positions prior to deduping.

- It is possible that both `mainkee` and `kee1` start as different keys, transactions are created and stored to disk. Then these keys could be updated to be the same key. Transactions intended for `mainkee` and `kee1` would contain the same API key now and thus duplicate keys could be sent to the endpoint.

- Whilst the agent is stopped, the config could be changed such that the order of the API keys are altered. On starting the agent, the agent would restore these transactions and send them to the wrong API key. I don't think there is any way to fix this... 


### Describe how you validated your changes

I have added some unit tests. 

Have tested manually with this config:

```
dd_url: http://localhost:8080
api_key: ENC[mainkee]

forwarder_storage_max_size_in_bytes: 50000000
forwarder_retry_queue_payloads_max_size: 10

additional_endpoints:
  "http://localhost:8080":
  - ENC[kee1]
``` 

Updating the secrets, then check the files created in `/opt/datadog-agent/run/transactions_to_retry/core/**/*` to ensure they do not contain an API key in them.

### Possible Drawbacks / Trade-offs

### Additional Notes
<!--
* Anything else we should know when reviewing?
* Include benchmarking information here whenever possible.
* Include info about alternatives that were considered and why the proposed
  version was chosen.
-->


Diff:
diff --git a/comp/forwarder/defaultforwarder/default_forwarder.go b/comp/forwarder/defaultforwarder/default_forwarder.go
index d4f753a52b43..15ceaf69b74d 100644
--- a/comp/forwarder/defaultforwarder/default_forwarder.go
+++ b/comp/forwarder/defaultforwarder/default_forwarder.go
@@ -134,9 +134,10 @@ func NewOptions(config config.Component, log log.Component, keysPerDomain map[st
 	}
 	if r, ok := resolvers[utils.GetInfraEndpoint(config)]; ok && vectorMetricsURL != "" {
 		log.Debugf("Configuring forwarder to send metrics to observability_pipelines_worker: %s", vectorMetricsURL)
+		apiKeys, _ := r.GetAPIKeysInfo()
 		resolver, err := pkgresolver.NewDomainResolverWithMetricToVector(
 			r.GetBaseDomain(),
-			r.GetAPIKeysInfo(),
+			apiKeys,
 			vectorMetricsURL,
 		)
 		if err != nil {
diff --git a/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go b/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
index e38d5e4853df..c3952ad3ae0c 100644
--- a/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
+++ b/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
@@ -11,46 +11,62 @@ import (
 	"net/http"
 	"sort"
 	"strings"
+	"sync"
 	"time"
 
 	log "github.com/DataDog/datadog-agent/comp/core/log/def"
 	"github.com/DataDog/datadog-agent/comp/forwarder/defaultforwarder/resolver"
 	"github.com/DataDog/datadog-agent/comp/forwarder/defaultforwarder/transaction"
+	"github.com/DataDog/datadog-agent/pkg/config/utils"
+	"github.com/DataDog/datadog-agent/pkg/telemetry"
 	"github.com/DataDog/datadog-agent/pkg/util/common"
 
 	proto "github.com/golang/protobuf/proto"
 )
 
-const transactionsSerializerVersion = 1
+const transactionsSerializerVersion = 2
 
 // Use an non US ASCII char as a separator (Should neither appear in an HTTP header value nor in a URL).
 const squareChar = "\xfe"
 const placeHolderPrefix = squareChar + "API_KEY" + squareChar
 const placeHolderFormat = placeHolderPrefix + "%v" + squareChar
 
+var tlmV1TransactionsDeserialized = telemetry.NewCounterWithOpts("transactions", "v1deserialized", []string{}, "", telemetry.Options{DefaultMetric: true})
+
 // HTTPTransactionsSerializer serializes Transaction instances.
 // To support a new Transaction implementation, add a new
 // method `func (s *HTTPTransactionsSerializer) Add(transaction NEW_TYPE) error {`
 type HTTPTransactionsSerializer struct {
-	log                 log.Component
-	collection          HttpTransactionProtoCollection
-	apiKeyToPlaceholder *strings.Replacer
-	placeholderToAPIKey *strings.Replacer
-	resolver            resolver.DomainResolver
+	log        log.Component
+	collection HttpTransactionProtoCollection
+
+	currentKeyVersion     int
+	placeholderMutex      sync.RWMutex
+	apiKeyToPlaceholder   []*strings.Replacer
+	placeholderToAPIKey   *strings.Replacer
+	placeholderToAPIKeyV1 *strings.Replacer
+
+	resolver resolver.DomainResolver
 }
 
 // NewHTTPTransactionsSerializer creates a new instance of HTTPTransactionsSerializer
 func NewHTTPTransactionsSerializer(log log.Component, resolver resolver.DomainResolver) *HTTPTransactionsSerializer {
-	apiKeyToPlaceholder, placeholderToAPIKey := createReplacers(resolver.GetAPIKeys())
+	keys, version := resolver.GetAPIKeysInfo()
+	apiKeyToPlaceholder, placeholderToAPIKey := createReplacers(keys, []*strings.Replacer{})
+
+	dedupedKeys := resolver.GetAPIKeys()
+	placeholderToAPIKeyV1 := createReplacerV1(dedupedKeys)
 
 	return &HTTPTransactionsSerializer{
 		log: log,
 		collection: HttpTransactionProtoCollection{
 			Version: transactionsSerializerVersion,
 		},
-		apiKeyToPlaceholder: apiKeyToPlaceholder,
-		placeholderToAPIKey: placeholderToAPIKey,
-		resolver:            resolver,
+		apiKeyToPlaceholder:   apiKeyToPlaceholder,
+		placeholderToAPIKey:   placeholderToAPIKey,
+		placeholderToAPIKeyV1: placeholderToAPIKeyV1,
+		currentKeyVersion:     version,
+		resolver:              resolver,
 	}
 }
 
@@ -80,6 +96,8 @@ func (s *HTTPTransactionsSerializer) Add(transaction *transaction.HTTPTransactio
 		pointCount = int32(transaction.Payload.GetPointCount())
 	}
 
+	s.checkAPIKeyUpdate()
+
 	endpoint := transaction.Endpoint
 	transactionProto := HttpTransactionProto{
 		// The domain is not stored on the disk for security reasons.
@@ -101,6 +119,21 @@ func (s *HTTPTransactionsSerializer) Add(transaction *transaction.HTTPTransactio
 	return nil
 }
 
+// checkAPIKeyUpdate checks if the resolver has updated it's API keys - it does this by maintaining a version
+// number that increments with every update. If they have been updated, we reload the new set of keys and
+// update our replacers.
+func (s *HTTPTransactionsSerializer) checkAPIKeyUpdate() {
+	if s.resolver.GetAPIKeyVersion() != s.currentKeyVersion {
+		// API keys have been updated so we need to rebuild.
+		s.placeholderMutex.Lock()
+		defer s.placeholderMutex.Unlock()
+
+		keys, version := s.resolver.GetAPIKeysInfo()
+		s.apiKeyToPlaceholder, s.placeholderToAPIKey = createReplacers(keys, s.apiKeyToPlaceholder)
+		s.currentKeyVersion = version
+	}
+}
+
 // GetBytesAndReset returns as bytes the serialized transactions and reset
 // the transaction collection.
 func (s *HTTPTransactionsSerializer) GetBytesAndReset() ([]byte, error) {
@@ -117,6 +150,8 @@ func (s *HTTPTransactionsSerializer) Deserialize(bytes []byte) ([]transaction.Tr
 		return nil, 0, err
 	}
 
+	s.checkAPIKeyUpdate()
+
 	var httpTransactions []transaction.Transaction
 	errorCount := 0
 	for _, tr := range collection.Values {
@@ -127,9 +162,9 @@ func (s *HTTPTransactionsSerializer) Deserialize(bytes []byte) ([]transaction.Tr
 
 		priority, err := fromTransactionPriorityProto(tr.Priority)
 		if err == nil {
-			route, err = s.restoreAPIKeys(e.Route)
+			route, err = s.restoreAPIKeys(e.Route, collection.Version)
 			if err == nil {
-				proto, err = s.fromHeaderProto(tr.Headers)
+				proto, err = s.fromHeaderProto(tr.Headers, collection.Version)
 				if err == nil { // TODO: the reason for this nesting pattern is unclear to me
 					destination, err = fromTransactionDestinationProto(tr.Destination)
 				}
@@ -162,12 +197,38 @@ func (s *HTTPTransactionsSerializer) Deserialize(bytes []byte) ([]transaction.Tr
 	return httpTransactions, errorCount, nil
 }
 
+// replaceAPIKeys iterates backward through our list of replacer, returns when
+// one of them is able to replace the key.
+// Needs to iterate backwards to ensure the latest version of the replacers takes
+// priority.
 func (s *HTTPTransactionsSerializer) replaceAPIKeys(str string) string {
-	return s.apiKeyToPlaceholder.Replace(str)
+	s.placeholderMutex.RLock()
+	defer s.placeholderMutex.RUnlock()
+
+	for replacer := len(s.apiKeyToPlaceholder) - 1; replacer >= 0; replacer-- {
+		replaced := s.apiKeyToPlaceholder[replacer].Replace(str)
+		if str != replaced {
+			return replaced
+		}
+	}
+
+	return str
+
 }
 
-func (s *HTTPTransactionsSerializer) restoreAPIKeys(str string) (string, error) {
-	newStr := s.placeholderToAPIKey.Replace(str)
+func (s *HTTPTransactionsSerializer) restoreAPIKeys(str string, protoVersion int32) (string, error) {
+	var newStr string
+	if protoVersion == 1 {
+		// Handle transactions serialized in a prior version
+		newStr = s.placeholderToAPIKeyV1.Replace(str)
+		if newStr != str {
+			tlmV1TransactionsDeserialized.Inc()
+		}
+	} else {
+		s.placeholderMutex.RLock()
+		newStr = s.placeholderToAPIKey.Replace(str)
+		s.placeholderMutex.RUnlock()
+	}
 
 	if strings.Contains(newStr, placeHolderPrefix) {
 		return "", errors.New("cannot restore the transaction as an API Key is missing")
@@ -175,12 +236,12 @@ func (s *HTTPTransactionsSerializer) restoreAPIKeys(str string) (string, error)
 	return newStr, nil
 }
 
-func (s *HTTPTransactionsSerializer) fromHeaderProto(headersProto map[string]*HeaderValuesProto) (http.Header, error) {
+func (s *HTTPTransactionsSerializer) fromHeaderProto(headersProto map[string]*HeaderValuesProto, protoVersion int32) (http.Header, error) {
 	headers := make(http.Header)
 	for key, headerValuesProto := range headersProto {
 		var headerValues []string
 		for _, v := range headerValuesProto.Values {
-			value, err := s.restoreAPIKeys(v)
+			value, err := s.restoreAPIKeys(v, protoVersion)
 			if err != nil {
 				return nil, err
 			}
@@ -248,19 +309,41 @@ func toTransactionDestinationProto(destination transaction.Destination) (Transac
 	}
 }
 
-func createReplacers(apiKeys []string) (*strings.Replacer, *strings.Replacer) {
+// createReplacersV1 creates replacers for transactions created with V1 of the serializer.
+// This version sorted the keys prior to serializing. The location in this list is used to
+// unscrub the key.
+func createReplacerV1(apiKeys []string) *strings.Replacer {
 	// Copy to not modify apiKeys order
 	keys := make([]string, len(apiKeys))
 	copy(keys, apiKeys)
-
 	// Sort to always have the same order
 	sort.Strings(keys)
-	var apiKeyPlaceholder []string
 	var placeholderToAPIKey []string
 	for i, k := range keys {
 		placeholder := fmt.Sprintf(placeHolderFormat, i)
-		apiKeyPlaceholder = append(apiKeyPlaceholder, k, placeholder)
 		placeholderToAPIKey = append(placeholderToAPIKey, placeholder, k)
 	}
-	return strings.NewReplacer(apiKeyPlaceholder...), strings.NewReplacer(placeholderToAPIKey...)
+
+	return strings.NewReplacer(placeholderToAPIKey...)
+}
+
+// createReplacers will create the replacers from and to an API key.
+// This relies on the position that the key is found in our list. If the position of the keys in the config
+// are changed, then the transaction will be restored to a different, likely wrong, API key.
+// We take the list of current APIKey->Placeholder replacers and append the new one to it because we need to
+// maintain a history of API keys since a transaction being serialized may contain an old API key and we
+// still need to make sure we can replace that one.
+func createReplacers(apiKeys []utils.APIKeys, currentAPIKeyPlaceholder []*strings.Replacer) ([]*strings.Replacer, *strings.Replacer) {
+	var apiKeyPlaceholder []string
+	var placeholderToAPIKey []string
+	index := 0
+	for path := range apiKeys {
+		for _, key := range apiKeys[path].Keys {
+			placeholder := fmt.Sprintf(placeHolderFormat, index)
+			apiKeyPlaceholder = append(apiKeyPlaceholder, key, placeholder)
+			placeholderToAPIKey = append(placeholderToAPIKey, placeholder, key)
+			index++
+		}
+	}
+	return append(currentAPIKeyPlaceholder, strings.NewReplacer(apiKeyPlaceholder...)), strings.NewReplacer(placeholderToAPIKey...)
 }
diff --git a/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go b/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
index bb10dbe595fd..fc3ec13513f0 100644
--- a/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
+++ b/comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
@@ -22,6 +22,9 @@ import (
 
 const apiKey1 = "apiKey1"
 const apiKey2 = "apiKey2"
+const apiKey3 = "apiKey3"
+const apiKey4 = "apiKey4"
+const apiKey5 = "apiKey5"
 const domain = "domain"
 const vectorDomain = "vectorDomain"
 
@@ -112,6 +115,89 @@ func TestHTTPTransactionSerializerMissingAPIKey(t *testing.T) {
 	r.Equal(1, errorCount)
 }
 
+func TestHTTPTransactionSerializerUpdateAPIKey(t *testing.T) {
+	r := require.New(t)
+	log := logmock.New(t)
+	res, err := resolver.NewSingleDomainResolver(domain, []utils.APIKeys{utils.NewAPIKeys("additional_endpoints", apiKey1, apiKey2)})
+	r.NoError(err)
+
+	serializer := NewHTTPTransactionsSerializer(log, res)
+
+	r.NoError(serializer.Add(createHTTPTransactionWithHeaderTests(http.Header{"Key": []string{apiKey1}}, domain)))
+	bytes, err := serializer.GetBytesAndReset()
+	r.NoError(err)
+
+	r.NotContains(string(bytes), apiKey1, "Serialized data should not contain %s", apiKey1)
+
+	// Update the API keys.
+	res.UpdateAPIKeys("additional_endpoints", []utils.APIKeys{utils.NewAPIKeys("additional_endpoints", apiKey4, apiKey2, apiKey3)})
+
+	r.NoError(serializer.Add(createHTTPTransactionWithHeaderTests(http.Header{"Key": []string{apiKey3}}, domain)))
+	bytes, err = serializer.GetBytesAndReset()
+	r.NoError(err)
+
+	// New API key should be scrubbed.
+	r.NotContains(string(bytes), apiKey3, "Serialized data should not contain %s", apiKey3)
+
+	// Ensure it can be restored
+	transactions, _, err := serializer.Deserialize(bytes)
+	r.NoError(err)
+	r.Contains(transactions[0].(*transaction.HTTPTransaction).Headers["Key"], apiKey3)
+}
+
+func TestHTTPTransactionSerializerUpdateDedupedAPIKey(t *testing.T) {
+	r := require.New(t)
+	log := logmock.New(t)
+
+	// apiKey1 is duplicated.
+	res, err := resolver.NewSingleDomainResolver(domain, []utils.APIKeys{
+		utils.NewAPIKeys("api_key", apiKey1),
+		utils.NewAPIKeys("additional_endpoints", apiKey1, apiKey2),
+	})
+	r.NoError(err)
+
+	serializer := NewHTTPTransactionsSerializer(log, res)
+
+	r.NoError(serializer.Add(createHTTPTransactionWithHeaderTests(http.Header{"Key": []string{apiKey1}}, domain)))
+	bytes, err := serializer.GetBytesAndReset()
+	r.NoError(err)
+
+	r.NotContains(string(bytes), apiKey1, "Serialized data should not contain %s", apiKey1)
+
+	// Update the API keys, there are now no duplicates.
+	res.UpdateAPIKeys("api_key", []utils.APIKeys{utils.NewAPIKeys("api_key", apiKey3)})
+	res.UpdateAPIKeys("additional_endpoints", []utils.APIKeys{utils.NewAPIKeys("additional_endpoints", apiKey4, apiKey5)})
+
+	// When this transaction is restored, what was apiKey1 could now be either apiKey3 or apiKey4
+	transactions, _, err := serializer.Deserialize(bytes)
+	r.NoError(err)
+	r.Contains(transactions[0].(*transaction.HTTPTransaction).Headers["Key"], apiKey3)
+}
+
+func TestHTTPTransactionSerializerUpdateAPIKeyBeforeSerializing(t *testing.T) {
+	r := require.New(t)
+	log := logmock.New(t)
+	res, err := resolver.NewSingleDomainResolver(domain, []utils.APIKeys{utils.NewAPIKeys("additional_endpoints", apiKey1, apiKey2)})
+	r.NoError(err)
+
+	serializer := NewHTTPTransactionsSerializer(log, res)
+	txn := createHTTPTransactionWithHeaderTests(http.Header{"Key": []string{apiKey1}}, domain)
+
+	// Update the API keys.
+	res.UpdateAPIKeys("additional_endpoints", []utils.APIKeys{utils.NewAPIKeys("additional_endpoints", apiKey4, apiKey2)})
+
+	r.NoError(serializer.Add(txn))
+
+	bytes, err := serializer.GetBytesAndReset()
+	r.NoError(err)
+
+	r.NotContains(string(bytes), apiKey1, "Serialized data should not contain %s", apiKey1)
+
+	transactions, _, err := serializer.Deserialize(bytes)
+	r.NoError(err)
+	r.Contains(transactions[0].(*transaction.HTTPTransaction).Headers["Key"], apiKey4)
+}
+
 func TestHTTPTransactionFieldsCount(t *testing.T) {
 	tr := transaction.HTTPTransaction{}
 	transactionType := reflect.TypeOf(tr)
diff --git a/comp/forwarder/defaultforwarder/resolver/domain_resolver.go b/comp/forwarder/defaultforwarder/resolver/domain_resolver.go
index 247cf5429667..743919694b93 100644
--- a/comp/forwarder/defaultforwarder/resolver/domain_resolver.go
+++ b/comp/forwarder/defaultforwarder/resolver/domain_resolver.go
@@ -40,11 +40,14 @@ type DomainResolver interface {
 	// destination type
 	Resolve(endpoint transaction.Endpoint) (string, DestinationType)
 	// GetAPIKeysInfo returns the list of API Keys and config paths associated with this `DomainResolver`
-	GetAPIKeysInfo() []utils.APIKeys
+	GetAPIKeysInfo() ([]utils.APIKeys, int)
 	// GetAPIKeys returns the list of API Keys associated with this `DomainResolver`
 	GetAPIKeys() []string
 	// UpdateAPIKeys updates the api keys at the given config path and sets the deduped keys to the new list.
 	UpdateAPIKeys(configPath string, newKeys []utils.APIKeys)
+	// GetAPIKeyVersion gets the current version for the API keys (version should be incremented each time the
+	// keys are updated).
+	GetAPIKeyVersion() int
 	// GetBaseDomain returns the base domain for this `DomainResolver`
 	GetBaseDomain() string
 	// GetAlternateDomains returns all the domains that can be returned by `Resolve()` minus the base domain
@@ -61,6 +64,7 @@ type DomainResolver interface {
 type SingleDomainResolver struct {
 	domain         string
 	apiKeys        []utils.APIKeys
+	keyVersion     int
 	dedupedAPIKeys []string
 	mu             sync.Mutex
 }
@@ -71,7 +75,8 @@ func OnUpdateConfig(resolver DomainResolver, log log.Component, config config.Co
 	config.OnUpdate(func(setting string, oldValue, newValue any) {
 		found := false
 
-		for _, endpoint := range resolver.GetAPIKeysInfo() {
+		apiKeys, _ := resolver.GetAPIKeysInfo()
+		for _, endpoint := range apiKeys {
 			if endpoint.ConfigSettingPath == setting {
 				found = true
 				break
@@ -156,10 +161,11 @@ func NewSingleDomainResolver(domain string, apiKeys []utils.APIKeys) (*SingleDom
 	deduped := utils.DedupAPIKeys(apiKeys)
 
 	return &SingleDomainResolver{
-		domain,
-		apiKeys,
-		deduped,
-		sync.Mutex{},
+		domain:         domain,
+		apiKeys:        apiKeys,
+		keyVersion:     0,
+		dedupedAPIKeys: deduped,
+		mu:             sync.Mutex{},
 	}, nil
 }
 
@@ -193,6 +199,14 @@ func (r *SingleDomainResolver) GetAPIKeys() []string {
 	return r.dedupedAPIKeys
 }
 
+// GetAPIKeyVersion get the version of the keys.
+func (r *SingleDomainResolver) GetAPIKeyVersion() int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	return r.keyVersion
+}
+
 // missing returns a list of elements that are in list a, but not in list b.
 // This is inefficient for large lists, but the assumption is that a config
 // will only have a very small number of API keys specified.
@@ -210,10 +224,10 @@ func missing(a []string, b []string) []string {
 }
 
 // GetAPIKeysInfo returns the list of APIKeys and config paths associated with this `DomainResolver`
-func (r *SingleDomainResolver) GetAPIKeysInfo() []utils.APIKeys {
+func (r *SingleDomainResolver) GetAPIKeysInfo() ([]utils.APIKeys, int) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
-	return r.apiKeys
+	return r.apiKeys, r.keyVersion
 }
 
 // SetBaseDomain sets the only destination available for a SingleDomainResolver
@@ -239,6 +253,7 @@ func (r *SingleDomainResolver) UpdateAPIKeys(configPath string, newKeys []utils.
 
 	r.apiKeys = append(newAPIKeys, newKeys...)
 	r.dedupedAPIKeys = utils.DedupAPIKeys(r.apiKeys)
+	r.keyVersion++
 }
 
 // UpdateAPIKey replaces instances of the oldKey with the newKey
@@ -262,6 +277,7 @@ func (r *SingleDomainResolver) UpdateAPIKey(configPath, oldKey, newKey string) {
 	}
 
 	r.dedupedAPIKeys = utils.DedupAPIKeys(r.apiKeys)
+	r.keyVersion++
 }
 
 // GetBearerAuthToken is not implemented for SingleDomainResolver
@@ -278,6 +294,7 @@ type destination struct {
 type MultiDomainResolver struct {
 	baseDomain          string
 	apiKeys             []utils.APIKeys
+	keyVersion          int
 	dedupedAPIKeys      []string
 	overrides           map[string]destination
 	alternateDomainList []string
@@ -297,12 +314,13 @@ func NewMultiDomainResolver(baseDomain string, apiKeys []utils.APIKeys) (*MultiD
 	deduped := utils.DedupAPIKeys(apiKeys)
 
 	return &MultiDomainResolver{
-		baseDomain,
-		apiKeys,
-		deduped,
-		make(map[string]destination),
-		[]string{},
-		sync.Mutex{},
+		baseDomain:          baseDomain,
+		apiKeys:             apiKeys,
+		keyVersion:          0,
+		dedupedAPIKeys:      deduped,
+		overrides:           make(map[string]destination),
+		alternateDomainList: []string{},
+		mu:                  sync.Mutex{},
 	}, nil
 }
 
@@ -313,6 +331,14 @@ func (r *MultiDomainResolver) GetAPIKeys() []string {
 	return r.dedupedAPIKeys
 }
 
+// GetAPIKeyVersion get the version of the keys
+func (r *MultiDomainResolver) GetAPIKeyVersion() int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	return r.keyVersion
+}
+
 // UpdateAPIKeys updates the api keys at the given config path and sets the deduped keys to the new list.
 func (r *MultiDomainResolver) UpdateAPIKeys(configPath string, newKeys []utils.APIKeys) {
 	r.mu.Lock()
@@ -326,13 +352,15 @@ func (r *MultiDomainResolver) UpdateAPIKeys(configPath string, newKeys []utils.A
 
 	r.apiKeys = append(newAPIKeys, newKeys...)
 	r.dedupedAPIKeys = utils.DedupAPIKeys(r.apiKeys)
+
+	r.keyVersion++
 }
 
 // GetAPIKeysInfo returns the list of endpoints associated with this `DomainResolver`
-func (r *MultiDomainResolver) GetAPIKeysInfo() []utils.APIKeys {
+func (r *MultiDomainResolver) GetAPIKeysInfo() ([]utils.APIKeys, int) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
-	return r.apiKeys
+	return r.apiKeys, r.keyVersion
 }
 
 // Resolve returns the destiation for a given request endpoint
@@ -443,9 +471,14 @@ func (r *LocalDomainResolver) GetAPIKeys() []string {
 	return []string{}
 }
 
+// GetAPIKeyVersion get the version of the keys
+func (r *LocalDomainResolver) GetAPIKeyVersion() int {
+	return 0
+}
+
 // GetAPIKeysInfo returns the list of endpoints associated with this `DomainResolver`
-func (r *LocalDomainResolver) GetAPIKeysInfo() []utils.APIKeys {
-	return []utils.APIKeys{}
+func (r *LocalDomainResolver) GetAPIKeysInfo() ([]utils.APIKeys, int) {
+	return []utils.APIKeys{}, 0
 }
 
 // SetBaseDomain sets the base domain to a new value
diff --git a/releasenotes/notes/refreshed-api-key-sanitizer-0bb1588c65bd9b18.yaml b/releasenotes/notes/refreshed-api-key-sanitizer-0bb1588c65bd9b18.yaml
new file mode 100644
index 000000000000..f7a0807c96e8
--- /dev/null
+++ b/releasenotes/notes/refreshed-api-key-sanitizer-0bb1588c65bd9b18.yaml
@@ -0,0 +1,14 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+fixes:
+  - |
+    If a metric transaction can't be sent to the endpoint, this transaction
+    can be serialized to disk. When this occurs, the API key must be sanitized.
+    This ensures that when an API key sourced from a secret is refreshed, the
+    replacer continues to sanitize the new key.


Reviews:
user: login: hush-hush
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-13T09:19:17Z
user: login: vickenty
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T11:10:33Z
user: login: StephenWakely
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T11:26:29Z
user: login: StephenWakely
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T12:23:20Z
user: login: StephenWakely
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T12:34:02Z
user: login: vickenty
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:42:00Z
user: login: vickenty
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:45:54Z
user: login: drichards-87
body: Left a suggestion from Docs and approved the PR.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T15:27:09Z

Review comments:
diff_hunk: @@ -101,6 +116,21 @@ func (s *HTTPTransactionsSerializer) Add(transaction *transaction.HTTPTransactio
 	return nil
 }
 
+// checkAPIKeyUpdate checks if the resolver has updated it's API keys - it does this by maintaining a version
+// number that increments with every update. If they have been updated, we reload the new set of keys and
+// update our replacers.
+func (s *HTTPTransactionsSerializer) checkAPIKeyUpdate() {
+	if s.resolver.GetAPIKeyVersion() != s.currentKeyVersion {
+		// API keys have been updated so we need to rebuild.
+		s.placeholderMutex.Lock()
+		defer s.placeholderMutex.Unlock()
+
+		keys, version := s.resolver.GetAPIKeysInfo()
+		s.apiKeyToPlaceholder, s.placeholderToAPIKey = createReplacers(keys)
+		s.currentKeyVersion = version
+	}
+}
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
user: login: vickenty
body: Would something like this make sense? Reading keys and version in one go?
```suggestion
	keys, version := s.resolver.GetAPIKeysInfo()
	if version != s.currentKeyVersion {
		// API keys have been updated so we need to rebuild.
		s.placeholderMutex.Lock()
		defer s.placeholderMutex.Unlock()	
		s.apiKeyToPlaceholder, s.placeholderToAPIKey = createReplacers(keys)
		s.currentKeyVersion = version
	}
}
```
created_at: 2025-05-13T10:57:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -248,19 +291,38 @@ func toTransactionDestinationProto(destination transaction.Destination) (Transac
 	}
 }
 
-func createReplacers(apiKeys []string) (*strings.Replacer, *strings.Replacer) {
+// createReplacersV1 creates replacers for transactions created with V1 of the serializer.
+// This version sorted the keys prior to serializing. The location in this list is used to
+// unscrub the key.
+func createReplacerV1(apiKeys []string) *strings.Replacer {
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
user: login: vickenty
body: Would it make sense to add telemetry to know when we can drop this method?
created_at: 2025-05-13T11:00:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +115,65 @@ func TestHTTPTransactionSerializerMissingAPIKey(t *testing.T) {
 	r.Equal(1, errorCount)
 }
 
+func TestHTTPTransactionSerializerUpdateAPIKey(t *testing.T) {
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
user: login: vickenty
body: Would it make sense to also test this scenario?

- Transaction is created with the old key. 
- Keys are updated.
- Transaction is being serialized.
created_at: 2025-05-13T11:10:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -101,6 +116,21 @@ func (s *HTTPTransactionsSerializer) Add(transaction *transaction.HTTPTransactio
 	return nil
 }
 
+// checkAPIKeyUpdate checks if the resolver has updated it's API keys - it does this by maintaining a version
+// number that increments with every update. If they have been updated, we reload the new set of keys and
+// update our replacers.
+func (s *HTTPTransactionsSerializer) checkAPIKeyUpdate() {
+	if s.resolver.GetAPIKeyVersion() != s.currentKeyVersion {
+		// API keys have been updated so we need to rebuild.
+		s.placeholderMutex.Lock()
+		defer s.placeholderMutex.Unlock()
+
+		keys, version := s.resolver.GetAPIKeysInfo()
+		s.apiKeyToPlaceholder, s.placeholderToAPIKey = createReplacers(keys)
+		s.currentKeyVersion = version
+	}
+}
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer.go
user: login: StephenWakely
body: My thinking is that because the initial `GetAPIKeyVersion()` call is outside the `s.placeholderMutex.Lock()`, between getting the keys and version another thread could come in, find a newer `APIKeyVersion` and `Keys` and update the replacers. We then acquire `s.placeholderMutex` and then update the replacers to an older version.

To fix, I would have to wrap the whole function in `s.placeholderMutex`, but I don't think that would be desirable since most of the time the version will not be updated.

The way it currently is, I believe, we could potentially have multiple threads call `createReplacers` unnecessarily, but at least each time they would get the latest version. 

Does that sound right to you?
created_at: 2025-05-13T11:26:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +115,65 @@ func TestHTTPTransactionSerializerMissingAPIKey(t *testing.T) {
 	r.Equal(1, errorCount)
 }
 
+func TestHTTPTransactionSerializerUpdateAPIKey(t *testing.T) {
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
user: login: StephenWakely
body: Urg... good thinking.
created_at: 2025-05-13T12:23:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +115,65 @@ func TestHTTPTransactionSerializerMissingAPIKey(t *testing.T) {
 	r.Equal(1, errorCount)
 }
 
+func TestHTTPTransactionSerializerUpdateAPIKey(t *testing.T) {
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
user: login: StephenWakely
body: I actually need to maintain a history of replacers, so we catch any outdated API keys. Performance is a concern, but hopefully API keys are updated rarely enough that it shouldn't be an issue..
created_at: 2025-05-13T12:34:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +115,65 @@ func TestHTTPTransactionSerializerMissingAPIKey(t *testing.T) {
 	r.Equal(1, errorCount)
 }
 
+func TestHTTPTransactionSerializerUpdateAPIKey(t *testing.T) {
path: comp/forwarder/defaultforwarder/internal/retry/http_transactions_serializer_test.go
user: login: vickenty
body: Keeping a history of API keys works, and I could be missing something, but I think it would be better to not rely on string replacement at all: keep the api key identity (index) in HTTPTransaction as it is created, serialize it in a dedicated `HTTPTransactionProto` field and re-set the correct key when reading. Support for API in the query string seems to have been removed from the code, so all that remains for the serializer to do is remove the api key header header.
created_at: 2025-05-13T14:42:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,15 @@
+# Each section from every release note are combined when the
+# CHANGELOG.rst is rendered. So the text needs to be worded so that
+# it does not depend on any information only available in another
+# section. This may mean repeating some details, but each section
+# must be readable independently of the other.
+#
+# Each section note must be formatted as reStructuredText.
+---
+fixes:
+  - |
+    If a metric transaction can't be sent to the endpoint for whatever
+    reason, this transaction can be serialized to disk. When this happens
+    the API key needs to be sanitised. This ensures that when an API key
+    that comes from a secret is refreshed the replacer will continue
+    to sanitise the new API key.
path: releasenotes/notes/refreshed-api-key-sanitizer-0bb1588c65bd9b18.yaml
user: login: drichards-87
body: ```suggestion
    If a metric transaction can't be sent to the endpoint, the transaction can be serialized to disk. When this occurs,
    the API key must be sanitized. This ensures that when an API key sourced from a secret is refreshed, the replacer continues to sanitize the new key.
```
created_at: 2025-05-13T15:26:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: datadog-datadog-prod-us1[bot]
created_at: 2025-05-12T16:39:56Z
body: # Datadog Summary

<b>✅&nbsp;Code Quality</b>&nbsp;&nbsp;&nbsp;&nbsp;<b>✅&nbsp;Code Security</b>&nbsp;&nbsp;&nbsp;&nbsp;<b>✅&nbsp;Dependencies</b>



---
Was this helpful? Give us [feedback](https://dd.enterprise.slack.com/archives/C01FA7FASUR)!

author_association: CONTRIBUTOR
user: login: agent-platform-auto-pr[bot]
created_at: 2025-05-12T17:08:26Z
body: ## Uncompressed package size comparison

Comparison with [ancestor](https://github.com/DataDog/datadog-agent/commit/eb6f39e26b38176dc7a6df76225740721949a03e) `eb6f39e26b38176dc7a6df76225740721949a03e`


<details open>
  <summary> Diff per package </summary>

|package|diff|status|size|ancestor|threshold|
|--|--|--|--|--|--|
|datadog-agent-x86_64-rpm|0.01MB|⚠️|783.25MB|783.24MB|0.50MB|
|datadog-agent-x86_64-suse|0.01MB|⚠️|783.25MB|783.24MB|0.50MB|
|datadog-agent-amd64-deb|0.01MB|⚠️|774.30MB|774.29MB|0.50MB|
|datadog-agent-arm64-deb|0.01MB|⚠️|763.71MB|763.69MB|0.50MB|
|datadog-agent-aarch64-rpm|0.01MB|⚠️|772.63MB|772.62MB|0.50MB|
|datadog-dogstatsd-amd64-deb|0.01MB|⚠️|31.21MB|31.20MB|0.50MB|
|datadog-dogstatsd-x86_64-rpm|0.01MB|⚠️|31.29MB|31.28MB|0.50MB|
|datadog-dogstatsd-x86_64-suse|0.01MB|⚠️|31.29MB|31.28MB|0.50MB|
|datadog-dogstatsd-arm64-deb|0.01MB|⚠️|30.04MB|30.03MB|0.50MB|
|datadog-heroku-agent-amd64-deb|0.01MB|⚠️|377.96MB|377.95MB|0.50MB|
|datadog-iot-agent-arm64-deb|0.01MB|⚠️|57.91MB|57.90MB|0.50MB|
|datadog-iot-agent-aarch64-rpm|0.01MB|⚠️|57.99MB|57.98MB|0.50MB|
|datadog-iot-agent-amd64-deb|0.00MB|✅|60.62MB|60.61MB|0.50MB|
|datadog-iot-agent-x86_64-rpm|0.00MB|✅|60.70MB|60.69MB|0.50MB|
|datadog-iot-agent-x86_64-suse|0.00MB|✅|60.70MB|60.69MB|0.50MB|

</details>

## Decision
⚠️ Warning



author_association: CONTRIBUTOR
user: login: cit-pr-commenter[bot]
created_at: 2025-05-12T17:28:34Z
body: ## Regression Detector

# Regression Detector Results

[Metrics dashboard](https://app.datadoghq.com/dashboard/ykh-ua8-vcu/SMP-Regression-Detector-Metrics?fromUser=true&refresh_mode=paused&tpl_var_run-id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)  
[Target profiles](https://app.datadoghq.com/profiling/explorer?query=env%3Asingle-machine-performance%20service%3Adatadog-agent%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&viz=stream&start=1747204086000&end=1747215486000&paused=true)  
Run ID: 88d5255b-94a0-441b-baad-6a6e0a1f58c6

Baseline: eb6f39e26b38176dc7a6df76225740721949a03e
Comparison: 0d272dd6328a908aa1e6af6d2c539b4915aa2a0b
[Diff](eb6f39e26b38176dc7a6df76225740721949a03e...0d272dd6328a908aa1e6af6d2c539b4915aa2a0b)


## Optimization Goals: ✅ No significant changes detected


<details>
<summary><h2>
Fine details of change detection per experiment
</h2></summary>

| perf | experiment                                   | goal               | Δ mean % | Δ mean % CI    | trials | links                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|------|----------------------------------------------|--------------------|----------|----------------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ➖   | uds_dogstatsd_to_api_cpu                     | % cpu utilization  | +1.51    | [+0.61, +2.40] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api_cpu%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                        |
| ➖   | docker_containers_cpu                        | % cpu utilization  | +1.41    | [-2.44, +5.26] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Adocker_containers_cpu%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                           |
| ➖   | quality_gate_idle_all_features               | memory utilization | +0.69    | [+0.59, +0.79] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle_all_features%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false) |
| ➖   | docker_containers_memory                     | memory utilization | +0.43    | [+0.38, +0.48] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Adocker_containers_memory%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                        |
| ➖   | ddot_metrics                                 | memory utilization | +0.10    | [-0.02, +0.22] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Addot_metrics%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                                    |
| ➖   | file_to_blackhole_0ms_latency_http2          | egress throughput  | +0.08    | [-0.58, +0.73] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http2%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                             |
| ➖   | file_to_blackhole_1000ms_latency             | egress throughput  | +0.03    | [-0.62, +0.67] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                |
| ➖   | file_to_blackhole_300ms_latency              | egress throughput  | +0.01    | [-0.62, +0.65] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_300ms_latency%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                 |
| ➖   | uds_dogstatsd_to_api                         | ingress throughput | +0.01    | [-0.30, +0.32] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                            |
| ➖   | file_to_blackhole_500ms_latency              | egress throughput  | +0.01    | [-0.62, +0.64] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_500ms_latency%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                 |
| ➖   | file_to_blackhole_100ms_latency              | egress throughput  | +0.01    | [-0.64, +0.65] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_100ms_latency%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                 |
| ➖   | tcp_dd_logs_filter_exclude                   | ingress throughput | +0.00    | [-0.02, +0.02] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_dd_logs_filter_exclude%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                      |
| ➖   | file_to_blackhole_1000ms_latency_linear_load | egress throughput  | -0.00    | [-0.23, +0.23] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency_linear_load%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                    |
| ➖   | file_to_blackhole_0ms_latency_http1          | egress throughput  | -0.06    | [-0.71, +0.59] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency_http1%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                             |
| ➖   | file_to_blackhole_0ms_latency                | egress throughput  | -0.08    | [-0.71, +0.54] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                   |
| ➖   | otlp_ingest_logs                             | memory utilization | -0.08    | [-0.21, +0.04] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aotlp_ingest_logs%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                                |
| ➖   | otlp_ingest_metrics                          | memory utilization | -0.18    | [-0.33, -0.03] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aotlp_ingest_metrics%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                             |
| ➖   | ddot_logs                                    | memory utilization | -0.30    | [-0.43, -0.17] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Addot_logs%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                                       |
| ➖   | tcp_syslog_to_blackhole                      | ingress throughput | -0.57    | [-0.64, -0.50] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_syslog_to_blackhole%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                         |
| ➖   | quality_gate_idle                            | memory utilization | -0.73    | [-0.79, -0.67] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)                           |
| ➖   | file_tree                                    | memory utilization | -0.76    | [-0.93, -0.59] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_tree%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                                                       |
| ➖   | quality_gate_logs                            | % cpu utilization  | -0.95    | [-3.68, +1.77] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_logs%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_logs&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)                           |
| ➖   | uds_dogstatsd_20mb_12k_contexts_20_senders   | memory utilization | -1.38    | [-1.42, -1.34] | 1      | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_20mb_12k_contexts_20_senders%20job_id%3A88d5255b-94a0-441b-baad-6a6e0a1f58c6&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1747204086000&to_ts=1747215486000&live=false)                                                                                                                                                                                                                                                                                      |
</details>


<details>
<summary><h2>
Bounds Checks: ✅ Passed
</h2></summary>

| perf | experiment                                   | bounds_check_name  | replicates_passed | links                                                                                                                                                                                                                                                                                            |
|------|----------------------------------------------|--------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ✅   | docker_containers_cpu                        | simple_check_run   | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | docker_containers_memory                     | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | docker_containers_memory                     | simple_check_run   | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency                | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency                | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency_http1          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency_http1          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency_http2          | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_0ms_latency_http2          | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_1000ms_latency             | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_1000ms_latency_linear_load | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_100ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_100ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_300ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_300ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_500ms_latency              | lost_bytes         | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | file_to_blackhole_500ms_latency              | memory_usage       | 10/10             |                                                                                                                                                                                                                                                                                                  |
| ✅   | quality_gate_idle                            | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)              |
| ✅   | quality_gate_idle                            | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)              |
| ✅   | quality_gate_idle_all_features               | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false) |
| ✅   | quality_gate_idle_all_features               | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false) |
| ✅   | quality_gate_logs                            | intake_connections | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_logs&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)              |
| ✅   | quality_gate_logs                            | lost_bytes         | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_logs&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)              |
| ✅   | quality_gate_logs                            | memory_usage       | 10/10             | [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_logs&tpl_var_job_id%5B0%5D=88d5255b-94a0-441b-baad-6a6e0a1f58c6&view=spans&from_ts=1747211286000&to_ts=1747211886000&live=false)              |
</details>


<details>
<summary><h2>
Explanation
</h2></summary>

**Confidence level:** 90.00%
**Effect size tolerance:** |Δ mean %| ≥ 5.00%

Performance changes are noted in the **perf** column of each table:

* ✅ = significantly better comparison variant performance
* ❌ = significantly worse comparison variant performance
* ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".
</details>

## CI Pass/Fail Decision

✅ **Passed.** All Quality Gates passed.

- **quality_gate_logs**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **lost_bytes**: 10/10 replicas passed. Gate passed.
- **quality_gate_logs**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **memory_usage**: 10/10 replicas passed. Gate passed.
- **quality_gate_idle_all_features**, bounds check **intake_connections**: 10/10 replicas passed. Gate passed.
author_association: NONE
user: login: agent-platform-auto-pr[bot]
created_at: 2025-05-12T18:03:39Z
body: ## Static quality checks

✅ Please find below the results from static quality gates


<details>
<summary>Successful checks</summary>

### Info

|Result|Quality gate|On disk size|On disk size limit|On wire size|On wire size limit|
|----|----|----|----|----|----|
|✅|static_quality_gate_agent_deb_amd64|746.98 MiB|778.06 MiB|185.43 MiB|191.06 MiB|
|✅|static_quality_gate_agent_deb_amd64_fips|744.82 MiB|776.09 MiB|184.66 MiB|190.72 MiB|
|✅|static_quality_gate_agent_heroku_amd64|366.43 MiB|434.99 MiB|98.04 MiB|114.34 MiB|
|✅|static_quality_gate_agent_msi|967.04 MiB|978.45 MiB|146.82 MiB|151.65 MiB|
|✅|static_quality_gate_agent_rpm_amd64|746.97 MiB|778.06 MiB|187.11 MiB|193.42 MiB|
|✅|static_quality_gate_agent_rpm_amd64_fips|744.81 MiB|776.06 MiB|186.63 MiB|192.61 MiB|
|✅|static_quality_gate_agent_rpm_arm64|736.84 MiB|768.33 MiB|169.74 MiB|174.71 MiB|
|✅|static_quality_gate_agent_rpm_arm64_fips|734.89 MiB|766.55 MiB|168.67 MiB|173.92 MiB|
|✅|static_quality_gate_agent_suse_amd64|746.97 MiB|778.08 MiB|187.11 MiB|193.42 MiB|
|✅|static_quality_gate_agent_suse_amd64_fips|744.81 MiB|776.11 MiB|186.63 MiB|192.78 MiB|
|✅|static_quality_gate_agent_suse_arm64|736.84 MiB|768.31 MiB|169.74 MiB|174.71 MiB|
|✅|static_quality_gate_agent_suse_arm64_fips|734.89 MiB|766.5 MiB|168.67 MiB|173.92 MiB|
|✅|static_quality_gate_docker_agent_amd64|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_arm64|856.4 MiB|876.0 MiB|272.44 MiB|278.3 MiB|
|✅|static_quality_gate_docker_agent_jmx_amd64|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_jmx_arm64|856.4 MiB|876.63 MiB|272.44 MiB|278.4 MiB|
|✅|static_quality_gate_docker_agent_windows1809|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows1809_core|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows1809_core_jmx|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows1809_jmx|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows2022|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows2022_core|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows2022_core_jmx|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_agent_windows2022_jmx|843.37 MiB|862.5 MiB|286.0 MiB|292.0 MiB|
|✅|static_quality_gate_docker_cluster_agent_amd64|256.76 MiB|263.4 MiB|101.87 MiB|104.07 MiB|
|✅|static_quality_gate_docker_cluster_agent_arm64|272.8 MiB|279.38 MiB|96.86 MiB|98.95 MiB|
|✅|static_quality_gate_docker_cws_instrumentation_amd64|6.66 MiB|7.12 MiB|2.82 MiB|3.29 MiB|
|✅|static_quality_gate_docker_cws_instrumentation_arm64|6.44 MiB|6.92 MiB|2.6 MiB|3.07 MiB|
|✅|static_quality_gate_docker_dogstatsd_amd64|37.89 MiB|46.39 MiB|14.61 MiB|17.78 MiB|
|✅|static_quality_gate_docker_dogstatsd_arm64|36.84 MiB|45.05 MiB|13.73 MiB|16.65 MiB|
|✅|static_quality_gate_dogstatsd_deb_amd64|29.84 MiB|38.4 MiB|7.9 MiB|10.26 MiB|
|✅|static_quality_gate_dogstatsd_deb_arm64|28.72 MiB|36.98 MiB|6.87 MiB|8.96 MiB|
|✅|static_quality_gate_dogstatsd_rpm_amd64|29.84 MiB|38.4 MiB|7.91 MiB|10.27 MiB|
|✅|static_quality_gate_dogstatsd_suse_amd64|29.84 MiB|38.4 MiB|7.91 MiB|10.27 MiB|
|✅|static_quality_gate_iot_agent_deb_amd64|57.89 MiB|58.51 MiB|14.57 MiB|15.02 MiB|
|✅|static_quality_gate_iot_agent_deb_arm64|55.3 MiB|55.94 MiB|12.58 MiB|13.05 MiB|
|✅|static_quality_gate_iot_agent_deb_armhf|54.06 MiB|54.32 MiB|12.6 MiB|13.05 MiB|
|✅|static_quality_gate_iot_agent_rpm_amd64|57.89 MiB|58.51 MiB|14.59 MiB|15.04 MiB|
|✅|static_quality_gate_iot_agent_rpm_arm64|55.31 MiB|55.94 MiB|12.61 MiB|13.07 MiB|
|✅|static_quality_gate_iot_agent_suse_amd64|57.89 MiB|58.51 MiB|14.59 MiB|15.04 MiB|

</details>

author_association: CONTRIBUTOR
user: login: FlorentClarret
created_at: 2025-05-13T05:03:47Z
body: Thanks for the fix! Can we add a changelog for this one?
author_association: MEMBER
user: login: StephenWakely
created_at: 2025-05-13T15:33:21Z
body: /merge
author_association: CONTRIBUTOR
user: login: dd-devflow[bot]
created_at: 2025-05-13T15:33:25Z
body: <!-- "FeedbackWorkflowId":"67217e09-d5a8-46c4-89d7-1ffc7de6c9a5_36" -->
[View all feedbacks in Devflow UI.](https://mosaic.us1.ddbuild.io/devflow/logs/67217e09-d5a8-46c4-89d7-1ffc7de6c9a5_36)

`2025-05-13 15:33:25 UTC` :information_source: Start processing command `/merge`

---
`2025-05-13 15:33:35 UTC` :information_source: **MergeQueue:** waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
**Note:** if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with `/remove` command.

---
`2025-05-13 19:34:03 UTC` :warning: **MergeQueue:** This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

author_association: NONE
user: login: StephenWakely
created_at: 2025-05-14T09:59:59Z
body: /merge
author_association: CONTRIBUTOR
user: login: dd-devflow[bot]
created_at: 2025-05-14T10:00:03Z
body: <!-- "FeedbackWorkflowId":"0e66920f-931e-4c90-aa15-b937d003838e_36" -->
[View all feedbacks in Devflow UI.](https://mosaic.us1.ddbuild.io/devflow/logs/0e66920f-931e-4c90-aa15-b937d003838e_36)

`2025-05-14 10:00:03 UTC` :information_source: Start processing command `/merge`

---
`2025-05-14 10:00:12 UTC` :information_source: **MergeQueue:** pull request added to the queue

The expected merge time in `main` is approximately 1h (p90).

---
`2025-05-14 11:05:49 UTC` :information_source: **MergeQueue:** This merge request was merged

author_association: NONE
user: login: agent-platform-auto-pr[bot]
created_at: 2025-05-14T11:06:39Z
body: The backport to `7.66.x` failed:
```
The process '/usr/bin/git' failed with exit code 1
```
To backport manually, run these commands in your terminal:
```bash
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-7.66.x 7.66.x
# Navigate to the new working tree
cd .worktrees/backport-7.66.x
# Create a new branch
git switch --create backport-36879-to-7.66.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 02e6464d6722d6072cb41148c00fd1546e0ba561
# Push it to GitHub
git push --set-upstream origin backport-36879-to-7.66.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-7.66.x
```
Then, create a pull request where the `base` branch is `7.66.x` and the `compare`/`head` branch is `backport-36879-to-7.66.x`.
author_association: CONTRIBUTOR