Repository: elastic/elastic-agent
Pull request #7635: Fix upgrade for same versions
Author: login: michalpristas
Labels: name: bug
description: Something isn't working
name: Team:Elastic-Agent-Control-Plane
description: Label for the Agent Control Plane team
name: skip-changelog
name: backport-active-9
description: Automated backport with mergify to all the active 9.[0-9]+ branches
name: backport-active-8
description: Automated backport with mergify to all the active 8.[0-9]+ branches
Created: 2025-03-31T12:46:00Z | Merged: 2025-04-08T07:14:26Z

## What does this PR do?

What we're seeing here is upgrade marked uncomplete in 2 scenarios.
First one is when upgrade action for same version arrives for the first time.
We go through logic up until the steps and then we set nil to details. What we're not doing is Acking the action though. 
This can lead to having upgraded at property still populated and hence upgrade showing as ongoing.

Second case is when we trigger the action second time, maybe in a hope to fix the label/state.
In this case we have already first one in a local state and we detect that this action is upgrade to same version, althought it's not the same action. We append local queue of actions and do nothing. Again there is a need to ack because upgraded at property is set.

Logic determining Updating label looks like this

```javacript
function isAgentUpgrading(agent: Agent) {
  if (agent.upgrade_details) {
    return agent.upgrade_details.state !== 'UPG_FAILED';
  }
  return agent.upgrade_started_at && !agent.upgraded_at;
}
```

## Why is it important?

Leads to weird state being displayed in a UI

## Checklist

- [x] I have read and understood the [pull request guidelines](https://github.com/elastic/elastic-agent/blob/main/CONTRIBUTING.md#pull-request-guidelines) of this project.
- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have made corresponding change to the default configuration files
- [ x I have added tests that prove my fix is effective or that my feature works
- [ ] I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent#changelog)
- [x] I have added an integration test or an E2E test

Fixes: #6186 


Diff:
diff --git a/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go b/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
index ad7c1bc711c..1520677f06b 100644
--- a/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
+++ b/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
@@ -92,9 +92,7 @@ func (h *Upgrade) Handle(ctx context.Context, a fleetapi.Action, ack acker.Acker
 // User is responsible for obtaining and releasing bkgMutex lock
 func (h *Upgrade) ackActions(ctx context.Context, ack acker.Acker) {
 	for _, a := range h.bkgActions {
-		if err := ack.Ack(ctx, a); err != nil {
-			h.log.Errorf("ack of failed upgrade failed: %v", err)
-		}
+		h.ackAction(ctx, ack, a, false)
 	}
 	h.bkgActions = nil
 	if err := ack.Commit(ctx); err != nil {
@@ -102,6 +100,20 @@ func (h *Upgrade) ackActions(ctx context.Context, ack acker.Acker) {
 	}
 }
 
+// ackActions Acks all the actions in bkgActions, and deletes entries from bkgActions.
+// User is responsible for obtaining and releasing bkgMutex lock
+func (h *Upgrade) ackAction(ctx context.Context, ack acker.Acker, action fleetapi.Action, commit bool) {
+	if err := ack.Ack(ctx, action); err != nil {
+		h.log.Errorf("ack of failed upgrade failed: %v", err)
+	}
+
+	if commit {
+		if err := ack.Commit(ctx); err != nil {
+			h.log.Errorf("commit of ack for failed upgrade failed: %v", err)
+		}
+	}
+}
+
 // getAsyncContext returns a cancelContext and whether or not to run the upgrade
 func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, ack acker.Acker) (context.Context, bool) {
 	h.bkgMutex.Lock()
@@ -125,11 +137,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
+		// not the same action this one needs to be acked
+		go func() {
+			// kick it off and don't block, lock to prevent race with ackActions from finished upgrade
+			h.bkgMutex.Lock()
+			defer h.bkgMutex.Unlock()
+
+			h.ackAction(ctx, ack, upgradeAction, true)
+		}()
 		return nil, false
 	}
 
diff --git a/internal/pkg/agent/application/actions/handlers/handler_action_upgrade_test.go b/internal/pkg/agent/application/actions/handlers/handler_action_upgrade_test.go
index 1cac13e00f1..10ac83f536e 100644
--- a/internal/pkg/agent/application/actions/handlers/handler_action_upgrade_test.go
+++ b/internal/pkg/agent/application/actions/handlers/handler_action_upgrade_test.go
@@ -11,6 +11,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/mock"
 	"github.com/stretchr/testify/require"
 
 	"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator"
@@ -68,7 +69,11 @@ func (u *mockUpgradeManager) Upgrade(
 		pgpBytes...)
 }
 
-func (u *mockUpgradeManager) Ack(ctx context.Context, acker acker.Acker) error {
+func (u *mockUpgradeManager) Ack(_ context.Context, _ acker.Acker) error {
+	return nil
+}
+
+func (u *mockUpgradeManager) AckAction(_ context.Context, _ acker.Acker, _ fleetapi.Action) error {
 	return nil
 }
 
@@ -110,7 +115,7 @@ func TestUpgradeHandler(t *testing.T) {
 				return nil, nil
 			},
 		},
-		nil, nil, nil, nil, nil, false, nil)
+		nil, nil, nil, nil, nil, false, nil, nil)
 	//nolint:errcheck // We don't need the termination state of the Coordinator
 	go c.Run(ctx)
 
@@ -169,7 +174,7 @@ func TestUpgradeHandlerSameVersion(t *testing.T) {
 				return nil, err
 			},
 		},
-		nil, nil, nil, nil, nil, false, nil)
+		nil, nil, nil, nil, nil, false, nil, nil)
 	//nolint:errcheck // We don't need the termination state of the Coordinator
 	go c.Run(ctx)
 
@@ -190,6 +195,98 @@ func TestUpgradeHandlerSameVersion(t *testing.T) {
 	}
 }
 
+func TestDuplicateActionsHandled(t *testing.T) {
+	// Create a cancellable context that will shut down the coordinator after
+	// the test.
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	log, _ := logger.New("", false)
+	upgradeCalledChan := make(chan string)
+
+	agentInfo := &info.AgentInfo{}
+	acker := &fakeAcker{}
+
+	// Create and start the Coordinator
+	c := coordinator.New(
+		log,
+		configuration.DefaultConfiguration(),
+		logger.DefaultLogLevel,
+		agentInfo,
+		component.RuntimeSpecs{},
+		nil,
+		&mockUpgradeManager{
+			UpgradeFn: func(
+				ctx context.Context,
+				version string,
+				sourceURI string,
+				action *fleetapi.ActionUpgrade,
+				details *details.Details,
+				skipVerifyOverride bool,
+				skipDefaultPgp bool,
+				pgpBytes ...string) (reexec.ShutdownCallbackFn, error) {
+
+				defer func() {
+					upgradeCalledChan <- action.ActionID
+				}()
+
+				return nil, nil
+			},
+		},
+		nil, nil, nil, nil, nil, false, nil, acker)
+	//nolint:errcheck // We don't need the termination state of the Coordinator
+	go c.Run(ctx)
+
+	u := NewUpgrade(log, c)
+	a1 := fleetapi.ActionUpgrade{
+		ActionID: "action-8.5-1",
+		Data: fleetapi.ActionUpgradeData{
+			Version: "8.5.0", SourceURI: "http://localhost",
+		},
+	}
+	a2 := fleetapi.ActionUpgrade{
+		ActionID: "action-8.5-2",
+		Data: fleetapi.ActionUpgradeData{
+			Version: "8.5.0", SourceURI: "http://localhost",
+		},
+	}
+
+	checkMsg := func(c <-chan string, expected, errMsg string) error {
+		t.Helper()
+		// Make sure this test does not dead lock or wait for too long
+		// For some reason < 1s sometimes makes the test fail.
+		select {
+		case <-time.Tick(1500 * time.Millisecond):
+			return errors.New("timed out waiting for Upgrade to return")
+		case msg := <-c:
+			require.Equal(t, expected, msg, errMsg)
+		}
+
+		return nil
+	}
+
+	acker.On("Ack", mock.Anything, mock.Anything).Return(nil)
+	acker.On("Commit", mock.Anything).Return(nil)
+
+	t.Log("First upgrade action should be processed")
+	require.NoError(t, u.Handle(ctx, &a1, acker))
+	require.Nil(t, checkMsg(upgradeCalledChan, a1.ActionID, "action was not processed"))
+	c.ClearOverrideState() // it's upgrading, normally we would restart
+
+	t.Log("Action with different ID but same version should not be propagated to upgrader but acked")
+	require.NoError(t, u.Handle(ctx, &a2, acker))
+	require.NotNil(t, checkMsg(upgradeCalledChan, a2.ActionID, "action was not processed"))
+	acker.AssertCalled(t, "Ack", ctx, &a2)
+	acker.AssertCalled(t, "Commit", ctx)
+
+	c.ClearOverrideState() // it's upgrading, normally we would restart
+
+	t.Log("Resending action with same ID should be skipped")
+	require.NoError(t, u.Handle(ctx, &a1, acker))
+	require.NotNil(t, checkMsg(upgradeCalledChan, a1.ActionID, "action was not processed"))
+	acker.AssertNotCalled(t, "Ack", ctx, &a1)
+}
+
 func TestUpgradeHandlerNewVersion(t *testing.T) {
 	// Create a cancellable context that will shut down the coordinator after
 	// the test.
@@ -230,15 +327,23 @@ func TestUpgradeHandlerNewVersion(t *testing.T) {
 				return nil, nil
 			},
 		},
-		nil, nil, nil, nil, nil, false, nil)
+		nil, nil, nil, nil, nil, false, nil, nil)
 	//nolint:errcheck // We don't need the termination state of the Coordinator
 	go c.Run(ctx)
 
 	u := NewUpgrade(log, c)
-	a1 := fleetapi.ActionUpgrade{Data: fleetapi.ActionUpgradeData{
-		Version: "8.2.0", SourceURI: "http://localhost"}}
-	a2 := fleetapi.ActionUpgrade{Data: fleetapi.ActionUpgradeData{
-		Version: "8.5.0", SourceURI: "http://localhost"}}
+	a1 := fleetapi.ActionUpgrade{
+		ActionID: "action-8.2",
+		Data: fleetapi.ActionUpgradeData{
+			Version: "8.2.0", SourceURI: "http://localhost",
+		},
+	}
+	a2 := fleetapi.ActionUpgrade{
+		ActionID: "action-8.5",
+		Data: fleetapi.ActionUpgradeData{
+			Version: "8.5.0", SourceURI: "http://localhost",
+		},
+	}
 	ack := noopacker.New()
 
 	checkMsg := func(c <-chan string, expected, errMsg string) {
@@ -262,3 +367,17 @@ func TestUpgradeHandlerNewVersion(t *testing.T) {
 	require.NoError(t, err2)
 	checkMsg(upgradeCalledChan, "8.5.0", "second call to Upgrade must be with version 8.5.0")
 }
+
+type fakeAcker struct {
+	mock.Mock
+}
+
+func (f *fakeAcker) Ack(ctx context.Context, action fleetapi.Action) error {
+	args := f.Called(ctx, action)
+	return args.Error(0)
+}
+
+func (f *fakeAcker) Commit(ctx context.Context) error {
+	args := f.Called(ctx)
+	return args.Error(0)
+}
diff --git a/internal/pkg/agent/application/application.go b/internal/pkg/agent/application/application.go
index fae181e034a..8300b49fe3b 100644
--- a/internal/pkg/agent/application/application.go
+++ b/internal/pkg/agent/application/application.go
@@ -21,10 +21,16 @@ import (
 	"github.com/elastic/elastic-agent/internal/pkg/agent/configuration"
 	"github.com/elastic/elastic-agent/internal/pkg/agent/errors"
 	"github.com/elastic/elastic-agent/internal/pkg/agent/storage"
+	stateStore "github.com/elastic/elastic-agent/internal/pkg/agent/storage/store"
 	"github.com/elastic/elastic-agent/internal/pkg/capabilities"
 	"github.com/elastic/elastic-agent/internal/pkg/composable"
 	"github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes"
 	"github.com/elastic/elastic-agent/internal/pkg/config"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/fleet"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/lazy"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/retrier"
+	fleetclient "github.com/elastic/elastic-agent/internal/pkg/fleetapi/client"
 	otelmanager "github.com/elastic/elastic-agent/internal/pkg/otel/manager"
 	"github.com/elastic/elastic-agent/internal/pkg/release"
 	"github.com/elastic/elastic-agent/pkg/component"
@@ -136,6 +142,7 @@ func New(
 	var composableManaged bool
 	var isManaged bool
 
+	var actionAcker acker.Acker
 	if testingMode {
 		log.Info("Elastic Agent has been started in testing mode and is managed through the control protocol")
 
@@ -182,8 +189,29 @@ func New(
 				InjectProxyEndpointModifier(),
 			)
 
+			client, err := fleetclient.NewAuthWithConfig(log, cfg.Fleet.AccessAPIKey, cfg.Fleet.Client)
+			if err != nil {
+				return nil, nil, nil, errors.New(err,
+					"fail to create API client",
+					errors.TypeNetwork,
+					errors.M(errors.MetaKeyURI, cfg.Fleet.Client.Host))
+			}
+			stateStorage, err := stateStore.NewStateStoreWithMigration(ctx, log, paths.AgentActionStoreFile(), paths.AgentStateStoreFile())
+			if err != nil {
+				return nil, nil, nil, errors.New(err, fmt.Sprintf("fail to read state store '%s'", paths.AgentStateStoreFile()))
+			}
+
+			fleetAcker, err := fleet.NewAcker(log, agentInfo, client)
+			if err != nil {
+				return nil, nil, nil, fmt.Errorf("failed to create acker: %w", err)
+			}
+
+			retrier := retrier.New(fleetAcker, log)
+			batchedAcker := lazy.NewAcker(fleetAcker, log, lazy.WithRetrier(retrier))
+			actionAcker = stateStore.NewStateStoreActionAcker(batchedAcker, stateStorage)
+
 			// TODO: stop using global state
-			managed, err = newManagedConfigManager(ctx, log, agentInfo, cfg, store, runtime, fleetInitTimeout, paths.Top(), upgrader)
+			managed, err = newManagedConfigManager(ctx, log, agentInfo, cfg, store, runtime, fleetInitTimeout, paths.Top(), client, fleetAcker, actionAcker, retrier, stateStorage, upgrader)
 			if err != nil {
 				return nil, nil, nil, err
 			}
@@ -197,7 +225,7 @@ func New(
 	}
 
 	otelManager := otelmanager.NewOTelManager(log.Named("otel_manager"))
-	coord := coordinator.New(log, cfg, logLevel, agentInfo, specs, reexec, upgrader, runtime, configMgr, varsManager, caps, monitor, isManaged, otelManager, compModifiers...)
+	coord := coordinator.New(log, cfg, logLevel, agentInfo, specs, reexec, upgrader, runtime, configMgr, varsManager, caps, monitor, isManaged, otelManager, actionAcker, compModifiers...)
 	if managed != nil {
 		// the coordinator requires the config manager as well as in managed-mode the config manager requires the
 		// coordinator, so it must be set here once the coordinator is created
diff --git a/internal/pkg/agent/application/coordinator/coordinator.go b/internal/pkg/agent/application/coordinator/coordinator.go
index 88c900b6f64..bb1ae2ec643 100644
--- a/internal/pkg/agent/application/coordinator/coordinator.go
+++ b/internal/pkg/agent/application/coordinator/coordinator.go
@@ -75,6 +75,9 @@ type UpgradeManager interface {
 	// Ack is used on startup to check if the agent has upgraded and needs to send an ack for the action
 	Ack(ctx context.Context, acker acker.Acker) error
 
+	// AckAction is used to ack not persisted action.
+	AckAction(ctx context.Context, acker acker.Acker, action fleetapi.Action) error
+
 	// MarkerWatcher returns a watcher for the upgrade marker.
 	MarkerWatcher() upgrade.MarkerWatcher
 }
@@ -204,8 +207,9 @@ type Coordinator struct {
 	agentInfo info.Agent
 	isManaged bool
 
-	cfg   *configuration.Configuration
-	specs component.RuntimeSpecs
+	cfg        *configuration.Configuration
+	specs      component.RuntimeSpecs
+	fleetAcker acker.Acker
 
 	reexecMgr  ReExecManager
 	upgradeMgr UpgradeManager
@@ -372,7 +376,7 @@ type UpdateComponentChange struct {
 }
 
 // New creates a new coordinator.
-func New(logger *logger.Logger, cfg *configuration.Configuration, logLevel logp.Level, agentInfo info.Agent, specs component.RuntimeSpecs, reexecMgr ReExecManager, upgradeMgr UpgradeManager, runtimeMgr RuntimeManager, configMgr ConfigManager, varsMgr VarsManager, caps capabilities.Capabilities, monitorMgr MonitorManager, isManaged bool, otelMgr OTelManager, modifiers ...ComponentsModifier) *Coordinator {
+func New(logger *logger.Logger, cfg *configuration.Configuration, logLevel logp.Level, agentInfo info.Agent, specs component.RuntimeSpecs, reexecMgr ReExecManager, upgradeMgr UpgradeManager, runtimeMgr RuntimeManager, configMgr ConfigManager, varsMgr VarsManager, caps capabilities.Capabilities, monitorMgr MonitorManager, isManaged bool, otelMgr OTelManager, fleetAcker acker.Acker, modifiers ...ComponentsModifier) *Coordinator {
 	var fleetState cproto.State
 	var fleetMessage string
 	if !isManaged {
@@ -426,6 +430,8 @@ func New(logger *logger.Logger, cfg *configuration.Configuration, logLevel logp.
 		heartbeatChan:              make(chan struct{}),
 		componentPIDTicker:         time.NewTicker(time.Second * 30),
 		componentPidRequiresUpdate: &atomic.Bool{},
+
+		fleetAcker: fleetAcker,
 	}
 	// Setup communication channels for any non-nil components. This pattern
 	// lets us transparently accept nil managers / simulated events during
@@ -585,7 +591,7 @@ func (c *Coordinator) Upgrade(ctx context.Context, version string, sourceURI str
 		if errors.Is(err, upgrade.ErrUpgradeSameVersion) {
 			// Set upgrade state to completed so update no longer shows in-progress.
 			det.SetState(details.StateCompleted)
-			return nil
+			return c.upgradeMgr.AckAction(ctx, c.fleetAcker, action)
 		}
 		det.Fail(err)
 		return err
diff --git a/internal/pkg/agent/application/coordinator/coordinator_test.go b/internal/pkg/agent/application/coordinator/coordinator_test.go
index 7e6a547b265..3ea4054097c 100644
--- a/internal/pkg/agent/application/coordinator/coordinator_test.go
+++ b/internal/pkg/agent/application/coordinator/coordinator_test.go
@@ -16,6 +16,8 @@ import (
 	"testing"
 	"time"
 
+	"go.uber.org/zap/zapcore"
+	"go.uber.org/zap/zaptest/observer"
 	"gopkg.in/yaml.v3"
 
 	"github.com/elastic/elastic-agent-libs/mapstr"
@@ -24,6 +26,7 @@ import (
 
 	"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/status"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
 	"github.com/stretchr/testify/require"
 	"go.elastic.co/apm/v2/apmtest"
 	"go.opentelemetry.io/collector/confmap"
@@ -109,7 +112,14 @@ func waitForState(
 
 func TestComponentUpdateDiff(t *testing.T) {
 
-	err := logp.DevelopmentSetup(logp.ToObserverOutput())
+	observedCore, observedLogs := observer.New(zapcore.DebugLevel)
+	err := logp.ConfigureWithOutputs(logp.Config{
+		Level:      logp.DebugLevel,
+		ToStderr:   false,
+		ToSyslog:   false,
+		ToFiles:    false,
+		ToEventLog: false,
+	}, observedCore)
 	require.NoError(t, err)
 
 	cases := []struct {
@@ -315,15 +325,13 @@ func TestComponentUpdateDiff(t *testing.T) {
 			}
 			testCoord.checkAndLogUpdate(testcase.old)
 
-			obsLogs := logp.ObserverLogs().TakeAll()
+			obsLogs := observedLogs.All()
 			last := obsLogs[len(obsLogs)-1]
 
 			// extract the structured data from the log message
 			testcase.logtest(t, last.Context[0].Interface.(UpdateStats))
 		})
-
 	}
-
 }
 
 func mustNewStruct(t *testing.T, v map[string]interface{}) *structpb.Struct {
@@ -493,6 +501,47 @@ func TestCoordinator_State_ConfigError_NotManaged(t *testing.T) {
 	require.NoError(t, err)
 }
 
+func TestUpgradeSameErrorAcked(t *testing.T) {
+	coordCh := make(chan error)
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	upgradeManager := &fakeUpgradeManager{
+		upgradeable: true,
+		upgradeErr:  upgrade.ErrUpgradeSameVersion,
+	}
+
+	acker := &fakeActionAcker{}
+	coord, _, _ := createCoordinator(t, ctx,
+		ManagedCoordinator(true),
+		WithUpgradeManager(upgradeManager),
+		WithActionAcker(acker))
+
+	var coordWait sync.WaitGroup
+	coordWait.Add(1)
+	go func() {
+		coordWait.Done()
+		err := coord.Run(ctx)
+		if errors.Is(err, context.Canceled) {
+			// allowed error
+			err = nil
+		}
+		coordCh <- err
+	}()
+
+	coordWait.Wait()
+
+	action := fleetapi.NewAction(fleetapi.ActionTypeUpgrade)
+	actionUpgrade, ok := action.(*fleetapi.ActionUpgrade)
+	require.True(t, ok)
+
+	acker.On("Ack", mock.Anything, actionUpgrade).Return(nil)
+
+	require.NoError(t, coord.Upgrade(t.Context(), "9.0", "http://localhost", actionUpgrade, true, true))
+
+	acker.AssertCalled(t, "Ack", mock.Anything, actionUpgrade)
+}
+
 func TestCoordinator_State_ConfigError_Managed(t *testing.T) {
 	coordCh := make(chan error)
 	ctx, cancel := context.WithCancel(context.Background())
@@ -965,6 +1014,7 @@ type createCoordinatorOpts struct {
 	managed        bool
 	upgradeManager UpgradeManager
 	compInputSpec  component.InputSpec
+	acker          acker.Acker
 }
 
 type CoordinatorOpt func(o *createCoordinatorOpts)
@@ -981,6 +1031,12 @@ func WithUpgradeManager(upgradeManager UpgradeManager) CoordinatorOpt {
 	}
 }
 
+func WithActionAcker(acker acker.Acker) CoordinatorOpt {
+	return func(o *createCoordinatorOpts) {
+		o.acker = acker
+	}
+}
+
 func WithComponentInputSpec(spec component.InputSpec) CoordinatorOpt {
 	return func(o *createCoordinatorOpts) {
 		o.compInputSpec = spec
@@ -1035,7 +1091,12 @@ func createCoordinator(t testing.TB, ctx context.Context, opts ...CoordinatorOpt
 		upgradeManager = &fakeUpgradeManager{}
 	}
 
-	coord := New(l, nil, logp.DebugLevel, ai, specs, &fakeReExecManager{}, upgradeManager, rm, cfgMgr, varsMgr, caps, monitoringMgr, o.managed, otelMgr)
+	acker := o.acker
+	if acker == nil {
+		acker = &fakeActionAcker{}
+	}
+
+	coord := New(l, nil, logp.DebugLevel, ai, specs, &fakeReExecManager{}, upgradeManager, rm, cfgMgr, varsMgr, caps, monitoringMgr, o.managed, otelMgr, acker)
 	return coord, cfgMgr, varsMgr
 }
 
@@ -1071,6 +1132,22 @@ func (f *fakeReExecManager) ReExec(callback reexec.ShutdownCallbackFn, _ ...stri
 	}
 }
 
+var _ acker.Acker = &fakeActionAcker{}
+
+type fakeActionAcker struct {
+	mock.Mock
+}
+
+func (f *fakeActionAcker) Ack(ctx context.Context, action fleetapi.Action) error {
+	args := f.Called(ctx, action)
+	return args.Error(0)
+}
+
+func (f *fakeActionAcker) Commit(ctx context.Context) error {
+	args := f.Called(ctx)
+	return args.Error(0)
+}
+
 type fakeUpgradeManager struct {
 	upgradeable   bool
 	upgradeErr    error // An error to return when Upgrade is called
@@ -1094,6 +1171,16 @@ func (f *fakeUpgradeManager) Upgrade(ctx context.Context, version string, source
 }
 
 func (f *fakeUpgradeManager) Ack(ctx context.Context, acker acker.Acker) error {
+	if acker != nil {
+		return acker.Ack(ctx, fleetapi.NewAction(fleetapi.ActionTypeUnknown))
+	}
+	return nil
+}
+
+func (f *fakeUpgradeManager) AckAction(ctx context.Context, acker acker.Acker, action fleetapi.Action) error {
+	if acker != nil {
+		return acker.Ack(ctx, action)
+	}
 	return nil
 }
 
diff --git a/internal/pkg/agent/application/managed_mode.go b/internal/pkg/agent/application/managed_mode.go
index 59966183f99..db850aa6503 100644
--- a/internal/pkg/agent/application/managed_mode.go
+++ b/internal/pkg/agent/application/managed_mode.go
@@ -25,9 +25,7 @@ import (
 	"github.com/elastic/elastic-agent/internal/pkg/fleetapi"
 	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker"
 	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/fleet"
-	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/lazy"
 	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/retrier"
-	fleetclient "github.com/elastic/elastic-agent/internal/pkg/fleetapi/client"
 	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/uploader"
 	"github.com/elastic/elastic-agent/internal/pkg/queue"
 	"github.com/elastic/elastic-agent/internal/pkg/remote"
@@ -52,6 +50,9 @@ type managedConfigManager struct {
 	coord                *coordinator.Coordinator
 	fleetInitTimeout     time.Duration
 	initialClientSetters []actions.ClientSetter
+	fleetAcker           *fleet.Acker
+	actionAcker          acker.Acker
+	retrier              *retrier.Retrier
 
 	ch    chan coordinator.ConfigChange
 	errCh chan error
@@ -66,22 +67,13 @@ func newManagedConfigManager(
 	runtime *runtime.Manager,
 	fleetInitTimeout time.Duration,
 	topPath string,
+	client *remote.Client,
+	fleetAcker *fleet.Acker,
+	actionAcker acker.Acker,
+	retrier *retrier.Retrier,
+	stateStore *store.StateStore,
 	clientSetters ...actions.ClientSetter,
 ) (*managedConfigManager, error) {
-	client, err := fleetclient.NewAuthWithConfig(log, cfg.Fleet.AccessAPIKey, cfg.Fleet.Client)
-	if err != nil {
-		return nil, errors.New(err,
-			"fail to create API client",
-			errors.TypeNetwork,
-			errors.M(errors.MetaKeyURI, cfg.Fleet.Client.Host))
-	}
-
-	// Create the state store that will persist the last good policy change on disk.
-	stateStore, err := store.NewStateStoreWithMigration(ctx, log, paths.AgentActionStoreFile(), paths.AgentStateStoreFile())
-	if err != nil {
-		return nil, errors.New(err, fmt.Sprintf("fail to read state store '%s'", paths.AgentStateStoreFile()))
-	}
-
 	actionQueue, err := queue.NewActionQueue(stateStore.Queue(), stateStore)
 	if err != nil {
 		return nil, fmt.Errorf("unable to initialize action queue: %w", err)
@@ -106,6 +98,9 @@ func newManagedConfigManager(
 		ch:                   make(chan coordinator.ConfigChange),
 		errCh:                make(chan error),
 		initialClientSetters: clientSetters,
+		fleetAcker:           fleetAcker,
+		actionAcker:          actionAcker,
+		retrier:              retrier,
 	}, nil
 }
 
@@ -134,15 +129,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 	policyChanger := m.initDispatcher(gatewayCancel)
 
 	// Create ackers to enqueue/retry failed acks
-	ack, err := fleet.NewAcker(m.log, m.agentInfo, m.client)
-	if err != nil {
-		return fmt.Errorf("failed to create acker: %w", err)
-	}
-	retrier := retrier.New(ack, m.log)
-	batchedAcker := lazy.NewAcker(ack, m.log, lazy.WithRetrier(retrier))
-	actionAcker := store.NewStateStoreActionAcker(batchedAcker, m.stateStore)
-
-	if err := m.coord.AckUpgrade(ctx, actionAcker); err != nil {
+	if err := m.coord.AckUpgrade(ctx, m.actionAcker); err != nil {
 		m.log.Warnf("Failed to ack upgrade: %v", err)
 	}
 
@@ -154,7 +141,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 		<-retrierRun
 	}()
 	go func() {
-		retrier.Run(retrierCtx)
+		m.retrier.Run(retrierCtx)
 		close(retrierRun)
 	}()
 
@@ -165,7 +152,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 		// persisted action on disk we should be able to ask Fleet to get the latest configuration.
 		// But at the moment this is not possible because the policy change was acked.
 		m.log.Info("restoring current policy from disk")
-		m.dispatcher.Dispatch(ctx, m.coord.SetUpgradeDetails, actionAcker, action)
+		m.dispatcher.Dispatch(ctx, m.coord.SetUpgradeDetails, m.actionAcker, action)
 		stateRestored = true
 	}
 
@@ -173,12 +160,12 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 	// the Fleet Server is running before the Fleet gateway is started.
 	if m.cfg.Fleet.Server != nil {
 		if stateRestored {
-			err = m.waitForFleetServer(ctx)
+			err := m.waitForFleetServer(ctx)
 			if err != nil {
 				return fmt.Errorf("failed to initialize Fleet Server: %w", err)
 			}
 		} else {
-			err = m.initFleetServer(ctx, m.cfg.Fleet.Server)
+			err := m.initFleetServer(ctx, m.cfg.Fleet.Server)
 			if err != nil {
 				return fmt.Errorf("failed to initialize Fleet Server: %w", err)
 			}
@@ -189,7 +176,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 		m.log,
 		m.agentInfo,
 		m.client,
-		actionAcker,
+		m.actionAcker,
 		m.coord.State,
 		m.stateStore,
 	)
@@ -200,7 +187,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 	// Not running a Fleet Server so the gateway and acker can be changed based on the configuration change.
 	if m.cfg.Fleet.Server == nil {
 		policyChanger.AddSetter(gateway)
-		policyChanger.AddSetter(ack)
+		policyChanger.AddSetter(m.fleetAcker)
 
 		for _, cs := range m.initialClientSetters {
 			policyChanger.AddSetter(cs)
@@ -231,7 +218,7 @@ func (m *managedConfigManager) Run(ctx context.Context) error {
 		return gateway.Run(ctx)
 	})
 
-	go runDispatcher(ctx, m.dispatcher, gateway, m.coord.SetUpgradeDetails, actionAcker, dispatchFlushInterval)
+	go runDispatcher(ctx, m.dispatcher, gateway, m.coord.SetUpgradeDetails, m.actionAcker, dispatchFlushInterval)
 
 	<-ctx.Done()
 	return gatewayRunner.Err()
diff --git a/internal/pkg/agent/application/upgrade/upgrade.go b/internal/pkg/agent/application/upgrade/upgrade.go
index d0dcfa032fc..a83549db9c2 100644
--- a/internal/pkg/agent/application/upgrade/upgrade.go
+++ b/internal/pkg/agent/application/upgrade/upgrade.go
@@ -433,11 +433,7 @@ func (u *Upgrader) Ack(ctx context.Context, acker acker.Acker) error {
 	// Should handle gracefully
 	// https://github.com/elastic/elastic-agent/issues/1788
 	if marker.Action != nil {
-		if err := acker.Ack(ctx, marker.Action); err != nil {
-			return err
-		}
-
-		if err := acker.Commit(ctx); err != nil {
+		if err := u.AckAction(ctx, acker, marker.Action); err != nil {
 			return err
 		}
 	}
@@ -447,6 +443,22 @@ func (u *Upgrader) Ack(ctx context.Context, acker acker.Acker) error {
 	return SaveMarker(marker, false)
 }
 
+func (u *Upgrader) AckAction(ctx context.Context, acker acker.Acker, action fleetapi.Action) error {
+	if acker == nil {
+		return nil
+	}
+
+	if err := acker.Ack(ctx, action); err != nil {
+		return err
+	}
+
+	if err := acker.Commit(ctx); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 func (u *Upgrader) MarkerWatcher() MarkerWatcher {
 	return u.markerWatcher
 }
diff --git a/internal/pkg/agent/application/upgrade/upgrade_test.go b/internal/pkg/agent/application/upgrade/upgrade_test.go
index e6404454e49..17d19252f6e 100644
--- a/internal/pkg/agent/application/upgrade/upgrade_test.go
+++ b/internal/pkg/agent/application/upgrade/upgrade_test.go
@@ -17,6 +17,7 @@ import (
 
 	"github.com/gofrs/flock"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
 	"github.com/stretchr/testify/require"
 	"gopkg.in/yaml.v2"
 
@@ -27,6 +28,8 @@ import (
 	"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/details"
 	"github.com/elastic/elastic-agent/internal/pkg/agent/errors"
 	"github.com/elastic/elastic-agent/internal/pkg/config"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi"
+	"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker"
 	"github.com/elastic/elastic-agent/internal/pkg/release"
 	v1 "github.com/elastic/elastic-agent/pkg/api/v1"
 	"github.com/elastic/elastic-agent/pkg/control/v2/client"
@@ -278,6 +281,52 @@ func TestUpgraderReload(t *testing.T) {
 	assert.Equal(t, &want, u.settings)
 }
 
+func TestUpgraderAckAction(t *testing.T) {
+	log, _ := loggertest.New("")
+	u := Upgrader{
+		log:      log,
+		settings: artifact.DefaultConfig(),
+	}
+
+	action := fleetapi.NewAction(fleetapi.ActionTypeUpgrade)
+	t.Run("AckAction without acker", func(t *testing.T) {
+		require.Nil(t, u.AckAction(t.Context(), nil, action))
+	})
+	t.Run("AckAction with acker", func(t *testing.T) {
+		acker := &fakeAcker{}
+		acker.On("Ack", mock.Anything, action).Return(nil)
+		acker.On("Commit", mock.Anything).Return(nil)
+
+		require.Nil(t, u.AckAction(t.Context(), acker, action))
+		acker.AssertCalled(t, "Ack", mock.Anything, action)
+		acker.AssertCalled(t, "Commit", mock.Anything)
+	})
+
+	t.Run("AckAction with acker - failing commit", func(t *testing.T) {
+		acker := &fakeAcker{}
+
+		errCommit := errors.New("failed commit")
+		acker.On("Ack", mock.Anything, action).Return(nil)
+		acker.On("Commit", mock.Anything).Return(errCommit)
+
+		require.ErrorIs(t, u.AckAction(t.Context(), acker, action), errCommit)
+		acker.AssertCalled(t, "Ack", mock.Anything, action)
+		acker.AssertCalled(t, "Commit", mock.Anything)
+	})
+
+	t.Run("AckAction with acker - failed ack", func(t *testing.T) {
+		acker := &fakeAcker{}
+
+		errAck := errors.New("ack error")
+		acker.On("Ack", mock.Anything, action).Return(errAck)
+		acker.On("Commit", mock.Anything).Return(nil)
+
+		require.ErrorIs(t, u.AckAction(t.Context(), acker, action), errAck)
+		acker.AssertCalled(t, "Ack", mock.Anything, action)
+		acker.AssertNotCalled(t, "Commit", mock.Anything)
+	})
+}
+
 func prepareTestUpgraderReload() (string, artifact.Config) {
 	cfgyaml := `
 agent.download:
@@ -1227,3 +1276,19 @@ func TestIsSameReleaseVersion(t *testing.T) {
 		})
 	}
 }
+
+var _ acker.Acker = &fakeAcker{}
+
+type fakeAcker struct {
+	mock.Mock
+}
+
+func (f *fakeAcker) Ack(ctx context.Context, action fleetapi.Action) error {
+	args := f.Called(ctx, action)
+	return args.Error(0)
+}
+
+func (f *fakeAcker) Commit(ctx context.Context) error {
+	args := f.Called(ctx)
+	return args.Error(0)
+}


Reviews:
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T12:47:01Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T13:13:12Z
user: login: blakerouse
body: Overall this code looks good. Changes are clear and easy to understand. Will hold off on +1 until passing CI. Also, see inline comment. Maybe that could be reduced?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T13:27:50Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T20:57:11Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:01:34Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:05:22Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:08:36Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T06:57:43Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T10:03:14Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T07:05:15Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T16:57:00Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:52:21Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T13:18:05Z
user: login: blakerouse
body: This looks good. I believe you are now covering all 3 cases that Craig mentioned, and the code change is clear. Plus green CI!
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T15:42:39Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T19:53:45Z
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T19:54:06Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:00:04Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:01:21Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T06:15:41Z
user: login: cmacknz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T16:11:11Z
user: login: michalpristas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T07:39:04Z
user: login: cmacknz
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-07T16:49:27Z

Review comments:
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: michalpristas
body: the reason for such a long wait is that we need to wait for Fleet to start another round of checkin to recalculate state.
created_at: 2025-03-31T12:47:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: michalpristas
body: also i need to add check for upgraded at/upgrade started fields but for that i need change in elastic-agent-libs. so followup in case i don't make the change in libs in time
created_at: 2025-03-31T13:13:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: blakerouse
body: Yeah that is a long time, any ideas on how to shorten it? Reduce the check-in window?
created_at: 2025-03-31T13:26:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: cmacknz
body: We had the same problem in https://github.com/elastic/elastic-agent/pull/7190 where all the timeouts got bumped to 5 minutes.

Shortening the checkin window works and I suspect is the fastest path. I like the idea of being able to trigger a checkin to happen and return on demand but that's a bit more work. Any action triggers the long poll to return so stuff like changing the log level triggers this in a hacky way, maybe a NOOP action would be the tidiest way to get an on-demand checkin response without changing too much.
created_at: 2025-03-31T20:57:10Z
author_association: MEMBER
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: cmacknz
body: Also, we should evaluate if we can test this fix without an integration test. We already have really good upgrade integration test coverage so adding more isn't strictly necessary.

Would calling Upgrade() with a mock acker implementation be enough to validate this instead?
created_at: 2025-03-31T21:01:34Z
author_association: MEMBER
diff_hunk: @@ -125,8 +125,7 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: Is removing the source URI check actually correct? In air gapped if someone changes the source URI from one that doesn't work to one that does, the source URI changing means it is not the same upgrade action anymore.
created_at: 2025-03-31T21:05:22Z
author_association: MEMBER
diff_hunk: @@ -585,7 +592,7 @@ func (c *Coordinator) Upgrade(ctx context.Context, version string, sourceURI str
 		if errors.Is(err, upgrade.ErrUpgradeSameVersion) {
 			// Set upgrade state to completed so update no longer shows in-progress.
 			det.SetState(details.StateCompleted)
-			return nil
+			return c.upgradeMgr.AckAction(ctx, c.fleetAcker, action)
path: internal/pkg/agent/application/coordinator/coordinator.go
user: login: cmacknz
body: What timeout and retry policy does this have? Is their one? 

If Fleet Server is unavailable for an hour and all acks are failing what is the behavior of this call?
created_at: 2025-03-31T21:08:36Z
author_association: MEMBER
diff_hunk: @@ -125,8 +125,7 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: michalpristas
body: but in this case action ID should be different right. 
in case we have same version and same source uri it does not necessarily means it is the same action that we don't need to ack. 
we still need to ack and resolve as duplicate later more precisely using a hash.
created_at: 2025-04-01T06:57:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -585,7 +592,7 @@ func (c *Coordinator) Upgrade(ctx context.Context, version string, sourceURI str
 		if errors.Is(err, upgrade.ErrUpgradeSameVersion) {
 			// Set upgrade state to completed so update no longer shows in-progress.
 			det.SetState(details.StateCompleted)
-			return nil
+			return c.upgradeMgr.AckAction(ctx, c.fleetAcker, action)
path: internal/pkg/agent/application/coordinator/coordinator.go
user: login: michalpristas
body: i went with simple case with no retries for readability and that the probability with immediate ack is low. 
it can still happen though. i refactored code to pass things around from the top but it don't like it that much, it makes this a mess 
created_at: 2025-04-01T10:03:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -341,6 +341,135 @@ func testFleetAirGappedUpgrade(t *testing.T, stack *define.Info, unprivileged bo
 	testUpgradeFleetManagedElasticAgent(ctx, t, stack, fixture, upgradeTo, policy, unprivileged)
 }
 
+func TestDuplicateAgentUpgradeAction(t *testing.T) {
+	info := define.Require(t, define.Requirements{
+		Group: Fleet,
+		Stack: &define.Stack{},
+		Local: false, // requires Agent installation
+		Sudo:  true,  // requires Agent installation
+	})
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	defer cancel()
+
+	// Start at the build version as we want to test the retry
+	// logic that is in the build.
+	startFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+	err = startFixture.Prepare(ctx)
+	require.NoError(t, err)
+	startVersionInfo, err := startFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	// Upgrade to a very same version.
+	endFixture, err := define.NewFixtureFromLocalBuild(t, define.Version())
+	require.NoError(t, err)
+
+	err = endFixture.Prepare(ctx)
+	require.NoError(t, err)
+
+	endVersionInfo, err := endFixture.ExecVersion(ctx)
+	require.NoError(t, err)
+
+	t.Logf("Testing Elastic Agent upgrade from %s to %s with Fleet...",
+		define.Version(), endVersionInfo.Binary.String())
+
+	kibClient := info.KibanaClient
+	startParsedVersion, err := version.ParseVersion(startVersionInfo.Binary.String())
+	require.NoError(t, err)
+
+	t.Log("Creating Agent policy...")
+	policy := defaultPolicy()
+	policyResp, err := kibClient.CreatePolicy(ctx, policy)
+	require.NoError(t, err, "failed creating policy")
+	policy = policyResp.AgentPolicy
+
+	t.Log("Creating Agent enrollment API key...")
+	createEnrollmentApiKeyReq := kibana.CreateEnrollmentAPIKeyRequest{
+		PolicyID: policyResp.ID,
+	}
+	enrollmentToken, err := kibClient.CreateEnrollmentAPIKey(ctx, createEnrollmentApiKeyReq)
+	require.NoError(t, err, "failed creating enrollment API key")
+
+	t.Log("Getting default Fleet Server URL...")
+	fleetServerURL, err := fleettools.DefaultURL(ctx, kibClient)
+	require.NoError(t, err, "failed getting Fleet Server URL")
+
+	t.Logf("Installing Elastic Agent (privileged)...")
+	var nonInteractiveFlag bool
+	if upgradetest.Version_8_2_0.Less(*startParsedVersion) {
+		nonInteractiveFlag = true
+	}
+	installOpts := atesting.InstallOpts{
+		NonInteractive: nonInteractiveFlag,
+		Force:          true,
+		EnrollOpts: atesting.EnrollOpts{
+			URL:             fleetServerURL,
+			EnrollmentToken: enrollmentToken.APIKey,
+		},
+		Privileged: true,
+	}
+	output, err := startFixture.Install(ctx, &installOpts)
+	require.NoError(t, err, "failed to install start agent [output: %s]", string(output))
+
+	t.Log("Waiting for Agent to be correct version and healthy...")
+	err = upgradetest.WaitHealthyAndVersion(ctx, startFixture, startVersionInfo.Binary, 2*time.Minute, 10*time.Second, t)
+	require.NoError(t, err)
+
+	agentID, err := startFixture.AgentID(ctx)
+	require.NoError(t, err)
+	t.Logf("Agent ID: %q", agentID)
+
+	t.Logf("Upgrading from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	var agent kibana.GetAgentResponse
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Logf("Duplicate upgrade from version \"%s-%s\" to version \"%s-%s\"...",
+		startParsedVersion, startVersionInfo.Binary.Commit,
+		endVersionInfo.Binary.String(), endVersionInfo.Binary.Commit)
+	err = fleettools.UpgradeAgent(ctx, kibClient, agentID, endVersionInfo.Binary.String(), true)
+	require.NoError(t, err)
+
+	t.Log("Waiting for upgrade details to show up in Fleet")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails != nil
+	},
+		5*time.Minute, time.Second,
+		"last error: %v. agent.UpgradeDetails: %s",
+		err, agentUpgradeDetailsString(agent))
+
+	t.Log("Waiting for agent to have empty upgrade details")
+	require.Eventuallyf(t, func() bool {
+		agent, err = kibClient.GetAgent(ctx, kibana.GetAgentRequest{ID: agentID})
+		return err == nil && agent.UpgradeDetails == nil
+	},
+		20*time.Minute, time.Second,
path: testing/integration/upgrade_fleet_test.go
user: login: michalpristas
body: remove Integ test and added unit test for coordinator and upgrade handler
created_at: 2025-04-02T07:05:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,8 +125,7 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: Not necessarily, I don't think Fleet guarantees actions with the same content have the same ID if they are triggered separately in the UI.

This does call out there are three cases to handle properly:
1. Getting the exact same action twice, detectable by the action ID.
2. Getting different action IDs with the same content. For example a user spamming the upgrade agent button or a retry from a the upgrade agent Fleet API.
3. Getting an upgrade action for an upgrade that has no work to do. Having a single action in the queue telling an 8.16.3 agent it should upgrade to 8.16.3.

The 3rd case is the one I was focusing on in https://github.com/elastic/elastic-agent/issues/6186. InfoSec seems to just bulk upgrade all agents to 8.16.3 until they all are upgraded so we can run into all 3 of these cases as they do that. That is what is bringing these problems on.

The changes I see here now are only addressing case 1, when before it looked like we tried to handle both case 1 and 2 by looking only at the content, and the bug was in case 3 where we actually got into the upgrade handler. In that case we'd hit the code in https://github.com/elastic/elastic-agent/blob/56901edc3f2c6d1fc0c1271fad374cdbb24ed7f2/internal/pkg/agent/application/upgrade/upgrade.go#L215-L218

That returns an error that possibly causes us to not ack the action or ack it with an error.
created_at: 2025-04-02T16:57:00Z
author_association: MEMBER
diff_hunk: @@ -125,8 +125,7 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: michalpristas
body: i'm not assuming that actions with same content have same IDs quite the opposite and yes different action with same content can be delivered, and i don;t think this is a problem. upgrade handles them sequentially.
in case of snapshots it is a valid case, it can be 2 different snapshots. 
the thing is if we don't ack all of them agent will still be shown as upgrading.

changes here addresses deduplication of actions that are identical. 
deduplication on a content level is done later when we download a package. we determine that hash is the same as installed, ack and abort. this covers cases 2 and 3. (3 was not acked before)

here we ack on restart. that's valid for normal flow
but in case we run into a case where we receive action that differs in hash 
 - this adds to queue and ack that action in ugprade handler (now)
 - then we receive something that has same content 
 - we compare (old flow) add to queue never kick upgrade, never kick ack
 
the only way how to ack this one is to receive upgrade and not restart in between (queue is not persisted)
also not acking can lead to resending action to agent causing us to grow queue more and more. 
created_at: 2025-04-03T08:52:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,8 +125,7 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: michalpristas
body: to make it more robust and avoid cancelling first action in favor of second one i added one additional handling case.
checking source uri and version. in case these are the same we ack immediately but we don't append action to the queue to avoid queue getting too big in case of spamming.

created_at: 2025-04-03T13:18:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,11 +140,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: This makes sense, looking closer it is doing almost the same thing as https://github.com/elastic/elastic-agent/blob/894ef9385efddcff69cd77ab0da3635c1db31f6a/internal/pkg/agent/application/upgrade/upgrade.go#L167-L174

This case considers source URI, and the case I linked does not and most notably happens after the agent package is downloaded.

I think part of this bug happening is that we were doing duplicate upgrade detection in two different places. 

It seems like you could get rid of this check and the goroutine here, and let the implementation in upgrade.go do this detection, but before we actually download the agent package.

Then the ack gets covered by https://github.com/elastic/elastic-agent/blob/894ef9385efddcff69cd77ab0da3635c1db31f6a/internal/pkg/agent/application/coordinator/coordinator.go#L582-L589

I think what I might have missed in the first review was that this check was happening in two places. In either case the actual bug fix is we didn't ack when this happened and the action ID was different. 
created_at: 2025-04-03T19:53:45Z
author_association: MEMBER
diff_hunk: @@ -91,17 +92,31 @@ func (h *Upgrade) Handle(ctx context.Context, a fleetapi.Action, ack acker.Acker
 // ackActions Acks all the actions in bkgActions, and deletes entries from bkgActions.
 // User is responsible for obtaining and releasing bkgMutex lock
 func (h *Upgrade) ackActions(ctx context.Context, ack acker.Acker) {
+	h.ackLock.Lock()
+	defer h.ackLock.Unlock()
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: pkoutsovasilis
body: based on the godoc of `ackAction` I would expect here `ackLock` to be `bkgMutex`. Is `ackLock` intentional here, and if yes, could you please help me understand what concurrent pattern we are trying to prevent here? 🙂 
created_at: 2025-04-03T19:54:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,11 +140,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: Actually we already check for version equality before downloading the package in upgrade.go:

https://github.com/elastic/elastic-agent/blob/894ef9385efddcff69cd77ab0da3635c1db31f6a/internal/pkg/agent/application/upgrade/upgrade.go#L198-L203

So that part is already done. I think the ack goroutine here is unnecessary. I am trying to reason through if there is a possible deadlock between `ackLock` and `bkgMutex` and it's simpler if we just don't need `ackLock` at all.

`isSameReleaseVersion` doesn't consider sourceURI like this does, and it could, but it's hard for me to think through a real case where the version wouldn't change. The exception is probably just major.minor.patch-SNAPSHOT where the snapshot ID is different but not captured in the version.
created_at: 2025-04-03T20:00:04Z
author_association: MEMBER
diff_hunk: @@ -91,17 +92,31 @@ func (h *Upgrade) Handle(ctx context.Context, a fleetapi.Action, ack acker.Acker
 // ackActions Acks all the actions in bkgActions, and deletes entries from bkgActions.
 // User is responsible for obtaining and releasing bkgMutex lock
 func (h *Upgrade) ackActions(ctx context.Context, ack acker.Acker) {
+	h.ackLock.Lock()
+	defer h.ackLock.Unlock()
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: I think we can probably get rid of the need for the second lock: https://github.com/elastic/elastic-agent/pull/7635/files#r2027652932
created_at: 2025-04-03T20:01:21Z
author_association: MEMBER
diff_hunk: @@ -125,11 +140,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: michalpristas
body: yes the check before let snapshots pass otherwise it's early termination of processing without downloading a package, then the check in https://github.com/elastic/elastic-agent/blob/894ef9385efddcff69cd77ab0da3635c1db31f6a/internal/pkg/agent/application/upgrade/upgrade.go#L167-L174
takes hash into account to handle SNAPSHOT cases

dup lock was removed

created_at: 2025-04-04T06:15:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,11 +137,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
+		// not the same action this one needs to be acked
+		go func() {
+			// kick it off and don't block, lock to prevent race with ackActions from finished upgrade
+			h.bkgMutex.Lock()
+			defer h.bkgMutex.Unlock()
+
+			h.ackAction(ctx, ack, upgradeAction, true)
+		}()
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: cmacknz
body: Is there an actual reason for this to be async? Using the same lock, this should most of the time execute immediately after `getAsyncContext` returns. If another upgrade action came in, it would then block again in `getAsyncContext` until this completes anyway (or `getAsyncContext` wins and this keeps waiting). This is basically synchronous.

There's a synchronous call to ack actions right after this one in https://github.com/michalpristas/elastic-agent/blob/3229be39c543ea278bebaac99520b476268e19c6/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go#L164-L165

```
	// Versions must be different, cancel the first upgrade and run the new one
	h.log.Infof("Canceling upgrade to version %s received",
		bkgAction.Data.Version)
	h.bkgCancel()

	// Ack here because we have the lock, and we need to clear out the saved actions
	h.ackActions(ctx, ack)
```

That makes this a bit inconsistent, in that the same function either does or does not ack synchronously for no clear reason.

You could also just enqueue this action and then the upgrade handler will deal with detecting that the action has no work to do and ack it as a duplicate.
created_at: 2025-04-04T16:11:11Z
author_association: MEMBER
diff_hunk: @@ -125,11 +137,22 @@ func (h *Upgrade) getAsyncContext(ctx context.Context, action fleetapi.Action, a
 		h.log.Errorf("invalid type, expected ActionUpgrade and received %T", action)
 		return nil, false
 	}
-	if (upgradeAction.Data.Version == bkgAction.Data.Version) &&
-		(upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI) {
+	if upgradeAction.ActionID == bkgAction.ActionID {
 		h.log.Infof("Duplicate upgrade to version %s received",
 			bkgAction.Data.Version)
-		h.bkgActions = append(h.bkgActions, action)
+		return nil, false
+	}
+
+	if upgradeAction.Data.Version == bkgAction.Data.Version &&
+		upgradeAction.Data.SourceURI == bkgAction.Data.SourceURI {
+		// not the same action this one needs to be acked
+		go func() {
+			// kick it off and don't block, lock to prevent race with ackActions from finished upgrade
+			h.bkgMutex.Lock()
+			defer h.bkgMutex.Unlock()
+
+			h.ackAction(ctx, ack, upgradeAction, true)
+		}()
path: internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go
user: login: michalpristas
body: i did it async to allow early exit in favor of actually useful actions.
also uprade is performed in a separate goroutine acking on return what makes it consistent in my point of view.

sync in an example you shared is synchronous because you need to flush the queue (cancelled upgrades in favor of new one) so you cannot risk flushing action that sneaked in. 

this line acks just a single action, so risk of acking sneaked in action is non existent
created_at: 2025-04-07T07:39:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: elasticmachine
created_at: 2025-03-31T12:46:03Z
body: Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)
author_association: CONTRIBUTOR
user: login: elasticmachine
created_at: 2025-04-04T07:56:15Z
body: ## :green_heart: Build Succeeded
* [Buildkite Build](https://buildkite.com/elastic/elastic-agent/builds/19598)
* Commit: 3229be39c543ea278bebaac99520b476268e19c6

### History
* :yellow_heart: [Build #19544](https://buildkite.com/elastic/elastic-agent/builds/19544) was flaky d638e1f546a144058ac5a9f19023a32d6a64a7d3
* :yellow_heart: [Build #19467](https://buildkite.com/elastic/elastic-agent/builds/19467) was flaky c759091031ce0fc2af572482ab036b6fc4e3c832
* :yellow_heart: [Build #19426](https://buildkite.com/elastic/elastic-agent/builds/19426) was flaky 9bc0892d402c3cc8b3c47f9b833490699002f8de
* :broken_heart: [Build #19423](https://buildkite.com/elastic/elastic-agent/builds/19423) failed cb67fa9aaddabf676cd7879eaba93420789d2c6d
* :broken_heart: [Build #19409](https://buildkite.com/elastic/elastic-agent/builds/19409) failed 1c559d36d5d8081c009a28160d78cb68c5e5832d
* :broken_heart: [Build #19365](https://buildkite.com/elastic/elastic-agent/builds/19365) failed 2844ea5c1035e5b099e47786d4e03a4e8b0a1c9c

cc @michalpristas

<!--buildkite-pr-comment-elastic-agent
{"builds":[{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/elastic-agent/builds/19598","number":19598,"commit":"3229be39c543ea278bebaac99520b476268e19c6"},{"buildStatus":{"state":"passed","success":true,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19544","number":19544,"commit":"d638e1f546a144058ac5a9f19023a32d6a64a7d3"},{"buildStatus":{"state":"passed","success":true,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19467","number":19467,"commit":"c759091031ce0fc2af572482ab036b6fc4e3c832"},{"buildStatus":{"state":"passed","success":true,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19426","number":19426,"commit":"9bc0892d402c3cc8b3c47f9b833490699002f8de"},{"buildStatus":{"state":"failing","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19423","number":19423,"commit":"cb67fa9aaddabf676cd7879eaba93420789d2c6d"},{"buildStatus":{"state":"failed","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19409","number":19409,"commit":"1c559d36d5d8081c009a28160d78cb68c5e5832d"}],"number":19598}
buildkite-pr-comment-->
author_association: CONTRIBUTOR
user: login: cmacknz
created_at: 2025-04-04T16:13:04Z
body: Since we are only unit testing this one, which I agree makes the most sense, did you manually trigger duplicate actions via the Fleet API to make sure this actually resolved the problem in the way we expect?

The whole reason for this change is we never manually tested this with a Fleet managed agent so let's make sure we don't do the same thing again.
author_association: MEMBER
user: login: michalpristas
created_at: 2025-04-07T08:06:44Z
body: i did handful of scenarios when manually testing this
author_association: CONTRIBUTOR
user: login: elastic-sonarqube[bot]
created_at: 2025-04-07T09:16:28Z
body: ## [![Quality Gate passed](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonar.elastic.dev/dashboard?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635) **Quality Gate passed**  
Issues  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&resolved=false) [4 New issues](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&resolved=false)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/fixed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&fixedInPullRequest=7635) [1 Fixed issue](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&fixedInPullRequest=7635)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/accepted-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&issueStatuses=ACCEPTED) [0 Accepted issues](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&issueStatuses=ACCEPTED)

Measures  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/security_hotspots?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635) [0 Security Hotspots](https://sonar.elastic.dev/security_hotspots?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&metric=new_coverage&view=list) [45.5% Coverage on New Code](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&metric=new_coverage&view=list)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&metric=new_duplicated_lines_density&view=list) [0.0% Duplication on New Code](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube](https://sonar.elastic.dev/dashboard?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7635)


author_association: NONE