Repository: elastic/elastic-agent
Pull request #7651: Pre-install EPM packages needed by integration tests in Fleet
Author: login: ycombinator
Labels: name: Team:Elastic-Agent-Control-Plane
description: Label for the Agent Control Plane team
name: skip-changelog
name: backport-active-all
description: Automated backport with mergify to all the active branches
Created: 2025-03-31T22:21:11Z | Merged: 2025-04-03T16:50:48Z

## What does this PR do?

This PR installs EPM packages needed by integration tests into Fleet, prior to the tests being executed.

## Why is it important?
Several integration tests rely on EPM packages being installed in Fleet.  If two tests depending on the same package run concurrently, one test may try to install the package while the other has already started the process.  This results in conflicts and flaky tests. 

This PR provides a stop-gap solution in which the test runner installs the packages right after the stack has been provisioned and before any integration tests are executed against that stack.

<!-- Mandatory
Explain here the WHY, or the rationale/motivation for the changes.
-->

## Checklist

<!-- Mandatory
Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
-->

- [x] I have read and understood the [pull request guidelines](https://github.com/elastic/elastic-agent/blob/main/CONTRIBUTING.md#pull-request-guidelines) of this project.
- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- ~[ ] I have made corresponding changes to the documentation~
- ~[ ] I have made corresponding change to the default configuration files~
- ~[ ] I have added tests that prove my fix is effective or that my feature works~
- ~[ ] I have added an entry in `./changelog/fragments` using the [changelog tool](https://github.com/elastic/elastic-agent#changelog)~
- ~[ ] I have added an integration test or an E2E test~

## Disruptive User Impact

<!--
Will the changes introduced by this PR cause disruption to users in any way? If so, please describe what changes users
could make on their end to nullify or minimize this disruption. Consider impacts in related systems, not just directly
when using Elastic Agent.
-->
None, as this is impacts integration testing, not the built product.

## Related issues

<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.

- Closes #123
- Relates #123
- Requires #123
- Superseds #123
-->
Relates to #7529


Linked issue #123: [Elastic Agent] Add ability to debug a composable provider
## Overview

To help support dynamic inputs elastic/beats#19225 Elastic Agent needs to add the ability to debug the providers using for variable substitution. This issue is to track the debugging effort, for information about variable substitution review elastic/beats#20781

## Debugging

This obviously adds a lot of confusion to what the resulting configuration that Elastic Agent will be running with. To ensure that the feature is deployed correctly and that providers are working as expected debugging needs to be a top priority in the implementation.

### Debugging the running daemon

With the new ability to communicate with the running daemon the `inspect` command should be changed to talk to the running daemon and return the current configuration that is being used in memory. This will ensure that with running providers like Docker and Kubernetes it is easy to inspect what the resulting configuration is.

The current `inspect` and `output` commands can be combined and moved under the `debug` subcommand. (**Note**: This is not connecting to the currently running Elastic Agent)

```
$ ./elastic-agent debug config
```

Possible to watch the configuration as changes come in with `--watch`.

```
$ ./elastic-agent debug config --watch
```

### Debugging a single provider

A new `debug` command should be implemented that runs a single provider and output what it's currently providing back to the Elastic Agent. (**Note**: This is not connecting to the currently running Elastic Agent)

Example outputting docker provider inventory key/value mappings:

```
$ ./elastic-agent debug provider docker 
{"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}},}
{"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}},}
{"id": "2", "mapping": nil}
```

Example rendering configurations with changes:

```
$ ./elastic-agent debug provider docker -c testing-conf.yml
# {"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
# {"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: /var/log/containers/2.log
    processors:
      - add_fields:
          container.name: other-container
# {"id": "2", "mapping": nil}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
Linked issue comments: user: login: elasticmachine
created_at: 2020-09-17T17:47:38Z
body: Pinging @elastic/ingest-management (Team:Ingest Management)
author_association: CONTRIBUTOR
user: login: ruflin
created_at: 2020-09-18T07:16:53Z
body: ++ on the proposal. I would skip the `--watch` part at first probably. I really like that the state of a provider can be read out and also the current running config.
author_association: CONTRIBUTOR
user: login: elasticmachine
created_at: 2021-10-29T10:25:47Z
body: Pinging @elastic/elastic-agent-control-plane  (Team:Elastic-Agent-Control-Plane)
author_association: CONTRIBUTOR
user: login: jlind23
created_at: 2021-10-29T15:14:29Z
body: @michel-laterman was it part of your diagnosis job? If yes, can we close this?`
author_association: CONTRIBUTOR
user: login: michel-laterman
created_at: 2021-10-29T16:16:33Z
body: The scope of the outlined issue is more then what is currently included with `diagnostics`, let's keep this open as I think we should add this to the command (eventually)
author_association: CONTRIBUTOR
user: login: michel-laterman
created_at: 2022-03-21T19:25:23Z
body: I want to add a little more details here as it relates to diagnostics.

Currently `diagnostics` will use the exact same output of `inspect` (and `inspect output`) when gathering config from the agent and running processes.
`diagnostics` also gathers its own copy of the local (`elastic-agent.yml`/`fleet.yml`) config files (I think with variable substitution?).

The `diagnostics` command should provide the process config/policies/etc from the running agent.

However, I don't think we should name the command that does not contact from to the running agent debug as we tend to use diagnostics and debug interchangeably; maybe add it as a diagnostic subcommand, or add a flag to `inspect`?
author_association: CONTRIBUTOR

Diff:
diff --git a/.buildkite/scripts/steps/ess.sh b/.buildkite/scripts/steps/ess.sh
index 1da2ba5c165..563648f85a7 100755
--- a/.buildkite/scripts/steps/ess.sh
+++ b/.buildkite/scripts/steps/ess.sh
@@ -2,7 +2,7 @@
 set -euo pipefail
 
 function ess_up() {
-  echo "~~~ Staring ESS Stack"  
+  echo "~~~ Starting ESS Stack"
   local WORKSPACE=$(git rev-parse --show-toplevel)
   local TF_DIR="${WORKSPACE}/test_infra/ess/"
   local STACK_VERSION=$1
diff --git a/.buildkite/scripts/steps/ess_start.sh b/.buildkite/scripts/steps/ess_start.sh
index 09190afcd81..79322ef665c 100755
--- a/.buildkite/scripts/steps/ess_start.sh
+++ b/.buildkite/scripts/steps/ess_start.sh
@@ -2,14 +2,16 @@
 set -euo pipefail
 
 source .buildkite/scripts/common2.sh
-
 source .buildkite/scripts/steps/ess.sh
+source .buildkite/scripts/steps/fleet.sh
 
 OVERRIDE_STACK_VERSION="$(cat .package-version)"
 OVERRIDE_STACK_VERSION=${OVERRIDE_STACK_VERSION}"-SNAPSHOT"
 
 ess_up $OVERRIDE_STACK_VERSION
 
+preinstall_fleet_packages
+
 echo "ES_HOST: ${ELASTICSEARCH_HOST}"
 buildkite-agent meta-data set "es.host" $ELASTICSEARCH_HOST
 buildkite-agent meta-data set "es.username" $ELASTICSEARCH_USERNAME
diff --git a/.buildkite/scripts/steps/fleet.sh b/.buildkite/scripts/steps/fleet.sh
new file mode 100755
index 00000000000..21e3ff889ea
--- /dev/null
+++ b/.buildkite/scripts/steps/fleet.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+preinstalled_packages_filename="testing/integration/testdata/preinstalled_packages.json"
+
+# preinstall_fleet_packages installs EPM packages needed for integration tests into Fleet.
+function preinstall_fleet_packages() {
+  echo "Preinstalling the following EPM packages in Fleet:"
+  cat "$preinstalled_packages_filename"
+
+  install_fleet_packages "$preinstalled_packages_filename"
+  retcode=$?
+
+  return $retcode
+}
+
+# install_fleet_packages installs the EPM packages defined in the given file into Fleet. The file
+# must define the packages as the request body of the Bulk install packages Fleet API:
+# https://www.elastic.co/docs/api/doc/kibana/v8/operation/operation-bulk-install-packages
+function install_fleet_packages() {
+  local install_api_request_file=$1
+  if [ -z "$install_api_request_file" ]; then
+    echo "Error: Fleet packages installation request file not specified"
+    return 1
+  fi
+
+  if ! [ -f "$install_api_request_file" ]; then
+    echo "Error: Fleet packages installation request file [$install_api_request_file] does not exist"
+    return 2
+  fi
+
+  if [ -z "$KIBANA_HOST" ]; then
+    echo "Error: Kibana hostname not specified via KIBANA_HOST environment variable"
+    return 3
+  fi
+
+  if [ -z "$KIBANA_USERNAME" ]; then
+    echo "Error: Kibana username not specified via KIBANA_USERNAME environment variable"
+    return 4
+  fi
+
+  if [ -z "$KIBANA_PASSWORD" ]; then
+    echo "Error: Kibana password not specified via KIBANA_PASSWORD environment variable"
+    return 5
+  fi
+
+  resp=$(curl \
+    -v \
+    -s \
+    --fail-with-body \
+    -X "POST" \
+    -u "${KIBANA_USERNAME}:${KIBANA_PASSWORD}" \
+    -d @"${install_api_request_file}" \
+    -H 'Content-Type: application/json' \
+    -H 'kbn-xsrf: elastic-agent' \
+    "${KIBANA_HOST}/api/fleet/epm/packages/_bulk")
+
+  echo "$resp"
+
+  # Parse response body for any errors
+  num_errors=$(echo "$resp" | jq '.items[].statusCode | select(.>=400)' | wc -l)
+  if [ "$num_errors" -gt 0 ]; then
+    return 6
+  fi
+}
diff --git a/testing/integration/agent_long_running_leak_test.go b/testing/integration/agent_long_running_leak_test.go
index 69ed98fd882..89e3c02d5d3 100644
--- a/testing/integration/agent_long_running_leak_test.go
+++ b/testing/integration/agent_long_running_leak_test.go
@@ -120,10 +120,10 @@ func (runner *ExtendedRunner) SetupSuite() {
 	policyResp, _, err := tools.InstallAgentWithPolicy(ctx, runner.T(), installOpts, runner.agentFixture, runner.info.KibanaClient, basePolicy)
 	require.NoError(runner.T(), err)
 
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", "1.53.1", "agent_long_test_base_system_integ.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", preinstalledPackages["system"], "agent_long_test_base_system_integ.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "apache", "1.17.0", "agent_long_test_apache.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "apache", preinstalledPackages["apache"], "agent_long_test_apache.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
 }
diff --git a/testing/integration/agent_long_test_apache.json b/testing/integration/agent_long_test_apache.json
index f1f3a6d27b2..8cb921dadd7 100644
--- a/testing/integration/agent_long_test_apache.json
+++ b/testing/integration/agent_long_test_apache.json
@@ -7,7 +7,7 @@
     "package": {
       "name": "apache",
       "title": "Apache HTTP Server",
-      "version": "1.17.0"
+      "version": "1.30.0"
     },
     "enabled": true,
     "inputs": [
diff --git a/testing/integration/auditd_monitoring_test.go b/testing/integration/auditd_monitoring_test.go
index 31a34c3143b..61e91d2f61c 100644
--- a/testing/integration/auditd_monitoring_test.go
+++ b/testing/integration/auditd_monitoring_test.go
@@ -75,7 +75,7 @@ func (runner *AuditDRunner) SetupSuite() {
 	require.NoError(runner.T(), err)
 
 	packageFile := filepath.Join("testdata", "auditd_package.json")
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "auditd_manager", "1.18.3", packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "auditd_manager", preinstalledPackages["auditd_manager"], packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
 }
diff --git a/testing/integration/endpoint_test_tools.go b/testing/integration/endpoint_test_tools.go
index 6a8e974fc15..a78cb2cfee4 100644
--- a/testing/integration/endpoint_test_tools.go
+++ b/testing/integration/endpoint_test_tools.go
@@ -34,9 +34,6 @@ type endpointPackageTemplateVars struct {
 	Version  string
 }
 
-// TODO: Setup a GitHub Action to update this for each release of https://github.com/elastic/endpoint-package
-const endpointPackageVersion = "8.11.0"
-
 func agentAndEndpointAreHealthy(t *testing.T, ctx context.Context, agentClient client.Client) bool {
 	t.Helper()
 
@@ -105,7 +102,7 @@ func installElasticDefendPackage(t *testing.T, info *define.Info, policyID strin
 		ID:       packagePolicyID,
 		Name:     "Defend-" + packagePolicyID,
 		PolicyID: policyID,
-		Version:  endpointPackageVersion,
+		Version:  preinstalledPackages["endpoint"],
 	})
 	if err != nil {
 		return r, fmt.Errorf("error executing template: %w", err)
diff --git a/testing/integration/fleetserver_test.go b/testing/integration/fleetserver_test.go
index 830cf4cc693..7a8ca23eda5 100644
--- a/testing/integration/fleetserver_test.go
+++ b/testing/integration/fleetserver_test.go
@@ -71,7 +71,7 @@ func TestInstallFleetServerBootstrap(t *testing.T) {
 	policy := policyResp.AgentPolicy
 
 	packageFile := filepath.Join("testdata", "fleet-server.json")
-	_, err = tools.InstallPackageFromDefaultFile(ctx, info.KibanaClient, "fleet-server", "1.5.0", packageFile, uuid.Must(uuid.NewV4()).String(), policy.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, info.KibanaClient, "fleet-server", preinstalledPackages["fleet-server"], packageFile, uuid.Must(uuid.NewV4()).String(), policy.ID)
 	require.NoError(t, err, "failed creating fleet-server integration")
 
 	t.Log("Get fleet-server service token...")
diff --git a/testing/integration/init.go b/testing/integration/init.go
new file mode 100644
index 00000000000..0acb59b5b75
--- /dev/null
+++ b/testing/integration/init.go
@@ -0,0 +1,54 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+// or more contributor license agreements. Licensed under the Elastic License 2.0;
+// you may not use this file except in compliance with the Elastic License 2.0.
+
+package integration
+
+import (
+	_ "embed"
+	"encoding/json"
+	"fmt"
+)
+
+// FIXME: this code should be removed once https://github.com/elastic/kibana/issues/213337
+// is resolved.
+var (
+	// preinstalledPackages is a map of packages that are pre-installed by the
+	// integration tests runner before any integration tests are executed. This is
+	// to prevent multiple concurrent tests from trying to install the same package
+	// and causing a conflict.
+	preinstalledPackages map[string]string
+
+	//go:embed testdata/preinstalled_packages.json
+	preinstalledPackagesJSON []byte
+)
+
+func init() {
+	err := initializePreinstalledPackages()
+	if err != nil {
+		panic("unable to parse list of preinstalled packages: " + err.Error())
+	}
+}
+
+type Package struct {
+	Name    string `json:"name"`
+	Version string `json:"version"`
+}
+
+func initializePreinstalledPackages() error {
+	var packages struct {
+		Packages []Package `json:"packages"`
+	}
+
+	err := json.Unmarshal(preinstalledPackagesJSON, &packages)
+	if err != nil {
+		return fmt.Errorf("unable to parse preinstalled packages JSON: %w", err)
+	}
+
+	preinstalledPackages = make(map[string]string, len(packages.Packages))
+	for _, pkg := range packages.Packages {
+		preinstalledPackages[pkg.Name] = pkg.Version
+	}
+
+	return nil
+}
diff --git a/testing/integration/logs_ingestion_test.go b/testing/integration/logs_ingestion_test.go
index 816d3b37a7a..e87431ed8a8 100644
--- a/testing/integration/logs_ingestion_test.go
+++ b/testing/integration/logs_ingestion_test.go
@@ -355,11 +355,12 @@ func testFlattenedDatastreamFleetPolicy(
 	// in Fleet.
 	agentPolicyBuilder := strings.Builder{}
 	err = tmpl.Execute(&agentPolicyBuilder, policyVars{
-		Name:        "Log-Input-" + t.Name() + "-" + time.Now().Format(time.RFC3339),
-		PolicyID:    policy.ID,
-		LogFilePath: logFilePath,
-		Namespace:   dsNamespace,
-		Dataset:     dsDataset,
+		Name:              "Log-Input-" + t.Name() + "-" + time.Now().Format(time.RFC3339),
+		PolicyID:          policy.ID,
+		LogFilePath:       logFilePath,
+		Namespace:         dsNamespace,
+		Dataset:           dsDataset,
+		LogPackageVersion: preinstalledPackages["log"],
 	})
 	if err != nil {
 		t.Fatalf("could not render template: %s", err)
@@ -498,7 +499,7 @@ var policyJSON = `
   "policy_id": "{{.PolicyID}}",
   "package": {
     "name": "log",
-    "version": "2.3.0"
+    "version": "{{.LogPackageVersion}}"
   },
   "name": "{{.Name}}",
   "namespace": "{{.Namespace}}",
@@ -521,11 +522,12 @@ var policyJSON = `
 }`
 
 type policyVars struct {
-	Name        string
-	PolicyID    string
-	LogFilePath string
-	Namespace   string
-	Dataset     string
+	Name              string
+	PolicyID          string
+	LogFilePath       string
+	Namespace         string
+	Dataset           string
+	LogPackageVersion string
 }
 
 type ESDocument struct {
diff --git a/testing/integration/metrics_monitoring_test.go b/testing/integration/metrics_monitoring_test.go
index 3155f2f7925..329efcac52a 100644
--- a/testing/integration/metrics_monitoring_test.go
+++ b/testing/integration/metrics_monitoring_test.go
@@ -75,7 +75,7 @@ func (runner *MetricsRunner) SetupSuite() {
 	policyResp, _, err := tools.InstallAgentWithPolicy(ctx, runner.T(), installOpts, runner.agentFixture, runner.info.KibanaClient, basePolicy)
 	require.NoError(runner.T(), err)
 
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", "1.53.1", "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", preinstalledPackages["system"], "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
 }
diff --git a/testing/integration/monitoring_probe_preserve_text_cfg_test.go b/testing/integration/monitoring_probe_preserve_text_cfg_test.go
index f9f1026547e..0ec20fdf2e5 100644
--- a/testing/integration/monitoring_probe_preserve_text_cfg_test.go
+++ b/testing/integration/monitoring_probe_preserve_text_cfg_test.go
@@ -126,7 +126,7 @@ func (runner *MonitoringTextRunner) SetupSuite() {
 	runner.policyID = policyResp.ID
 	runner.policyName = basePolicy.Name
 
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", "1.53.1", "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", preinstalledPackages["system"], "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 }
 
diff --git a/testing/integration/monitoring_probe_reload_test.go b/testing/integration/monitoring_probe_reload_test.go
index fed31c2107e..a6beebfc556 100644
--- a/testing/integration/monitoring_probe_reload_test.go
+++ b/testing/integration/monitoring_probe_reload_test.go
@@ -86,7 +86,7 @@ func (runner *MonitoringRunner) SetupSuite() {
 	runner.policyID = policyResp.ID
 	runner.policyName = basePolicy.Name
 
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", "1.53.1", "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "system", preinstalledPackages["system"], "system_integration_setup.json", uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 }
 
diff --git a/testing/integration/network_traffic_monitoring_test.go b/testing/integration/network_traffic_monitoring_test.go
index 97e5f7475b8..989c0bf9afe 100644
--- a/testing/integration/network_traffic_monitoring_test.go
+++ b/testing/integration/network_traffic_monitoring_test.go
@@ -76,7 +76,7 @@ func (runner *NetworkTrafficRunner) SetupSuite() {
 	require.NoError(runner.T(), err)
 
 	packageFile := filepath.Join("testdata", "network_traffic_package.json")
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "network_traffic", "1.32.1", packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "network_traffic", preinstalledPackages["network_traffic"], packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
 }
diff --git a/testing/integration/osquery_monitoring_test.go b/testing/integration/osquery_monitoring_test.go
index e631643eddf..b4f15f4c523 100644
--- a/testing/integration/osquery_monitoring_test.go
+++ b/testing/integration/osquery_monitoring_test.go
@@ -75,7 +75,7 @@ func (runner *OsqueryManagerRunner) SetupSuite() {
 	require.NoError(runner.T(), err)
 
 	packageFile := filepath.Join("testdata", "osquery_package.json")
-	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "osquery_manager", "1.16.0", packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
+	_, err = tools.InstallPackageFromDefaultFile(ctx, runner.info.KibanaClient, "osquery_manager", preinstalledPackages["osquery_manager"], packageFile, uuid.Must(uuid.NewV4()).String(), policyResp.ID)
 	require.NoError(runner.T(), err)
 
 }
diff --git a/testing/integration/testdata/auditd_package.json b/testing/integration/testdata/auditd_package.json
index 021a41ed8f6..cdf62c42dda 100644
--- a/testing/integration/testdata/auditd_package.json
+++ b/testing/integration/testdata/auditd_package.json
@@ -5,7 +5,7 @@
     "namespace": "default",
     "package": {
         "name": "auditd_manager",
-        "version": "1.18.3"
+        "version": "1.19.0"
     },
     "enabled": true,
     "policy_id": "0a4f6c12-446a-401a-b0eb-96afea6ca92d",
diff --git a/testing/integration/testdata/network_traffic_package.json b/testing/integration/testdata/network_traffic_package.json
index 269ceaad3bd..35974d1bd7e 100644
--- a/testing/integration/testdata/network_traffic_package.json
+++ b/testing/integration/testdata/network_traffic_package.json
@@ -5,7 +5,7 @@
     "namespace": "default",
     "package": {
         "name": "network_traffic",
-        "version": "1.32.1"
+        "version": "1.33.0"
     },
     "enabled": true,
     "policy_id": "0a4f6c12-446a-401a-b0eb-96afea6ca92d",
diff --git a/testing/integration/testdata/preinstalled_packages.json b/testing/integration/testdata/preinstalled_packages.json
new file mode 100644
index 00000000000..ad596854d36
--- /dev/null
+++ b/testing/integration/testdata/preinstalled_packages.json
@@ -0,0 +1,12 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" },
+        { "name": "auditd_manager", "version": "1.19.0" },
+        { "name": "fleet_server", "version": "1.5.0" },
+        { "name": "network_traffic", "version": "1.33.0" },
+        { "name": "osquery_manager", "version": "1.16.0" },
+        { "name": "apache", "version": "1.30.0" },
+        { "name": "log", "version": "2.4.0" }
+    ]
+}


Reviews:
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T19:16:14Z
user: login: ycombinator
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T19:21:15Z
user: login: ycombinator
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T19:28:32Z
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T19:33:04Z
user: login: ycombinator
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T20:06:59Z
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T20:28:51Z
user: login: ycombinator
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T20:32:12Z
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T20:34:52Z
user: login: v1v
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-04-02T23:20:01Z
user: login: pkoutsovasilis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:40:01Z
user: login: ycombinator
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T14:49:43Z
user: login: v1v
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-03T16:32:54Z
user: login: pkoutsovasilis
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T16:37:12Z

Review comments:
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: pkoutsovasilis
body: we also need here the following

```json
{ "name": "auditd_manager", "version": "1.18.3" },
{ "name": "fleet_server", "version": "1.5.0" },
{ "name": "network_traffic", "version": "1.32.1" },
{ "name": "osquery_manager", "version": "1.16.0" },
{ "name": "apache", "version": "1.17.0" },
{ "name": "log", "version": "2.3.0" }
```

Also the `endpoint` package in use in our code is at `8.11.0` and the `system` is at `1.53.0`. So definitely we need to align all of these versions to be the same in our code. 

some of the above are statically defined in json files under the path `testing/integration/testdata` and `testing/integration/`. Some others are part of the go code 
created_at: 2025-04-02T19:16:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: ycombinator
body: > Also the endpoint package in use in our code is at 8.11.0 and the system is at 1.53.0. So definitely we need to align all of these versions to be the same in our code.

The reason I bumped the versions of these two packages is that the versions currently in our code are not compatible with 9.1.0 stacks, so CI in this PR was failing.

Will add the other packages to the list, thanks.
created_at: 2025-04-02T19:21:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: ycombinator
body: > some of the above are statically defined in json files under the path testing/integration/testdata and testing/integration/. Some others are part of the go code

I have aligned the versions in code using the `preinstalledPackages` map introduced in this PR. 

To align the versions defined in JSON files, we will need to templatize them.  I'm happy to do that, but I'm not sure if that's overkill for a stop-gap solution? WDYT?

[EDIT] BTW, these are the JSON files in question where the versions are not aligned currently:
- https://github.com/elastic/elastic-agent/blob/0fee15b35e26a6ccbd27adef83632e09763403d2/testing/integration/testdata/auditd_package.json#L8
- https://github.com/elastic/elastic-agent/blob/0fee15b35e26a6ccbd27adef83632e09763403d2/testing/integration/testdata/fleet-server.json#L10
- https://github.com/elastic/elastic-agent/blob/0fee15b35e26a6ccbd27adef83632e09763403d2/testing/integration/testdata/network_traffic_package.json#L8
- https://github.com/elastic/elastic-agent/blob/0fee15b35e26a6ccbd27adef83632e09763403d2/testing/integration/testdata/osquery_package.json#L8
- https://github.com/elastic/elastic-agent/blob/0fee15b35e26a6ccbd27adef83632e09763403d2/testing/integration/agent_long_test_apache.json#L10
created_at: 2025-04-02T19:28:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: pkoutsovasilis
body: > The reason I bumped the versions of these two packages is that the versions currently in our code are not compatible with 9.1.0 stacks, so CI in this PR was failing.

that's interesting 🤔 then how we are not seeing these failures when the integration tests are running?! Are we using force under the hood?

> I have aligned the versions in code using the preinstalledPackages map introduced in this PR. However, I don't have a good approach to aligning the ones defined in JSON files unless we want to templatize them? Happy to do that, but I'm not sure if that's overkill for a stop-gap solution? WDYT?

I would say leave them be; if we need to implement a more thorough approach for installing these packages we can address this aspect there. Makes sense? 🙂
created_at: 2025-04-02T19:33:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: ycombinator
body: > that's interesting 🤔 then how we are not seeing these failures when the integration tests are running?! Are we using force under the hood?

Yes, I am wondering the same thing as well.  

In fact, this seems to be the case with some of the packages I just added to the list as well:

```
{"name":"auditd_manager","statusCode":400,"error":"auditd_manager-1.18.3 is out-of-date and cannot be installed or updated"}
{"name":"network_traffic","statusCode":400,"error":"network_traffic-1.32.1 is out-of-date and cannot be installed or updated"}
{"name":"apache","statusCode":400,"error":"apache-1.17.0 is out-of-date and cannot be installed or updated"}
{"name":"log","statusCode":400,"error":"log-2.3.0 is out-of-date and cannot be installed or updated"}
```

I'll bump their versions in this PR and try to figure out why this is not currently a problem on the `main` branch.
created_at: 2025-04-02T20:06:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: pkoutsovasilis
body: ok I just tested it and this extra param in curl does the trick `-d '{ "force": true }'` so please stay with the same package versions as we had them for now but we should definitely create an issue of properly handling these versions, wdyt? 🙂 
created_at: 2025-04-02T20:28:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: ycombinator
body: Since we are testing against 9.1.0 stacks in `main` and 9.0.0 stacks in the `9.0` branch, I think we should use the versions of packages that are intended to work with 9.x.  But when we backport this PR to `8.*` branches, we can add the `force` parameter to the curl request.  WDYT? 🙂 
created_at: 2025-04-02T20:32:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+{
+    "packages": [
+        { "name": "endpoint", "version": "9.0.0" },
+        { "name": "system", "version": "1.67.3" }
path: testing/integration/testdata/preinstalled_packages.json
user: login: pkoutsovasilis
body: oops I didn't test this in time and you went the extra mile to bump the versions everywhere. Then yes what you say makes sense. Waiting for the CI to be 🟢 as this looks good to me 🙂 
created_at: 2025-04-02T20:34:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+preinstalled_packages_filename="testing/integration/testdata/preinstalled_packages.json"
+
+# preinstall_fleet_packages installs EPM packages needed for integration tests into Fleet.
+function preinstall_fleet_packages() {
+  echo "Preinstalling the following EPM packages in Fleet:"
+  cat "$preinstalled_packages_filename"
+
+  install_fleet_packages "$preinstalled_packages_filename"
+  retcode=$?
+
+  return $retcode
+}
+
+# install_fleet_packages installs the EPM packages defined in the given file into Fleet. The file
+# must define the packages as the request body of the Bulk install packages Fleet API:
+# https://www.elastic.co/docs/api/doc/kibana/v8/operation/operation-bulk-install-packages
+function install_fleet_packages() {
+  local install_api_request_file=$1
path: .buildkite/scripts/steps/fleet.sh
user: login: v1v
body: Do you think we should avoid any 'verbose output' by forcing `set +x`?

```suggestion
  set +x
  local install_api_request_file=$1
```
created_at: 2025-04-02T23:19:21Z
author_association: MEMBER
diff_hunk: @@ -498,7 +499,7 @@ var policyJSON = `
   "policy_id": "{{.PolicyID}}",
   "package": {
     "name": "log",
-    "version": "2.3.0"
+    "version": {{.LogPackageVersion}}"
path: testing/integration/logs_ingestion_test.go
user: login: pkoutsovasilis
body: ```suggestion
    "version": "{{.LogPackageVersion}}"
```
created_at: 2025-04-03T08:40:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+preinstalled_packages_filename="testing/integration/testdata/preinstalled_packages.json"
+
+# preinstall_fleet_packages installs EPM packages needed for integration tests into Fleet.
+function preinstall_fleet_packages() {
+  echo "Preinstalling the following EPM packages in Fleet:"
+  cat "$preinstalled_packages_filename"
+
+  install_fleet_packages "$preinstalled_packages_filename"
+  retcode=$?
+
+  return $retcode
+}
+
+# install_fleet_packages installs the EPM packages defined in the given file into Fleet. The file
+# must define the packages as the request body of the Bulk install packages Fleet API:
+# https://www.elastic.co/docs/api/doc/kibana/v8/operation/operation-bulk-install-packages
+function install_fleet_packages() {
+  local install_api_request_file=$1
path: .buildkite/scripts/steps/fleet.sh
user: login: ycombinator
body: My preference would be to keep the verbose output for now, while this change is still somewhat new, in case it becomes helpful for debugging any issues.  Once things start running smoothly, we can add `set +x`.
created_at: 2025-04-03T14:49:42Z
author_association: CONTRIBUTOR

PR comments:
user: login: mergify[bot]
created_at: 2025-03-31T22:21:55Z
body: This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:
* `backport-./d./d` is the label that automatically backports to the `8./d` branch. `/d` is the digit
* `backport-active-all` is the label that automatically backports to all active branches.
* `backport-active-8` is the label that automatically backports to all active minor branches for the 8 major.
* `backport-active-9` is the label that automatically backports to all active minor branches for the 9 major.
author_association: CONTRIBUTOR
user: login: ycombinator
created_at: 2025-04-02T15:41:26Z
body: This PR is ready for review now.  

I've implemented pre-installing of packages (using `endpoint` and `system` for now but it's easy to add more) for the CI integration testing path.  You can see an example of the pre-installation in this CI run: https://buildkite.com/elastic/elastic-agent/builds/19471#0195f642-015f-4d1c-863d-1c0e702f970b/132-321

Do we want to also implement the same for the local testing path, where the stack provisioning is done in the [runner](https://github.com/elastic/elastic-agent/blob/cf16186003b7146e8a88de4c3268a9658d94b660/pkg/testing/runner/runner.go#L610)?  
author_association: CONTRIBUTOR
user: login: elasticmachine
created_at: 2025-04-02T16:44:44Z
body: Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)
author_association: CONTRIBUTOR
user: login: ycombinator
created_at: 2025-04-02T21:16:27Z
body: @pkoutsovasilis Thanks for your review.  There is one [pending question](https://github.com/elastic/elastic-agent/pull/7651#issuecomment-2772995257) I'd like your or other reviewers' opinion on:

> I've implemented pre-installing of packages... for the CI integration testing path... Do we want to also implement the same for the local testing path, where the stack provisioning is done in the [runner](https://github.com/elastic/elastic-agent/blob/cf16186003b7146e8a88de4c3268a9658d94b660/pkg/testing/runner/runner.go#L610)?
author_association: CONTRIBUTOR
user: login: pkoutsovasilis
created_at: 2025-04-03T08:53:52Z
body: > @pkoutsovasilis Thanks for your review. There is one [pending question](https://github.com/elastic/elastic-agent/pull/7651#issuecomment-2772995257) I'd like your or other reviewers' opinion on:
> 
> > I've implemented pre-installing of packages... for the CI integration testing path... Do we want to also implement the same for the local testing path, where the stack provisioning is done in the [runner](https://github.com/elastic/elastic-agent/blob/cf16186003b7146e8a88de4c3268a9658d94b660/pkg/testing/runner/runner.go#L610)?

Sorry I missed this question @ycombinator. IMO we aren't required to do this for the local testing part; the issue you are tackling is observed only for the integration tests that run with Buildkite runners and test groups are in parallel. Also we have upcoming work to migrate other tests to buildkite runners that will use the same ESS stack provisioned from the step you added the preinstallation, so things will converge 🙂 
author_association: CONTRIBUTOR
user: login: elastic-sonarqube[bot]
created_at: 2025-04-03T15:42:59Z
body: ## [![Quality Gate passed](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonar.elastic.dev/dashboard?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651) **Quality Gate passed**  
Issues  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&resolved=false) [0 New issues](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&resolved=false)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/fixed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&fixedInPullRequest=7651) [0 Fixed issues](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&fixedInPullRequest=7651)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/accepted-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&issueStatuses=ACCEPTED) [0 Accepted issues](https://sonar.elastic.dev/project/issues?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&issueStatuses=ACCEPTED)

Measures  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/security_hotspots?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651) [0 Security Hotspots](https://sonar.elastic.dev/security_hotspots?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651)  
![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/no-data-16px.png '') No data about Coverage  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&metric=new_duplicated_lines_density&view=list) [0.0% Duplication on New Code](https://sonar.elastic.dev/component_measures?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube](https://sonar.elastic.dev/dashboard?id=elastic_elastic-agent_AYluowg0xMq8P7b4moiZ&pullRequest=7651)


author_association: NONE
user: login: elasticmachine
created_at: 2025-04-03T16:27:52Z
body: ## :yellow_heart: Build succeeded, but was flaky
* [Buildkite Build](https://buildkite.com/elastic/elastic-agent/builds/19552)
* Commit: fadb4deef304acdc5cf1dc6a428fac275ba46058

### Failed CI Steps
* [x86_64:sudo: fleet-privileged](https://buildkite.com/elastic/elastic-agent/builds/19552#0195fc3f-5a61-4bf5-93ea-59b6f43c3c17)

### History
* :broken_heart: [Build #19508](https://buildkite.com/elastic/elastic-agent/builds/19508) failed 8a83332dc816f9e78fcc4111d9938a1bad92c965
* :broken_heart: [Build #19503](https://buildkite.com/elastic/elastic-agent/builds/19503) failed 0f9fc54678f0e11ee73433145e84fa8041c3f014
* :green_heart: [Build #19471](https://buildkite.com/elastic/elastic-agent/builds/19471) succeeded 1207be102412a7eea0d6b602ed2cfc8c64eb4311
* :broken_heart: [Build #19468](https://buildkite.com/elastic/elastic-agent/builds/19468) failed dee227793807ca3c5f374092b689b2d402cc6c94
* :yellow_heart: [Build #19465](https://buildkite.com/elastic/elastic-agent/builds/19465) was flaky b491c47876ca4809ca5aedb972e40ccfc63cc113

cc @ycombinator

<!--buildkite-pr-comment-elastic-agent
{"builds":[{"buildStatus":{"state":"passed","success":true,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19552","number":19552,"commit":"fadb4deef304acdc5cf1dc6a428fac275ba46058"},{"buildStatus":{"state":"failed","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19508","number":19508,"commit":"8a83332dc816f9e78fcc4111d9938a1bad92c965"},{"buildStatus":{"state":"failing","success":false,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/elastic-agent/builds/19503","number":19503,"commit":"0f9fc54678f0e11ee73433145e84fa8041c3f014"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/elastic-agent/builds/19471","number":19471,"commit":"1207be102412a7eea0d6b602ed2cfc8c64eb4311"},{"buildStatus":{"state":"failing","success":false,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/elastic-agent/builds/19468","number":19468,"commit":"dee227793807ca3c5f374092b689b2d402cc6c94"},{"buildStatus":{"state":"passed","success":true,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/elastic-agent/builds/19465","number":19465,"commit":"b491c47876ca4809ca5aedb972e40ccfc63cc113"}],"number":19552}
buildkite-pr-comment-->
author_association: CONTRIBUTOR