Repository: canonical/github-runner-image-builder-operator
Pull request #88: Fix: move external script secret out of cloud-init
Author: login: cbartz
Labels: name: bug
description: Something isn't working
name: Libraries: Out of sync
Created: 2025-02-27T12:44:04Z | Merged: 2025-03-06T07:46:27Z

Applicable spec: n/a

### Overview

Don't run the external script in cloud-init but use SSH instead.

This PR also fixes the app integration tests, which were not even executed (missing pytest_asyncio marker).

It has been verified that `/var/log/auth.log` and the journal no longer contain traces of external script commands.

### Rationale

cloud-init userdata is preserved in the image. There are potentially secrets that can be leaked.

### Juju Events Changes

n/a

### Module Changes

`openstack_builder` in the app now contains code for SSH execution. 


### Library Changes

<!-- Any changes to charm libraries -->

### Checklist

- [x] The [charm style guide](https://juju.is/docs/sdk/styleguide) was applied
- [x] The [contributing guide](https://github.com/canonical/is-charms-contributing-guide) was applied
- [x] The changes are compliant with [ISD054 - Managing Charm Complexity](https://discourse.charmhub.io/t/specification-isd014-managing-charm-complexity/11619)
- [x] The documentation for charmhub is updated.
- [x] The PR is tagged with appropriate label (`urgent`, `trivial`, `senior-review-required`, `documentation`)
- [x] `RELEASE-NOTES.md` is updated (only for user-related changes)
<!-- Explanation for any unchecked items above -->



Diff:
diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md
index 70900cdc..49773023 100644
--- a/RELEASE-NOTES.md
+++ b/RELEASE-NOTES.md
@@ -1,3 +1,25 @@
+## [#88 Fix: move external script secret out of cloud-init](https://github.com/canonical/github-runner-image-builder-operator/pull/88) (2025-03-04)
+> Move running the external script out of cloud-init and use SSH instead.
+
+
+### Upgrade Steps
+*  Nothing in particular to consider.
+
+### Breaking Changes
+* None
+
+### New Features
+* None
+
+### Bug Fixes
+* cloud-init userdata is preserved in the image and should not contain traces of the external script and secrets.
+
+### Performance Improvements
+* None
+
+### Other Changes
+* None
+
 ## [#85 fix: Periodic rebuilding of images](https://github.com/canonical/github-runner-image-builder-operator/pull/85) (2025-02-24)
 > Fix the periodic rebuilding of images.
 
diff --git a/app/src/github_runner_image_builder/errors.py b/app/src/github_runner_image_builder/errors.py
index 099b869d..3edcd0f5 100644
--- a/app/src/github_runner_image_builder/errors.py
+++ b/app/src/github_runner_image_builder/errors.py
@@ -81,6 +81,10 @@ class HomeDirectoryChangeOwnershipError(BuildImageError):
     """Represents an error while changing ubuntu home directory."""
 
 
+class ExternalScriptError(BuildImageError):
+    """Represents an error while running/setting up or cleaning up the external script."""
+
+
 class OpenstackBaseError(Exception):
     """Represents an error while interacting with Openstack."""
 
diff --git a/app/src/github_runner_image_builder/openstack_builder.py b/app/src/github_runner_image_builder/openstack_builder.py
index e6323bf1..036eb751 100644
--- a/app/src/github_runner_image_builder/openstack_builder.py
+++ b/app/src/github_runner_image_builder/openstack_builder.py
@@ -11,6 +11,7 @@
 import shutil
 import time
 import typing
+from collections import namedtuple
 
 import fabric
 import invoke
@@ -50,9 +51,11 @@
 
 BUILDER_KEY_PATH = pathlib.Path("/home/ubuntu/.ssh/builder_key")
 SHARED_SECURITY_GROUP_NAME = "github-runner-image-builder-v1"
+EXTERNAL_SCRIPT_PATH = pathlib.Path("/root/external.sh")
 
 CREATE_SERVER_TIMEOUT = 20 * 60  # seconds
 DELETE_SERVER_TIMEOUT = 20 * 60  # seconds
+SHUTOFF_SERVER_TIMEOUT = 10 * 60  # seconds
 
 MIN_CPU = 2
 MIN_RAM = 1024  # M
@@ -282,9 +285,17 @@ def run(
             wait=True,
         )
         logger.info("Launched builder, waiting for cloud-init to complete: %s.", builder.id)
-        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        ssh_conn = _get_ssh_connection(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_conn=ssh_conn)
         log_output = conn.get_server_console(server=builder)
-        logger.info("Build output: %s", log_output)
+        logger.info("console log after cloud-init: %s", log_output)
+        if script_url := image_config.script_config.script_url:
+            _execute_external_script(
+                script_url=script_url.geturl(),
+                script_secrets=image_config.script_config.script_secrets,
+                ssh_conn=ssh_conn,
+            )
+        _shutoff_server(conn=conn, server=builder)
         image = store.create_snapshot(
             cloud_name=cloud_config.cloud_name,
             image_name=image_config.name,
@@ -488,22 +499,6 @@ def _generate_cloud_init_script(
         HWE_VERSION=BaseImage.get_version(image_config.base),
         RUNNER_VERSION=image_config.runner_version,
         RUNNER_ARCH=image_config.arch.value,
-        SCRIPT_URL=(
-            image_config.script_config.script_url.geturl()
-            if image_config.script_config.script_url
-            else ""
-        ),
-        SCRIPT_SECRETS=(
-            " ".join(
-                f"{secret_key}={secret_value}"
-                for (
-                    secret_key,
-                    secret_value,
-                ) in image_config.script_config.script_secrets.items()
-            )
-            if image_config.script_config.script_secrets
-            else ""
-        ),
     )
 
 
@@ -530,14 +525,14 @@ def _get_builder_name(arch: Arch, base: BaseImage, prefix: str) -> str:
 def _wait_for_cloud_init_complete(
     conn: openstack.connection.Connection,
     server: openstack.compute.v2.server.Server,
-    ssh_key: pathlib.Path,
+    ssh_conn: fabric.Connection,
 ) -> bool:
     """Wait until the userdata has finished installing expected components.
 
     Args:
         conn: The Openstach connection instance.
         server: The OpenStack server instance to check if cloud_init is complete.
-        ssh_key: The key to SSH RSA key to connect to the OpenStack server instance.
+        ssh_conn: The SSH connection instance to the OpenStack server instance.
 
     Raises:
         CloudInitFailError: if there was an error running cloud-init status command.
@@ -545,11 +540,8 @@ def _wait_for_cloud_init_complete(
     Returns:
         Whether the cloud init is complete. Used for tenacity retry to pick up return value.
     """
-    ssh_connection = _get_ssh_connection(conn=conn, server=server, ssh_key=ssh_key)
     try:
-        result: fabric.Result | None = ssh_connection.run(
-            "cloud-init status --wait", timeout=60 * 30
-        )
+        result: fabric.Result | None = ssh_conn.run("cloud-init status --wait", timeout=60 * 30)
     except invoke.exceptions.UnexpectedExit as exc:
         log_out = conn.get_server_console(server=server)
         logger.error("Cloud init output: %s", log_out)
@@ -562,6 +554,79 @@ def _wait_for_cloud_init_complete(
     return "status: done" in result.stdout
 
 
+def _execute_external_script(
+    script_url: str, script_secrets: dict[str, str], ssh_conn: fabric.Connection
+) -> None:
+    """Execute the external script on the OpenStack instance.
+
+    Args:
+        script_url: The external script URL to download and execute.
+        script_secrets: The secrets to pass as environment variables to the script.
+        ssh_conn: The SSH connection instance to the OpenStack server instance.
+
+    Raises:
+        ExternalScriptError: If the external script (or setup/cleanup of it) failed to execute.
+    """
+    general_timeout_in_minutes = 2
+    script_run_timeout_in_minutes = 60
+    Command = namedtuple("Command", ["name", "command", "timeout", "env"])
+    disable_sudo_log_cmd = Command(
+        name="Disable sudo log",
+        command="echo 'Defaults !syslog' | sudo tee /etc/sudoers.d/99-no-syslog",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    script_setup_cmd = Command(
+        name="Download the external script and set permissions",
+        command=f'sudo curl "{script_url}" -o {EXTERNAL_SCRIPT_PATH} '
+        f"&& sudo chmod +x {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    script_run_cmd = Command(
+        name="Run the external script using the secrets provided as environment variables",
+        command=f"sudo --preserve-env={','.join(script_secrets.keys())} {EXTERNAL_SCRIPT_PATH}",
+        timeout=script_run_timeout_in_minutes,
+        env=script_secrets,
+    )
+    script_rm_cmd = Command(
+        name="Remove the external script",
+        command=f"sudo rm {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    enable_sudo_log_cmd = Command(
+        name="Enable sudo log",
+        command="sudo rm /etc/sudoers.d/99-no-syslog",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+
+    try:
+        for cmd in (
+            disable_sudo_log_cmd,
+            script_setup_cmd,
+            script_run_cmd,
+            script_rm_cmd,
+            enable_sudo_log_cmd,
+        ):
+            logger.info("Running command via ssh: %s", cmd.name)
+            ssh_conn.run(cmd.command, timeout=cmd.timeout * 60, warn=False, env=cmd.env)
+    except invoke.exceptions.UnexpectedExit as exc:
+        raise github_runner_image_builder.errors.ExternalScriptError(
+            f"Unexpected exit code, reason: {exc.reason}, result: {exc.result}"
+        ) from exc
+
+
+def _shutoff_server(
+    conn: openstack.connection.Connection,
+    server: openstack.compute.v2.server.Server,
+) -> None:
+    """Shut off the server instance."""
+    conn.compute.stop_server(server=server)
+    conn.compute.wait_for_server(server=server, status="SHUTOFF", wait=SHUTOFF_SERVER_TIMEOUT)
+
+
 @tenacity.retry(wait=tenacity.wait_exponential(multiplier=2, max=30), reraise=True)
 def _get_ssh_connection(
     conn: openstack.connection.Connection,
diff --git a/app/src/github_runner_image_builder/templates/cloud-init.sh.j2 b/app/src/github_runner_image_builder/templates/cloud-init.sh.j2
index 2a8f92be..75315f4d 100644
--- a/app/src/github_runner_image_builder/templates/cloud-init.sh.j2
+++ b/app/src/github_runner_image_builder/templates/cloud-init.sh.j2
@@ -115,38 +115,12 @@ function configure_system_users() {
     /usr/sbin/usermod --append --groups docker,microk8s,lxd,sudo ubuntu
 }
 
-function execute_script() {
-    local script_url="$1"
-    local env_vars="$2"
-    if [[ -z "$script_url" ]]; then
-        echo "Script URL not provided, skipping."
-        return
-    fi
-    # Write temp environment variables file, load and delete.
-    TEMP_FILE=$(mktemp)
-    IFS=' ' read -r -a vars <<< "$env_vars"
-    for var in "${vars[@]}"; do
-        echo "$var" >> "$TEMP_FILE"
-    done
-    # Source the temporary file and run the script
-    set -a  # Automatically export all variables
-    source "$TEMP_FILE"
-    rm "$TEMP_FILE"
-    set +a  # Stop automatically exporting variables
-
-    wget "$script_url" -O external.sh
-    chmod +x external.sh
-    ./external.sh
-    rm external.sh
-}
 
 proxy="{{ PROXY_URL }}"
 apt_packages="{{ APT_PACKAGES }}"
 hwe_version="{{ HWE_VERSION }}"
 github_runner_version="{{ RUNNER_VERSION }}"
 github_runner_arch="{{ RUNNER_ARCH }}"
-script_url="{{ SCRIPT_URL }}"
-script_secrets="{{ SCRIPT_SECRETS }}"
 
 configure_proxy "$proxy"
 install_apt_packages "$apt_packages" "$hwe_version"
@@ -160,8 +134,3 @@ su ubuntu -c "bash -c 'install_yq'"
 install_github_runner "$github_runner_version" "$github_runner_arch"
 chown_home
 configure_system_users
-execute_script "$script_url" "$script_secrets"
-
-# Make sure the disk is synced for snapshot
-sync
-echo "Finished sync"
diff --git a/app/tests/integration/commands.py b/app/tests/integration/commands.py
index 499e09ab..ca0051bd 100644
--- a/app/tests/integration/commands.py
+++ b/app/tests/integration/commands.py
@@ -5,6 +5,8 @@
 
 import dataclasses
 
+from tests.integration.helpers import TESTDATA_TEST_SCRIPT_URL
+
 
 @dataclasses.dataclass
 class Commands:
@@ -14,13 +16,11 @@ class Commands:
         name: The test name.
         command: The command to execute.
         env: Additional run envs.
-        external: OpenStack VM test only (Chroot unsupported test).
     """
 
     name: str
     command: str
     env: dict | None = None
-    external: bool = False
 
 
 TEST_RUNNER_COMMANDS = (
@@ -30,6 +30,10 @@ class Commands:
     Commands(
         name="file permission to /usr/local/bin (create)", command="touch /usr/local/bin/test_file"
     ),
+    Commands(
+        name="check aproxy",
+        command="sudo snap info aproxy && sudo snap services aproxy",
+    ),
     Commands(name="update apt in docker", command="docker run python:3.10-slim apt-get update"),
     Commands(name="docker version", command="docker version"),
     Commands(name="check python3 alias", command="python --version"),
@@ -44,7 +48,11 @@ class Commands:
     Commands(
         name="test sctp support", command="sudo apt-get install lksctp-tools -yq && checksctp"
     ),
-    Commands(name="test HWE kernel", command="uname -a | grep generic"),
+    Commands(
+        name="test that HWE kernel is installed",
+        command="uname -a | "
+        "grep $(dpkg -l | grep linux-generic-hwe | awk '{print $3}' | cut -d'.' -f1-3)",
+    ),
     Commands(
         name="test network congestion policy(fq)",
         command="sudo sysctl -a | grep 'net.core.default_qdisc = fq'",
@@ -56,16 +64,47 @@ class Commands:
     Commands(
         name="test external script",
         command="cat /home/ubuntu/test.txt | grep 'hello world'",
-        external=True,
     ),
     Commands(
         name="test external script secrets (should exist)",
         command='grep -q "SHOULD_EXIST" secret.txt',
-        external=True,
     ),
     Commands(
         name="test external script secrets (should not exist)",
         command='! grep -q "SHOULD_NOT_EXIST" secret.txt',
-        external=True,
+    ),
+    # following commands are security related - ensure no traces of the external script are
+    # kept in the image
+    Commands(
+        name="journal does not contain external script secrets",
+        command="! journalctl | grep 'SHOULD_EXIST'",
+    ),
+    Commands(
+        name="journal does not contain external script secrets",
+        command="! journalctl | grep 'SHOULD_NOT_EXIST'",
+    ),
+    Commands(
+        name="journal does not contain external script url",
+        command=f"! journalctl | grep '{TESTDATA_TEST_SCRIPT_URL}'",
+    ),
+    Commands(
+        name="journal does not contain script content",
+        command="! journalctl | grep '/home/ubuntu/secret.txt'",
+    ),
+    Commands(
+        name="/var/log/auth.logs does not contain external script secrets",
+        command="! grep 'SHOULD_EXIST' /var/log/auth.log*",
+    ),
+    Commands(
+        name="/var/log/auth.logs does not contain external script secrets",
+        command="! grep 'SHOULD_NOT_EXIST' /var/log/auth.log*",
+    ),
+    Commands(
+        name="/var/log/auth.logs does not contain external script url",
+        command=f"! grep '{TESTDATA_TEST_SCRIPT_URL}' /var/log/auth.log*",
+    ),
+    Commands(
+        name="/var/log/auth.logs does not contain script content",
+        command="! grep '/home/ubuntu/secret.txt' /var/log/auth.log*",
     ),
 )
diff --git a/app/tests/integration/helpers.py b/app/tests/integration/helpers.py
index c439d2f4..1aff684c 100644
--- a/app/tests/integration/helpers.py
+++ b/app/tests/integration/helpers.py
@@ -36,6 +36,13 @@
 
 logger = logging.getLogger(__name__)
 
+
+TESTDATA_TEST_SCRIPT_URL = (
+    "https://raw.githubusercontent.com/canonical/github-runner-image-builder-operator/"
+    "cc9d06c43a5feabd278265ab580eca14d5acffd4/app/tests/integration/testdata/test_script.sh"
+)
+
+
 P = ParamSpec("P")
 R = TypeVar("R")
 S = Callable[P, R] | Callable[P, Awaitable[R]]
@@ -250,7 +257,6 @@ class OpenStackConnectionParams:
 async def wait_for_valid_connection(
     connection_params: OpenStackConnectionParams,
     timeout: int = 30 * 60,
-    proxy: types.ProxyConfig | None = None,
     dockerhub_mirror: urllib.parse.ParseResult | None = None,
 ) -> SSHConnection:
     """Wait for a valid SSH connection from Openstack server.
@@ -258,7 +264,6 @@ async def wait_for_valid_connection(
     Args:
         connection_params: Parameters for connecting to OpenStack instance.
         timeout: Number of seconds to wait before raising a timeout error.
-        proxy: The proxy to configure on host runner.
         dockerhub_mirror: The DockerHub mirror URL.
 
     Raises:
@@ -291,7 +296,6 @@ async def wait_for_valid_connection(
             try:
                 result: Result = ssh_connection.run("echo 'hello world'")
                 if result.ok:
-                    await _install_proxy(conn=ssh_connection, proxy=proxy)
                     _configure_dockerhub_mirror(
                         conn=ssh_connection, dockerhub_mirror=dockerhub_mirror
                     )
@@ -302,51 +306,6 @@ async def wait_for_valid_connection(
     raise TimeoutError("No valid ssh connections found.")
 
 
-async def _install_proxy(conn: SSHConnection, proxy: types.ProxyConfig | None = None):
-    """Run commands to install proxy.
-
-    Args:
-        conn: The SSH connection instance.
-        proxy: The proxy to apply if available.
-    """
-    if not proxy or not proxy.http:
-        return
-    await wait_for(partial(_snap_ready, conn))
-
-    command = "sudo snap install aproxy --edge"
-    logger.info("Running command: %s", command)
-    result: Result = conn.run(command)
-    assert result.ok, "Failed to install aproxy"
-
-    proxy_str = proxy.http.replace("http://", "").replace("https://", "")
-    command = f"sudo snap set aproxy proxy={proxy_str}"
-    logger.info("Running command: %s", command)
-    result = conn.run(command)
-    assert result.ok, "Failed to setup aproxy"
-
-    # ignore line too long since it is better read without line breaks
-    command = """/usr/bin/sudo nft -f - << EOF
-define default-ip = $(ip route get $(ip route show 0.0.0.0/0 | grep -oP 'via \\K\\S+') | grep -oP 'src \\K\\S+')
-define private-ips = { 10.0.0.0/8, 127.0.0.1/8, 172.16.0.0/12, 192.168.0.0/16 }
-table ip aproxy
-flush table ip aproxy
-table ip aproxy {
-    chain prerouting {
-            type nat hook prerouting priority dstnat; policy accept;
-            ip daddr != \\$private-ips tcp dport { 80, 443 } counter dnat to \\$default-ip:8443
-    }
-
-    chain output {
-            type nat hook output priority -100; policy accept;
-            ip daddr != \\$private-ips tcp dport { 80, 443 } counter dnat to \\$default-ip:8443
-    }
-}
-EOF"""  # noqa: E501
-    logger.info("Running command: %s", command)
-    result = conn.run(command)
-    assert result.ok, "Failed to configure iptable rules"
-
-
 def _snap_ready(conn: SSHConnection) -> bool:
     """Checks whether snapd is ready.
 
@@ -415,19 +374,13 @@ def format_dockerhub_mirror_microk8s_command(
     )
 
 
-def run_openstack_tests(
-    ssh_connection: SSHConnection,
-    external: bool = False,
-):
+def run_openstack_tests(ssh_connection: SSHConnection):
     """Run test commands on the openstack instance via ssh.
 
     Args:
         ssh_connection: The SSH connection instance to OpenStack test server.
-        external: Whether the test is for external VM builder image test.
     """
     for testcmd in commands.TEST_RUNNER_COMMANDS:
-        if not external and testcmd.external:
-            continue
         logger.info("Running command: %s", testcmd.command)
         result: Result = ssh_connection.run(testcmd.command, env=testcmd.env)
         logger.info("Command output: %s %s %s", result.return_code, result.stdout, result.stderr)
diff --git a/app/tests/integration/test_openstack_builder.py b/app/tests/integration/test_openstack_builder.py
index 87caf065..a7e856f5 100644
--- a/app/tests/integration/test_openstack_builder.py
+++ b/app/tests/integration/test_openstack_builder.py
@@ -16,12 +16,14 @@
 import pytest
 import pytest_asyncio
 from fabric.connection import Connection as SSHConnection
+from integration.helpers import TESTDATA_TEST_SCRIPT_URL
 from openstack.compute.v2.image import Image
 from openstack.compute.v2.server import Server
 from openstack.connection import Connection
 from openstack.network.v2.security_group import SecurityGroup
 
 from github_runner_image_builder import config, openstack_builder
+from github_runner_image_builder.openstack_builder import CREATE_SERVER_TIMEOUT
 from tests.integration import helpers, types
 
 logger = logging.getLogger(__name__)
@@ -87,72 +89,81 @@ def image_ids_fixture(
     openstack_metadata: types.OpenstackMeta,
     test_id: str,
     proxy: types.ProxyConfig,
-) -> list[str]:
+) -> typing.Iterator[list[str]]:
     """A CLI run.
 
     This fixture assumes pipx is installed in the system and the github-runner-image-builder has
     been installed using pipx. See testenv:integration section of tox.ini.
     """
-    image_ids = openstack_builder.run(
-        cloud_config=openstack_builder.CloudConfig(
-            cloud_name=openstack_metadata.cloud_name,
-            flavor=openstack_metadata.flavor,
-            network=openstack_metadata.network,
-            proxy=proxy.http,
-            prefix=test_id,
-            upload_cloud_names=[openstack_metadata.cloud_name],
-        ),
-        image_config=config.ImageConfig(
-            arch=image_config.arch,
-            base=config.BaseImage.from_str(image_config.image),
-            runner_version="",
-            name=f"{test_id}-image-builder-test",
-            script_config=config.ScriptConfig(
-                script_url=urllib.parse.urlparse(
-                    "https://raw.githubusercontent.com/canonical/github-runner-image-builder/"
-                    "eb0ca315bf8c7aa732b811120cbabca4b8d16216/tests/integration/testdata/"
-                    "test_script.sh"
+    try:
+        image_ids = openstack_builder.run(
+            cloud_config=openstack_builder.CloudConfig(
+                cloud_name=openstack_metadata.cloud_name,
+                flavor=openstack_metadata.flavor,
+                network=openstack_metadata.network,
+                proxy=proxy.http,
+                prefix=test_id,
+                upload_cloud_names=[openstack_metadata.cloud_name],
+            ),
+            image_config=config.ImageConfig(
+                arch=image_config.arch,
+                base=config.BaseImage.from_str(image_config.image),
+                runner_version="",
+                name=f"{test_id}-image-builder-test",
+                script_config=config.ScriptConfig(
+                    script_url=urllib.parse.urlparse(TESTDATA_TEST_SCRIPT_URL),
+                    script_secrets={
+                        "TEST_SECRET": "SHOULD_EXIST",
+                        "TEST_NON_SECRET": "SHOULD_NOT_EXIST",
+                    },
                 ),
-                script_secrets={
-                    "TEST_SECRET": "SHOULD_EXIST",
-                    "TEST_NON_SECRET": "SHOULD_NOT_EXIST",
-                },
             ),
-        ),
-        keep_revisions=1,
-    )
-    return image_ids.split(",")
+            keep_revisions=1,
+        )
+        yield image_ids.split(",")
+
+    finally:
+        # cleanup resources
+        openstack_metadata.connection.delete_server(
+            name_or_id=openstack_builder._get_builder_name(
+                arch=image_config.arch, base=config.BaseImage(image_config.image), prefix=test_id
+            )
+        )
+        openstack_metadata.connection.delete_keypair(
+            name=openstack_builder._get_keypair_name(prefix=test_id)
+        )
 
 
 @pytest.fixture(scope="module", name="make_dangling_resources")
-async def make_dangling_resources_fixture(
+def make_dangling_resources_fixture(
     openstack_metadata: types.OpenstackMeta, test_id: str, image_config: types.ImageConfig
 ):
     """Make OpenStack resources that imitates failed run."""
-    keypair = openstack_metadata.connection.create_keypair(
-        openstack_builder._get_keypair_name(prefix=test_id)
-    )
-    server = openstack_metadata.connection.create_server(
-        name=openstack_builder._get_builder_name(
-            arch=image_config.arch, base=config.BaseImage(image_config.image), prefix=test_id
-        ),
-        image=f"image-builder-base-jammy-{image_config.arch.value}",
-        flavor=openstack_metadata.flavor,
-        network=openstack_metadata.network,
-        security_groups=[openstack_builder.SHARED_SECURITY_GROUP_NAME],
-        wait=True,
-    )
-
-    yield
+    server = None
+    try:
+        server = openstack_metadata.connection.create_server(
+            name=openstack_builder._get_builder_name(
+                arch=image_config.arch, base=config.BaseImage(image_config.image), prefix=test_id
+            ),
+            image=f"image-builder-base-jammy-{image_config.arch.value}",
+            flavor=openstack_metadata.flavor,
+            network=openstack_metadata.network,
+            security_groups=[openstack_builder.SHARED_SECURITY_GROUP_NAME],
+            auto_ip=False,
+            wait=True,
+            timeout=CREATE_SERVER_TIMEOUT,
+        )
 
-    openstack_metadata.connection.delete_keypair(name=keypair.name)
-    openstack_metadata.connection.delete_server(name_or_id=server.id)
+        yield
+    finally:
+        if server:
+            openstack_metadata.connection.delete_server(name_or_id=server.id)
 
 
 # the code is similar but the fixture source is localized and is different.
 # pylint: disable=R0801
-@pytest_asyncio.fixture(scope="module", name="openstack_server")
-async def openstack_server_fixture(
+@pytest.fixture(scope="module", name="openstack_server")
+def openstack_server_fixture(
     openstack_metadata: types.OpenstackMeta,
     openstack_security_group: SecurityGroup,
     test_id: str,
@@ -161,21 +172,20 @@ async def openstack_server_fixture(
     """A testing openstack instance."""
     image: Image = openstack_metadata.connection.get_image(name_or_id=image_ids[0])
     server_name = f"test-image-builder-run-{test_id}"
-    for server in helpers.create_openstack_server(
+    yield from helpers.create_openstack_server(
         openstack_metadata=openstack_metadata,
         server_name=server_name,
         image=image,
         security_group=openstack_security_group,
-    ):
-        yield server
+    )
     openstack_metadata.connection.delete_image(image.id)
 
 
 @pytest_asyncio.fixture(scope="module", name="ssh_connection")
 async def ssh_connection_fixture(
     openstack_server: Server,
-    proxy: types.ProxyConfig,
     openstack_metadata: types.OpenstackMeta,
+    dockerhub_mirror: urllib.parse.ParseResult | None,
 ) -> SSHConnection:
     """The openstack server ssh connection fixture."""
     logger.info("Setting up SSH connection.")
@@ -186,19 +196,16 @@ async def ssh_connection_fixture(
             network=openstack_metadata.network,
             ssh_key=openstack_metadata.ssh_key.private_key,
         ),
-        proxy=proxy,
+        dockerhub_mirror=dockerhub_mirror,
     )
 
     return ssh_connection
 
 
-# pylint: enable=R0801
-
-
 @pytest.mark.amd64
 @pytest.mark.arm64
 @pytest.mark.usefixtures("make_dangling_resources")
-async def test_run(
+def test_run(
     ssh_connection: SSHConnection,
 ):
     """
@@ -206,12 +213,12 @@ async def test_run(
     act: when run (build image) is called.
     assert: an image snapshot of working VM is created with the ability to run expected commands.
     """
-    helpers.run_openstack_tests(ssh_connection=ssh_connection, external=True)
+    helpers.run_openstack_tests(ssh_connection=ssh_connection)
 
 
 @pytest.mark.amd64
 @pytest.mark.arm64
-async def test_openstack_state(
+def test_openstack_state(
     openstack_metadata: types.OpenstackMeta, test_id: str, image_config: types.ImageConfig
 ):
     """
@@ -219,7 +226,7 @@ async def test_openstack_state(
     act: None.
     assert: Dangling resources are cleaned up.
 
-    This test is dependent on the previons test_run test. Running a new test image building run
+    This test is dependent on the previous test_run test. Running a new test image building run
     is too costly at the moment.
     """
     server = openstack_metadata.connection.get_server(
diff --git a/app/tests/unit/test_openstack_builder.py b/app/tests/unit/test_openstack_builder.py
index 9993efd3..2285a485 100644
--- a/app/tests/unit/test_openstack_builder.py
+++ b/app/tests/unit/test_openstack_builder.py
@@ -20,6 +20,8 @@
 import yaml
 
 from github_runner_image_builder import cloud_image, errors, openstack_builder, store
+from github_runner_image_builder.errors import ExternalScriptError
+from github_runner_image_builder.openstack_builder import EXTERNAL_SCRIPT_PATH
 
 
 def test_determine_cloud_no_clouds_yaml_error(monkeypatch: pytest.MonkeyPatch):
@@ -171,7 +173,7 @@ def test__create_security_group():
 
 
 @pytest.mark.parametrize(
-    "cloud_config",
+    "cloud_config, with_external_script",
     [
         pytest.param(
             openstack_builder.CloudConfig(
@@ -182,6 +184,7 @@ def test__create_security_group():
                 proxy="test-proxy",
                 upload_cloud_names=[],
             ),
+            True,
             id="no upload-cloud-name",
         ),
         pytest.param(
@@ -193,6 +196,7 @@ def test__create_security_group():
                 proxy="test-proxy",
                 upload_cloud_names=["test-cloud-1"],
             ),
+            True,
             id="single upload-cloud-name defined",
         ),
         pytest.param(
@@ -204,16 +208,48 @@ def test__create_security_group():
                 proxy="test-proxy",
                 upload_cloud_names=["test-cloud-1", "test-cloud-2"],
             ),
+            True,
             id="multiple upload-cloud-name defined",
         ),
+        pytest.param(
+            openstack_builder.CloudConfig(
+                cloud_name="test-cloud",
+                flavor="test-flavor",
+                network="test-network",
+                prefix="",
+                proxy="test-proxy",
+                upload_cloud_names=["test-cloud-1"],
+            ),
+            False,
+            id="without external script",
+        ),
     ],
 )
-def test_run(monkeypatch: pytest.MonkeyPatch, cloud_config: openstack_builder.CloudConfig):
+def test_run(
+    monkeypatch: pytest.MonkeyPatch,
+    cloud_config: openstack_builder.CloudConfig,
+    with_external_script: bool,
+):
     """
     arrange: given monkeypatched sub functions for openstack_builder.run.
     act: when run is called.
     assert: all subfunctions are called.
     """
+    image_config = openstack_builder.config.ImageConfig(
+        arch=openstack_builder.Arch.X64,
+        base=openstack_builder.BaseImage.JAMMY,
+        runner_version="",
+        name="test-image",
+        script_config=openstack_builder.config.ScriptConfig(
+            script_url=(
+                urllib.parse.urlparse("https://test-url.com/script.sh")
+                if with_external_script
+                else None
+            ),
+            script_secrets={"TEST_SECRET_ONE": "HELLO"} if with_external_script else {},
+        ),
+    )
+
     monkeypatch.setattr(
         openstack_builder, "_generate_cloud_init_script", (generate_cloud_init_mock := MagicMock())
     )
@@ -235,16 +271,26 @@ def test_run(monkeypatch: pytest.MonkeyPatch, cloud_config: openstack_builder.Cl
         "connect",
         MagicMock(return_value=connection_enter_mock),
     )
+    monkeypatch.setattr(
+        openstack_builder,
+        "_get_ssh_connection",
+        (get_ssh_connection_mock := MagicMock(return_value=MagicMock())),
+    )
     monkeypatch.setattr(
         openstack_builder, "_wait_for_cloud_init_complete", (wait_cloud_init_mock := MagicMock())
     )
+    monkeypatch.setattr(
+        openstack_builder,
+        "_execute_external_script",
+        (execute_external_script_mock := MagicMock()),
+    )
     monkeypatch.setattr(
         openstack_builder, "_wait_for_snapshot_complete", (wait_snapshot_mock := MagicMock())
     )
 
     openstack_builder.run(
         cloud_config=cloud_config,
-        image_config=MagicMock(),
+        image_config=image_config,
         keep_revisions=5,
     )
 
@@ -252,7 +298,11 @@ def test_run(monkeypatch: pytest.MonkeyPatch, cloud_config: openstack_builder.Cl
     ensure_resources_mock.assert_called()
     determine_flavor_mock.assert_called()
     determine_network_mock.assert_called()
+    get_ssh_connection_mock.assert_called()
     wait_cloud_init_mock.assert_called()
+    assert execute_external_script_mock.call_count == (1 if with_external_script else 0)
+    connection_mock.compute.stop_server.assert_called()
+    connection_mock.compute.wait_for_server.assert_called()
     wait_snapshot_mock.assert_called()
     create_image_snapshot.assert_called()
     connection_mock.create_server.assert_called()
@@ -713,30 +763,6 @@ def test__generate_cloud_init_script():
     /usr/sbin/usermod --append --groups docker,microk8s,lxd,sudo ubuntu
 }
 
-function execute_script() {
-    local script_url="$1"
-    local env_vars="$2"
-    if [[ -z "$script_url" ]]; then
-        echo "Script URL not provided, skipping."
-        return
-    fi
-    # Write temp environment variables file, load and delete.
-    TEMP_FILE=$(mktemp)
-    IFS=' ' read -r -a vars <<< "$env_vars"
-    for var in "${vars[@]}"; do
-        echo "$var" >> "$TEMP_FILE"
-    done
-    # Source the temporary file and run the script
-    set -a  # Automatically export all variables
-    source "$TEMP_FILE"
-    rm "$TEMP_FILE"
-    set +a  # Stop automatically exporting variables
-
-    wget "$script_url" -O external.sh
-    chmod +x external.sh
-    ./external.sh
-    rm external.sh
-}
 
 proxy="test.proxy.internal:3128"
 apt_packages="build-essential docker.io gh jq npm python3-dev python3-pip python-is-python3 \
@@ -744,8 +770,6 @@ def test__generate_cloud_init_script():
 hwe_version="22.04"
 github_runner_version=""
 github_runner_arch="x64"
-script_url="https://test-url.com/script.sh"
-script_secrets="TEST_SECRET_ONE=HELLO TEST_SECRET_TWO=WORLD"
 
 configure_proxy "$proxy"
 install_apt_packages "$apt_packages" "$hwe_version"
@@ -758,39 +782,31 @@ def test__generate_cloud_init_script():
 su ubuntu -c "bash -c 'install_yq'"
 install_github_runner "$github_runner_version" "$github_runner_arch"
 chown_home
-configure_system_users
-execute_script "$script_url" "$script_secrets"
-
-# Make sure the disk is synced for snapshot
-sync
-echo "Finished sync"\
+configure_system_users\
 """
     )
     # pylint: enable=R0801
 
 
-def test__wait_for_cloud_init_complete_fail(monkeypatch: pytest.MonkeyPatch):
+def test__wait_for_cloud_init_complete_fail():
     """
     arrange: given a monkeypatched _get_ssh_connection and connection.run functions that raises an\
         error.
     act: when _wait_for_cloud_init_complete is called.
     assert: CloudInitFailError is raised.
     """
-    mock_connection = MagicMock()
-    mock_connection.run.return_value = None
-    monkeypatch.setattr(
-        openstack_builder, "_get_ssh_connection", MagicMock(return_value=mock_connection)
-    )
+    ssh_conn_mock = MagicMock()
+    ssh_conn_mock.run.return_value = None
 
     with pytest.raises(errors.CloudInitFailError) as exc:
         openstack_builder._wait_for_cloud_init_complete(
-            conn=mock_connection, server=MagicMock(), ssh_key=MagicMock()
+            conn=MagicMock(), server=MagicMock(), ssh_conn=ssh_conn_mock
         )
 
     assert "Invalid cloud-init status" in str(exc)
 
 
-def test__wait_for_cloud_init_unexpected_exit(monkeypatch: pytest.MonkeyPatch):
+def test__wait_for_cloud_init_unexpected_exit():
     """
     arrange: given a monkeypatched _get_ssh_connection and connection.run functions that raises an\
         error.
@@ -799,22 +815,19 @@ def test__wait_for_cloud_init_unexpected_exit(monkeypatch: pytest.MonkeyPatch):
     """
     mock_connection = MagicMock()
     mock_connection.get_server_console = (get_log_mock := MagicMock())
-    mock_connection.run.side_effect = openstack_builder.invoke.exceptions.UnexpectedExit(
+    ssh_conn_mock = MagicMock()
+    ssh_conn_mock.run.side_effect = openstack_builder.invoke.exceptions.UnexpectedExit(
         result=MagicMock(), reason=MagicMock()
     )
-    monkeypatch.setattr(
-        openstack_builder, "_get_ssh_connection", MagicMock(return_value=mock_connection)
-    )
 
     with pytest.raises(errors.CloudInitFailError):
         openstack_builder._wait_for_cloud_init_complete(
-            conn=mock_connection, server=MagicMock(), ssh_key=MagicMock()
+            conn=mock_connection, server=MagicMock(), ssh_conn=ssh_conn_mock
         )
-
     get_log_mock.assert_called_once()
 
 
-def test__wait_for_cloud_init_complete(monkeypatch: pytest.MonkeyPatch):
+def test__wait_for_cloud_init_complete():
     """
     arrange: given a monkeypatched _get_ssh_connection and connection.run functions.
     act: when _wait_for_cloud_init_complete is called.
@@ -823,18 +836,87 @@ def test__wait_for_cloud_init_complete(monkeypatch: pytest.MonkeyPatch):
     # patch tenacity retry to speed up testing
     openstack_builder._wait_for_cloud_init_complete.retry.wait = tenacity.wait_none()
     openstack_builder._wait_for_cloud_init_complete.retry.stop = tenacity.stop_after_attempt(1)
-    mock_connection = MagicMock()
+    ssh_conn_mock = MagicMock()
     result_mock = MagicMock()
     result_mock.stdout = "status: done"
-    mock_connection.run.return_value = result_mock
-    monkeypatch.setattr(
-        openstack_builder, "_get_ssh_connection", MagicMock(return_value=mock_connection)
-    )
+    ssh_conn_mock.run.return_value = result_mock
 
     assert openstack_builder._wait_for_cloud_init_complete(
-        conn=mock_connection, server=MagicMock(), ssh_key=MagicMock()
+        conn=MagicMock(), server=MagicMock(), ssh_conn=ssh_conn_mock
+    )
+
+
+@pytest.mark.parametrize(
+    "script_secrets",
+    [
+        pytest.param({"TEST_SECRET_ONE": "HELLO"}, id="single secret"),
+        pytest.param(
+            {"TEST_SECRET_ONE": "HELLO", "TEST_SECRET_TWO": "WORLD"}, id="multiple secrets"
+        ),
+        pytest.param({}, id="no secrets"),
+    ],
+)
+def test__execute_external_script(script_secrets: dict[str, str]):
+    """
+    arrange: given a monkeypatched _get_ssh_connection and connection.run functions.
+    act: when _execute_external_script is called.
+    assert: that the external script is executed using the secrets
+    """
+    mock_connection = MagicMock()
+    run_mock = MagicMock()
+    mock_connection.run = run_mock
+
+    openstack_builder._execute_external_script(
+        script_url="https://test-url.com/script.sh",
+        script_secrets=script_secrets,
+        ssh_conn=mock_connection,
     )
 
+    run_mock.assert_any_call(
+        f"sudo --preserve-env={','.join(script_secrets.keys())} {EXTERNAL_SCRIPT_PATH}",
+        env=script_secrets,
+        timeout=3600,
+        warn=False,
+    )
+    run_mock.assert_any_call(f"sudo rm {EXTERNAL_SCRIPT_PATH}", timeout=120, warn=False, env={})
+
+
+@pytest.mark.parametrize("run_pos", [pytest.param(i, id=str(i)) for i in range(5)])
+def test_execute_external_script_error(run_pos: int):
+    """
+    arrange: given a monkeypatched _get_ssh_connection and connection.run functions that raises an\
+        error on a given call on a certain position.
+    act: when _execute_external_script is called.
+    assert: ExternalScriptError is raised.
+    """
+    mock_connection = MagicMock()
+
+    call_count = 0
+
+    def run_side_effect(*_, **__):
+        """Raise an error on a given call position.
+
+        Raises:
+            UnexpectedExit: Raised on a given call position.
+        """
+        nonlocal call_count
+        if call_count == run_pos:
+            raise openstack_builder.invoke.exceptions.UnexpectedExit(
+                result=MagicMock(), reason=MagicMock()
+            )
+
+        call_count += 1
+
+    mock_connection.run.side_effect = run_side_effect
+
+    with pytest.raises(ExternalScriptError) as exc:
+        openstack_builder._execute_external_script(
+            script_url="https://test-url.com/script.sh",
+            script_secrets={"TEST_SECRET_ONE": "HELLO"},
+            ssh_conn=mock_connection,
+        )
+    assert "Unexpected exit code" in str(exc)
+
 
 def test__get_ssh_connection_no_networks():
     """


Reviews:
user: login: cbartz
body: self-review
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-04T14:21:21Z
user: login: javierdelapuente
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-04T15:16:11Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T08:21:11Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T08:22:52Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:12:12Z
user: login: javierdelapuente
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:16:33Z
user: login: javierdelapuente
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:17:02Z
user: login: javierdelapuente
body: LGTM. I would remove the tmate, as it may block runners. 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:18:01Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:18:32Z
user: login: yhaliaw
body: Is there a test for testing whether content of script_secrets are being leaked or not?
Maybe I missed the test for this?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T06:11:26Z
user: login: yhaliaw
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T06:14:24Z
user: login: yanksyoon
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T06:24:55Z
user: login: yanksyoon
body: Thanks for fixing it!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T06:25:52Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T07:45:24Z
user: login: javierdelapuente
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T12:28:15Z

Review comments:
diff_hunk: @@ -302,51 +306,6 @@ async def wait_for_valid_connection(
     raise TimeoutError("No valid ssh connections found.")
 
 
-async def _install_proxy(conn: SSHConnection, proxy: types.ProxyConfig | None = None):
path: app/tests/integration/helpers.py
user: login: cbartz
body: this is not required and wasn't even working
created_at: 2025-03-04T14:20:05Z
author_association: COLLABORATOR
diff_hunk: @@ -186,40 +196,37 @@ async def ssh_connection_fixture(
             network=openstack_metadata.network,
             ssh_key=openstack_metadata.ssh_key.private_key,
         ),
-        proxy=proxy,
+        dockerhub_mirror=dockerhub_mirror,
     )
 
     return ssh_connection
 
 
-# pylint: enable=R0801
-
-
 @pytest.mark.amd64
 @pytest.mark.arm64
 @pytest.mark.usefixtures("make_dangling_resources")
-async def test_run(
path: app/tests/integration/test_openstack_builder.py
user: login: cbartz
body: async was wrong, this was never executed
created_at: 2025-03-04T14:20:44Z
author_association: COLLABORATOR
diff_hunk: @@ -30,6 +30,10 @@ jobs:
         run: |
          cd app
          tox -e integration -- -m arm64 --image=${{ matrix.image }} ${{ secrets.INTEGRATION_TEST_ARGS_APP }}
+      - name: Tmate debugging session
path: .github/workflows/integration_test_app.yaml
user: login: javierdelapuente
body: I think this should be deleted
created_at: 2025-03-04T15:08:44Z
author_association: COLLABORATOR
diff_hunk: @@ -562,6 +552,78 @@ def _wait_for_cloud_init_complete(
     return "status: done" in result.stdout
 
 
+def _execute_external_script(
+    script_url: str, script_secrets: dict[str, str], ssh_conn: fabric.Connection
+) -> None:
+    """Execute the external script on the OpenStack instance.
+
+    Args:
+        script_url: The external script URL to download and execute.
+        script_secrets: The secrets to pass as environment variables to the script.
+        ssh_conn: The SSH connection instance to the OpenStack server instance.
+
+    Raises:
+        ExternalScriptError: If the external script (or setup/cleanup of it) failed to execute.
+    """
+    general_timeout_in_minutes = 2
+    script_run_timeout_in_minutes = 60
+    Command = namedtuple("Command", ["name", "command", "timeout", "env"])
+    script_setup_cmd = Command(
+        name="Download the external script and set permissions",
+        command=f'sudo curl "{script_url}" -o {EXTERNAL_SCRIPT_PATH} '
+        f"&& sudo chmod +x {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    script_run_cmd = Command(
+        name="Run the external script using the secrets provided as environment variables",
+        command=f"sudo --preserve-env={','.join(script_secrets.keys())} {EXTERNAL_SCRIPT_PATH}",
+        timeout=script_run_timeout_in_minutes,
+        env=script_secrets,
+    )
+    script_rm_cmd = Command(
+        name="Remove the external script",
+        command=f"sudo rm {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_journal_cmd = Command(
+        name="Clear the journal to remove script traces",
+        command="sudo journalctl --flush && sudo journalctl --rotate && "
+        "sudo journalctl --merge --vacuum-size=1",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_auth_logs_cmd = Command(
+        name="Clear the auth logs to remove script traces",
+        command="cat /dev/null | sudo tee /var/log/auth.log",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    sync_cmd = Command(
+        name="Sync the disk to ensure data is written for the snapshot",
+        command="sudo sync",
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: javierdelapuente
body: Maybe we could do this in every case, not just when there is a script? Or maybe just shutdown the instance before the snapshot (and maybe fix two things in one go)?
created_at: 2025-03-04T15:14:17Z
author_association: COLLABORATOR
diff_hunk: @@ -30,6 +30,10 @@ jobs:
         run: |
          cd app
          tox -e integration -- -m arm64 --image=${{ matrix.image }} ${{ secrets.INTEGRATION_TEST_ARGS_APP }}
+      - name: Tmate debugging session
path: .github/workflows/integration_test_app.yaml
user: login: cbartz
body: The idea was to keep it (note that it only runs when someone re-runs it with "debug enabled"). Although I have to admit that tmate is not that useful in application integration testing, as there is no juju model or charm deployed locally. So I am open to removing it.
created_at: 2025-03-05T08:21:11Z
author_association: COLLABORATOR
diff_hunk: @@ -562,6 +552,78 @@ def _wait_for_cloud_init_complete(
     return "status: done" in result.stdout
 
 
+def _execute_external_script(
+    script_url: str, script_secrets: dict[str, str], ssh_conn: fabric.Connection
+) -> None:
+    """Execute the external script on the OpenStack instance.
+
+    Args:
+        script_url: The external script URL to download and execute.
+        script_secrets: The secrets to pass as environment variables to the script.
+        ssh_conn: The SSH connection instance to the OpenStack server instance.
+
+    Raises:
+        ExternalScriptError: If the external script (or setup/cleanup of it) failed to execute.
+    """
+    general_timeout_in_minutes = 2
+    script_run_timeout_in_minutes = 60
+    Command = namedtuple("Command", ["name", "command", "timeout", "env"])
+    script_setup_cmd = Command(
+        name="Download the external script and set permissions",
+        command=f'sudo curl "{script_url}" -o {EXTERNAL_SCRIPT_PATH} '
+        f"&& sudo chmod +x {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    script_run_cmd = Command(
+        name="Run the external script using the secrets provided as environment variables",
+        command=f"sudo --preserve-env={','.join(script_secrets.keys())} {EXTERNAL_SCRIPT_PATH}",
+        timeout=script_run_timeout_in_minutes,
+        env=script_secrets,
+    )
+    script_rm_cmd = Command(
+        name="Remove the external script",
+        command=f"sudo rm {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_journal_cmd = Command(
+        name="Clear the journal to remove script traces",
+        command="sudo journalctl --flush && sudo journalctl --rotate && "
+        "sudo journalctl --merge --vacuum-size=1",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_auth_logs_cmd = Command(
+        name="Clear the auth logs to remove script traces",
+        command="cat /dev/null | sudo tee /var/log/auth.log",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    sync_cmd = Command(
+        name="Sync the disk to ensure data is written for the snapshot",
+        command="sudo sync",
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: cbartz
body: So we also do sync in cloud-init, so technically its not required, but I can move it out of cloud-init and do it always using ssh.

I'll investigate the shutdown (I assume you're referring to the HWE kernel issue you want to fix).
created_at: 2025-03-05T08:22:52Z
author_association: COLLABORATOR
diff_hunk: @@ -562,6 +552,78 @@ def _wait_for_cloud_init_complete(
     return "status: done" in result.stdout
 
 
+def _execute_external_script(
+    script_url: str, script_secrets: dict[str, str], ssh_conn: fabric.Connection
+) -> None:
+    """Execute the external script on the OpenStack instance.
+
+    Args:
+        script_url: The external script URL to download and execute.
+        script_secrets: The secrets to pass as environment variables to the script.
+        ssh_conn: The SSH connection instance to the OpenStack server instance.
+
+    Raises:
+        ExternalScriptError: If the external script (or setup/cleanup of it) failed to execute.
+    """
+    general_timeout_in_minutes = 2
+    script_run_timeout_in_minutes = 60
+    Command = namedtuple("Command", ["name", "command", "timeout", "env"])
+    script_setup_cmd = Command(
+        name="Download the external script and set permissions",
+        command=f'sudo curl "{script_url}" -o {EXTERNAL_SCRIPT_PATH} '
+        f"&& sudo chmod +x {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    script_run_cmd = Command(
+        name="Run the external script using the secrets provided as environment variables",
+        command=f"sudo --preserve-env={','.join(script_secrets.keys())} {EXTERNAL_SCRIPT_PATH}",
+        timeout=script_run_timeout_in_minutes,
+        env=script_secrets,
+    )
+    script_rm_cmd = Command(
+        name="Remove the external script",
+        command=f"sudo rm {EXTERNAL_SCRIPT_PATH}",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_journal_cmd = Command(
+        name="Clear the journal to remove script traces",
+        command="sudo journalctl --flush && sudo journalctl --rotate && "
+        "sudo journalctl --merge --vacuum-size=1",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    clear_auth_logs_cmd = Command(
+        name="Clear the auth logs to remove script traces",
+        command="cat /dev/null | sudo tee /var/log/auth.log",
+        timeout=general_timeout_in_minutes,
+        env={},
+    )
+    sync_cmd = Command(
+        name="Sync the disk to ensure data is written for the snapshot",
+        command="sudo sync",
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: cbartz
body: Thanks. I've removed the sync. I did also add a more specific test for the hwe kernel.
created_at: 2025-03-05T14:12:12Z
author_association: COLLABORATOR
diff_hunk: @@ -30,6 +30,10 @@ jobs:
         run: |
          cd app
          tox -e integration -- -m arm64 --image=${{ matrix.image }} ${{ secrets.INTEGRATION_TEST_ARGS_APP }}
+      - name: Tmate debugging session
path: .github/workflows/integration_test_app.yaml
user: login: javierdelapuente
body: I would remove them. The main issue, is that if a test fails, it will block the runner for another 60 min (and with out flaky infra, that is not good). If anyone wants to use tmate, it can be added in a PR.
created_at: 2025-03-05T14:16:33Z
author_association: COLLABORATOR
diff_hunk: @@ -282,9 +285,17 @@ def run(
             wait=True,
         )
         logger.info("Launched builder, waiting for cloud-init to complete: %s.", builder.id)
-        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        ssh_conn = _get_ssh_connection(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_conn=ssh_conn)
         log_output = conn.get_server_console(server=builder)
-        logger.info("Build output: %s", log_output)
+        logger.info("console log after cloud-init: %s", log_output)
+        if script_url := image_config.script_config.script_url:
+            _execute_external_script(
+                script_url=script_url.geturl(),
+                script_secrets=image_config.script_config.script_secrets,
+                ssh_conn=ssh_conn,
+            )
+        _shutoff_server(conn=conn, server=builder)
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: javierdelapuente
body: :heart: 
created_at: 2025-03-05T14:17:01Z
author_association: COLLABORATOR
diff_hunk: @@ -30,6 +30,10 @@ jobs:
         run: |
          cd app
          tox -e integration -- -m arm64 --image=${{ matrix.image }} ${{ secrets.INTEGRATION_TEST_ARGS_APP }}
+      - name: Tmate debugging session
path: .github/workflows/integration_test_app.yaml
user: login: cbartz
body: Done
created_at: 2025-03-05T14:18:32Z
author_association: COLLABORATOR
diff_hunk: @@ -282,9 +285,17 @@ def run(
             wait=True,
         )
         logger.info("Launched builder, waiting for cloud-init to complete: %s.", builder.id)
-        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        ssh_conn = _get_ssh_connection(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_conn=ssh_conn)
         log_output = conn.get_server_console(server=builder)
-        logger.info("Build output: %s", log_output)
+        logger.info("console log after cloud-init: %s", log_output)
+        if script_url := image_config.script_config.script_url:
+            _execute_external_script(
+                script_url=script_url.geturl(),
+                script_secrets=image_config.script_config.script_secrets,
+                ssh_conn=ssh_conn,
+            )
+        _shutoff_server(conn=conn, server=builder)
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: yanksyoon
body: Out of curiosity, why are we shutting off the server?
created_at: 2025-03-06T06:24:55Z
author_association: MEMBER
diff_hunk: @@ -30,6 +30,10 @@ jobs:
         run: |
          cd app
          tox -e integration -- -m arm64 --image=${{ matrix.image }} ${{ secrets.INTEGRATION_TEST_ARGS_APP }}
+      - name: Tmate debugging session
path: .github/workflows/integration_test_app.yaml
user: login: cbartz
body: > The main issue, is that if a test fails, it will block the runner for another 60 min (and with out flaky infra, that is not good).

Well, tmate is only used when a user explicitly clicks "re-run with debug", so this is not done on every failure. Still, I removed it because, as mentioned above, tmate is not that useful for app integration testing in general.

created_at: 2025-03-06T07:45:24Z
author_association: COLLABORATOR
diff_hunk: @@ -282,9 +285,17 @@ def run(
             wait=True,
         )
         logger.info("Launched builder, waiting for cloud-init to complete: %s.", builder.id)
-        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        ssh_conn = _get_ssh_connection(conn=conn, server=builder, ssh_key=BUILDER_KEY_PATH)
+        _wait_for_cloud_init_complete(conn=conn, server=builder, ssh_conn=ssh_conn)
         log_output = conn.get_server_console(server=builder)
-        logger.info("Build output: %s", log_output)
+        logger.info("console log after cloud-init: %s", log_output)
+        if script_url := image_config.script_config.script_url:
+            _execute_external_script(
+                script_url=script_url.geturl(),
+                script_secrets=image_config.script_config.script_secrets,
+                ssh_conn=ssh_conn,
+            )
+        _shutoff_server(conn=conn, server=builder)
path: app/src/github_runner_image_builder/openstack_builder.py
user: login: javierdelapuente
body: We are cloning a block device (snapshotting)  while the server is running, which may generate issues. This bug is fixed by shutting down the machine: https://warthogs.atlassian.net/browse/ISD-3026
created_at: 2025-03-06T12:28:15Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T14:20:49Z
body: ## Test results for commit 5fe4946ab17364581b05a489978fa5f40fdf5d1a

Test coverage for 5fe4946ab17364581b05a489978fa5f40fdf5d1a

  ```
Name                 Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------
src/builder.py         256      0     34      0   100%
src/charm.py            79      0      4      0   100%
src/charm_utils.py      24      0      2      0   100%
src/exceptions.py        8      0      0      0   100%
src/image.py            58      0     18      0   100%
src/pipx.py              5      0      0      0   100%
src/proxy.py            21      0      4      0   100%
src/state.py           273      0     54      0   100%
----------------------------------------------------------------
TOTAL                  724      0    116      0   100%
```
  Static code analysis report

  ```
Run started:2025-03-05 14:19:57.209292

Test results:
	No issues identified.

Code scanned:
	Total lines of code: 4757
	Total lines skipped (#nosec): 4
	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 16

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
Files skipped (0):
```
author_association: CONTRIBUTOR
user: login: cbartz
created_at: 2025-03-06T06:12:38Z
body: > Is there a test for testing whether content of script_secrets are being leaked or not? Maybe I missed the test for this?

@yhaliaw yes, in `commands.py` in the integration tests, I check the journal and audit logs (this are the places I have seen leakage when booting the image manually).
author_association: COLLABORATOR