Repository: openshift/aro-e2e
Pull request #11: feat(ansible): Add smoke tests for logging and monitoring, and support custom pull secrets
Author: login: cloudygreybeard
Labels: name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: Indicates that a PR is ready to be merged.
Created: 2025-04-30T22:52:48Z | Merged: 2025-05-12T18:23:16Z

### What this PR does / why we need it:

This PR introduces two enhancements to the `aro-e2e` test suite:

1. **Smoke test coverage for `openshift-azure-logging` and `openshift-monitoring` components**  
   - Adds Ansible tasks to verify that all pods in the `openshift-azure-logging` namespace are running and healthy.
   - Adds tasks to detect and fail on any firing alerts in Alertmanager (excluding Watchdog), using in-cluster execution of `amtool` within a pod in the `openshift-monitoring` namespace.
   - Failures from these checks are non-fatal during task execution and are instead aggregated into a common `smoke_test_failures` list for final evaluation.

2. **Custom pull secret support during test setup**  
   - Enables optional injection and use of a custom pull secret, allowing image pulls from mirrored or private registries in restricted CI environments.
   - Logic is environment-aware: it conditionally applies the secret depending on whether the context is a Day 0 (`azcli`) or Day 2 (`k8s`) deployment.

Together, these features improve post-upgrade component validation and make the test suite more adaptable to enterprise CI/CD constraints.

---

### Implementation Notes:

- Failures are tracked via the `smoke_test_failures` list, allowing for modular and extensible error reporting across multiple test sources.
- Final assertion and summary reporting is centralized in a dedicated task file:  
  `ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/summary.yaml`  
  This file is invoked explicitly at the end of `tasks/main.yaml`, allowing future smoke tests to contribute to the shared failure state in a composable manner.

---

### How to verify it:

Run `make cluster` as normal, with the optional addition of a valid pull secret included in ./secrets/pull-secret.txt, or at a path defined in PULL_SECRET_FILE env.

- The play will succeed if all checks pass and the cluster is healthy.
- If any openshift-azure-logging pods are unhealthy or non-Watchdog alerts are firing in openshift-monitoring, the failure will be captured and reported in the summary assertion.
- Custom pull secret logic can be verified by observing registry access logs or inspecting the created secret.

---

### Does this PR introduce a user-facing change?

```release-note
feat: Add smoke tests for openshift-azure-logging and openshift-monitoring
feat: Enable custom pull secret support during e2e ansible test setup
```


Diff:
diff --git a/Makefile b/Makefile
index 4f6044c..25500a5 100755
--- a/Makefile
+++ b/Makefile
@@ -117,11 +117,35 @@ ansible-image:
 LOCATION ?= eastus
 CLUSTERPREFIX ?= $(USER)
 CLUSTERPATTERN ?= basic
-CLEANUP := False
-INVENTORY := "hosts.yaml"
-SSH_CONFIG_DIR := $(HOME)/.ssh/
-SSH_KEY_BASENAME := id_rsa
-ANSIBLE_VERBOSITY := 0
+CLEANUP ?= False
+INVENTORY ?= "hosts.yaml"
+SSH_CONFIG_DIR ?= $(HOME)/.ssh/
+SSH_KEY_BASENAME ?= id_rsa
+ANSIBLE_VERBOSITY ?= 0
+# json-formatted array of severity strings to be ignored in Alertmanager smoke tests. If empty, playbook later defaults to '["none","info"]':
+IGNORED_ALERT_SEVERITIES ?=
+PULL_SECRET_FILE ?= $(CURDIR)/secrets/pull-secret.txt
+PULL_SECRET_FILE_AT_DELEGATE := /tmp/pull-secret.txt
+# Get python version from Dockerfile.ansible or from env if PYTHON_VERSION is set (e.g. `PYTHON_VERSION='python3.12'`)
+PYTHON_VERSION ?= $(shell grep '^FROM ' $(CURDIR)/Dockerfile.ansible | sed -nE 's|.*/(python)-([0-9])([0-9]+):.*|\1\2.\3|p')
+
+# Check if file exists at PULL_SECRET_FILE and set as empty string if not
+PULL_SECRET_FILE := $(shell if [ -f "$(PULL_SECRET_FILE)" ]; then echo "$(PULL_SECRET_FILE)"; else echo ''; fi)
+
+# Define a mount arg for podman. Commas [wreak havoc in a make function](https://www.gnu.org/software/make/manual/html_node/Functions.html)
+ifneq ($(PULL_SECRET_FILE),)
+	PODMAN_ARG_PULL_SECRET_MOUNT := -v "$(PULL_SECRET_FILE)":"$(PULL_SECRET_FILE_AT_DELEGATE)":ro,Z
+	ANSIBLE_ARG_PULL_SECRET_ENV := -e PULL_SECRET_FILE='$(PULL_SECRET_FILE_AT_DELEGATE)'
+endif
+
+ifneq ($(PULL_SECRET_FILE_METHOD),)
+	ANSIBLE_ARG_PULL_SECRET_FILE_METHOD_ENV := -e PULL_SECRET_FILE_METHOD='$(PULL_SECRET_FILE_METHOD)'
+endif
+
+ifneq ($(IGNORED_ALERT_SEVERITIES),)
+	ANSIBLE_ARG_IGNORED_ALERT_SEVERITIES_ENV := -e ignored_alert_severities='$(IGNORED_ALERT_SEVERITIES)'
+endif
+
 ifneq ($(CLUSTERPATTERN),*)
 	CLUSTERFILTER = -l $(CLUSTERPATTERN)
 endif
@@ -137,7 +161,8 @@ cluster:
 		-v $${AZURE_CONFIG_DIR:-~/.azure}:/opt/app-root/src/.azure$(PODMAN_VOLUME_OVERLAY) \
 		-v ./ansible:/ansible$(PODMAN_VOLUME_OVERLAY) \
 		-v $(SSH_CONFIG_DIR):/root/.ssh$(PODMAN_VOLUME_OVERLAY) \
-		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
+		$(PODMAN_ARG_PULL_SECRET_MOUNT) \
+		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/$(PYTHON_VERSION)/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
 		-e ANSIBLE_VERBOSITY=$(ANSIBLE_VERBOSITY) \
 		aro-ansible:$(VERSION) \
 			-i $(INVENTORY) \
@@ -146,7 +171,11 @@ cluster:
 			-e CLUSTERPREFIX=$(CLUSTERPREFIX) \
 			-e CLEANUP=$(CLEANUP) \
 			-e SSH_KEY_BASENAME=$(SSH_KEY_BASENAME) \
+			$(ANSIBLE_ARG_IGNORED_ALERT_SEVERITIES_ENV) \
+			$(ANSIBLE_ARG_PULL_SECRET_ENV) \
+			$(ANSIBLE_ARG_PULL_SECRET_FILE_METHOD_ENV) \
 			deploy.playbook.yaml
+
 .PHONY: cluster-cleanup
 cluster-cleanup:
 		podman $(PODMAN_REMOTE_ARGS) \
@@ -157,7 +186,7 @@ cluster-cleanup:
 			-v $${AZURE_CONFIG_DIR:-~/.azure}:/opt/app-root/src/.azure$(PODMAN_VOLUME_OVERLAY) \
 			-v ./ansible:/ansible$(PODMAN_VOLUME_OVERLAY) \
 			-v $(SSH_CONFIG_DIR):/root/.ssh$(PODMAN_VOLUME_OVERLAY) \
-			-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
+			-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/$(PYTHON_VERSION)/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
 			-e ANSIBLE_VERBOSITY=$(ANSIBLE_VERBOSITY) \
 			aro-ansible:$(VERSION) \
 				-i $(INVENTORY) \
@@ -180,21 +209,23 @@ lint-ansible:
 		-v $${AZURE_CONFIG_DIR:-~/.azure}:/opt/app-root/src/.azure$(PODMAN_VOLUME_OVERLAY) \
 		-v ./ansible:/ansible$(PODMAN_VOLUME_OVERLAY) \
 		-v $(SSH_CONFIG_DIR):/root/.ssh$(PODMAN_VOLUME_OVERLAY) \
-		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
+		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/$(PYTHON_VERSION)/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
 		--entrypoint ansible-lint \
 		aro-ansible:$(VERSION) \
-		    --offline \
-            --project-dir /ansible
+			--offline \
+			--project-dir /ansible
 
 .PHONY: test-ansible
 test-ansible:
 	podman $(PODMAN_REMOTE_ARGS) \
 		run \
 		--rm \
-		--entrypoint ansible-test \
-		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
-		--workdir /opt/app-root/src/.local/share/pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/azureredhatopenshift/cluster/ \
 		-it \
+		-v ./ansible_collections/azureredhatopenshift/cluster/:/opt/app-root/src/.local/share/pipx/venvs/ansible/lib/$(PYTHON_VERSION)/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
+		-v ./ansible:/ansible$(PODMAN_VOLUME_OVERLAY) \
+		-v $(SSH_CONFIG_DIR):/root/.ssh$(PODMAN_VOLUME_OVERLAY) \
+		--entrypoint ansible-test \
+		--workdir /opt/app-root/src/.local/share/pipx/venvs/ansible/lib/$(PYTHON_VERSION)/site-packages/ansible_collections/azureredhatopenshift/cluster$(PODMAN_VOLUME_OVERLAY) \
 		aro-ansible:$(VERSION) \
 			sanity \
 			-v
diff --git a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/defaults/main.yaml b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/defaults/main.yaml
new file mode 100644
index 0000000..2faa792
--- /dev/null
+++ b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/defaults/main.yaml
@@ -0,0 +1,12 @@
+# This file serves as the canonical place for default variable values in this role,
+# per Ansible's [standard role directory structure](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html#role-directory-structure).
+# Placing defaults in `roles/${role_name}/defaults/main.yaml` is the preferred way to define low-priority (i.e. easily overridable) role variables.
+# These can be overridden by:
+# * Extra vars (`-e`)
+# * Environment vars passed via Makefile
+# * Vars defined in playbooks, inventory, or host/group vars
+
+# health_check | Filter firing alerts based on severity:
+ignored_alert_severities:
+  - none
+  - info
diff --git a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/health_check.yaml b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/health_check.yaml
index 734cb75..c17d7bc 100644
--- a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/health_check.yaml
+++ b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/health_check.yaml
@@ -40,3 +40,116 @@
   ansible.builtin.fail:
     msg: "aro-operator-worker replicas {{ item.status.availableReplicas }}"
   loop: "{{ deployment_aro_operator_worker.resources }}"
+
+- name: health_check | openshift-azure-logging
+  block:
+
+    - name: health_check | Get pods in openshift-azure-logging
+      # Fetch all pod objects from the openshift-azure-logging namespace
+      kubernetes.core.k8s_info:
+        ca_cert: "{{ cluster_cert_file }}"
+        kubeconfig: "{{ inventory_hostname }}.kubeconfig"
+        api_version: v1
+        kind: Pod
+        namespace: openshift-azure-logging
+      delegate_to: "{{ delegation }}"
+      register: azure_logging_pods
+
+    - name: health_check | Init list of unhealthy pods in openshift-azure-logging
+      # Start with an empty list to collect unhealthy pod names
+      set_fact:
+        unhealthy_azure_logging_pods: []
+
+    - name: health_check | Identify unhealthy pods in openshift-azure-logging
+      # Evaluate each pod for Running status and restart count; add unhealthy pods to list
+      vars:
+        pod_name: "{{ item.metadata.name }}"
+        pod_phase: "{{ item.status.phase | default('Unknown') }}"
+        container_restarts: "{{ item.status.containerStatuses | default([]) }}"
+        restart_count_exceeded: "{{ container_restarts | selectattr('restartCount', '>', 1) | list | length > 0 }}"
+      set_fact:
+        unhealthy_azure_logging_pods: "{{ unhealthy_azure_logging_pods + [pod_name] }}"
+      when: pod_phase != 'Running' or restart_count_exceeded
+      loop: "{{ azure_logging_pods.resources }}"
+
+    - name: health_check | Assert all pods in openshift-azure-logging are healthy
+      # Assert that no unhealthy pods exist in this namespace
+      ansible.builtin.assert:
+        that:
+          - unhealthy_azure_logging_pods | length == 0
+        fail_msg: >-
+          Found unhealthy openshift-azure-logging pods: {{ unhealthy_azure_logging_pods | join(', ') }}
+        success_msg: "All openshift-azure-logging pods are running and healthy"
+      ignore_errors: true
+      register: azure_logging_assert_result
+
+    - name: health_check | Record failure if openshift-azure-logging check failed
+      # Append the namespace name to the global failure list if assertion failed
+      set_fact:
+        smoke_test_failures: "{{ smoke_test_failures | default([]) + ['openshift-azure-logging'] }}"
+      when: azure_logging_assert_result is defined and azure_logging_assert_result.failed
+
+- name: health_check | openshift-monitoring
+  block:
+
+    - name: health_check | Get alertmanager pods in openshift-monitoring
+      # Retrieve alertmanager pods in the openshift-monitoring namespace
+      kubernetes.core.k8s_info:
+        ca_cert: "{{ cluster_cert_file }}"
+        kubeconfig: "{{ inventory_hostname }}.kubeconfig"
+        api_version: v1
+        kind: Pod
+        namespace: openshift-monitoring
+        label_selectors:
+          - alertmanager=main
+      delegate_to: "{{ delegation }}"
+      register: alertmanager_pods
+
+    - name: health_check | Query firing alerts using amtool from alertmanager pod
+      # Use 'amtool' inside the first Alertmanager pod to list all currently firing alerts
+      kubernetes.core.k8s_exec:
+        ca_cert: "{{ cluster_cert_file }}"
+        kubeconfig: "{{ inventory_hostname }}.kubeconfig"
+        namespace: openshift-monitoring
+        pod: "{{ alertmanager_pods.resources[0].metadata.name }}"
+        command: "/usr/bin/amtool --alertmanager.url=http://localhost:9093 alert --output=json"
+      register: firing_alerts_output
+      delegate_to: "{{ delegation }}"
+
+    - name: health_check | Filter firing alerts based on severity
+      # Parse the output from amtool, removing any alerts matching ignored severity labels
+      vars:
+        all_alerts: "{{ firing_alerts_output.stdout | from_json }}"
+      set_fact:
+        filtered_alerts: >-
+          {{ all_alerts
+             | rejectattr('labels.severity', 'in', ignored_alert_severities)
+             | list }}
+
+    - name: health_check | Print filtered alerts firing
+      # Display details of filtered firing alerts if verbosity is high
+      ansible.builtin.debug:
+        msg: |
+          The following filtered alerts are currently firing:
+          {{ filtered_alerts | join('\n\n') }}
+        verbosity: 2
+      when: filtered_alerts | length > 0
+
+    - name: health_check | Assert no firing alerts in Alertmanager
+      # Assert that no filtered alerts are firing
+      ansible.builtin.assert:
+        that:
+          - filtered_alerts | length == 0
+        fail_msg: >-
+          Found {{ filtered_alerts | length }} alert(s) firing in Alertmanager: {{
+            filtered_alerts | map(attribute='labels.alertname') | join(', ')
+          }}
+        success_msg: "No firing alerts in Alertmanager"
+      ignore_errors: true
+      register: alertmanager_assert_result
+
+    - name: health_check | Record failure if openshift-monitoring check failed
+      # Append the namespace name to the global failure list if assertion failed
+      set_fact:
+        smoke_test_failures: "{{ smoke_test_failures | default([]) + ['openshift-monitoring'] }}"
+      when: alertmanager_assert_result is defined and alertmanager_assert_result.failed
diff --git a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/main.yaml b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/main.yaml
index 87d6583..8df78c0 100644
--- a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/main.yaml
+++ b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/main.yaml
@@ -17,3 +17,6 @@
 - name: Refresh credentials
   ansible.builtin.include_tasks:
     file: refresh_credentials.yaml
+- name: Summarise smoke test results
+  ansible.builtin.include_tasks:
+    file: summary.yaml
diff --git a/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/summary.yaml b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/summary.yaml
new file mode 100644
index 0000000..d81a748
--- /dev/null
+++ b/ansible_collections/azureredhatopenshift/cluster/roles/smoketest/tasks/summary.yaml
@@ -0,0 +1,15 @@
+- name: smoke_tests | summary
+  block:
+
+    - name: smoke_tests | Aggregate failure summary
+      # Collect all namespaces where checks failed
+      set_fact:
+        smoke_test_failures: "{{ smoke_test_failures | default([]) }}"
+
+    - name: smoke_tests | Final assertion
+      # Fail the play if any namespace failed its health check
+      ansible.builtin.assert:
+        that:
+          - smoke_test_failures | length == 0
+        fail_msg: "Smoke tests failed for: {{ smoke_test_failures | join(', ') }}"
+        success_msg: "All smoke tests passed successfully"
diff --git a/ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml b/ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml
index 2452dd3..5b1eeae 100644
--- a/ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml
+++ b/ansible_collections/azureredhatopenshift/cluster/tasks/create_aro_cluster.yaml
@@ -16,6 +16,31 @@
     var: localhost_facts
     verbosity: 2 # Higher verbosity because this prints a lot of data
 
+- name: create_aro_cluster | Check for local pull secret file and set facts if available
+  when: PULL_SECRET_FILE is defined and PULL_SECRET_FILE | length > 0
+  delegate_to: localhost
+  block:
+    - name: create_aro_cluster | Stat pull secret file
+      stat:
+        path: "{{ PULL_SECRET_FILE }}"
+      register: pull_secret_file_stat
+      delegate_to: localhost
+
+    - name: create_aro_cluster | Set pull_secret_file_stat_exists fact
+      set_fact:
+        pull_secret_file_stat_exists: "{{ pull_secret_file_stat.stat.exists | default(false) }}"
+      delegate_to: localhost
+
+    - name: create_aro_cluster | Set pull secret handling facts
+      set_fact:
+        pull_secret_file: "{{ PULL_SECRET_FILE }}"
+        pull_secret_should_copy: "{{ delegation != 'localhost' }}"
+        pull_secret_should_azcli: "{{ PULL_SECRET_FILE_METHOD is not defined or ( PULL_SECRET_FILE_METHOD | default('') ) in ['azcli', '', 'day0'] }}"
+        pull_secret_should_k8s: "{{ ( PULL_SECRET_FILE_METHOD | default('') ) == 'day2' }}"
+      when: pull_secret_file_stat_exists | default(false)
+      run_once: true
+      delegate_to: localhost
+
 - name: create_aro_cluster | Enable preview az aro extension (by version number)
   when: AZAROEXT_VERSION is defined and "://" not in AZAROEXT_VERSION
   block:
@@ -109,6 +134,11 @@
         var: csp_info
         verbosity: 2
 
+- name: create_aro_cluster | Prepare pull secret
+  ansible.builtin.include_tasks:
+    file: ../../tasks/create_pullsecret.yaml
+  when: pull_secret_file is defined
+
 # In case ansible was stopped and then re-run on a cluster that's still creating or deleting
 - name: create_aro_cluster | Wait for existing cluster to finish
   azureredhatopenshift.cluster.azure_rm_openshiftmanagedcluster_info:
diff --git a/ansible_collections/azureredhatopenshift/cluster/tasks/create_cluster_azcli.yaml b/ansible_collections/azureredhatopenshift/cluster/tasks/create_cluster_azcli.yaml
index 109aed2..cd8328b 100644
--- a/ansible_collections/azureredhatopenshift/cluster/tasks/create_cluster_azcli.yaml
+++ b/ansible_collections/azureredhatopenshift/cluster/tasks/create_cluster_azcli.yaml
@@ -27,6 +27,7 @@
       - --master-subnet=master
       - --vnet=aro-vnet
       - --worker-subnet=worker
+      - "{% if pull_secret_should_azcli is defined and pull_secret_file is defined %}--pull-secret=@{{ pull_secret_file }}{% else %}{{ omit }}{% endif %}"
       - "{% if csp_info is defined %}--client-id={{ csp_info.appId }}{% else %}{{ omit }}{% endif %}"
       - "{% if csp_info is defined %}--client-secret={{ csp_info.password }}{% else %}{{ omit }}{% endif %}"
       - "{% if apiserver_visibility is defined %}--apiserver-visibility={{ apiserver_visibility }}{% else %}{{ omit }}{% endif %}"
diff --git a/ansible_collections/azureredhatopenshift/cluster/tasks/create_pullsecret.yaml b/ansible_collections/azureredhatopenshift/cluster/tasks/create_pullsecret.yaml
new file mode 100644
index 0000000..dbaa0ed
--- /dev/null
+++ b/ansible_collections/azureredhatopenshift/cluster/tasks/create_pullsecret.yaml
@@ -0,0 +1,50 @@
+# Copy the pull secret to the delegate host if needed
+- name: prepare_pullsecret | Copy pull secret to delegate
+  when: pull_secret_should_copy
+  delegate_to: "{{ delegation }}"
+  ansible.builtin.copy:
+    src: "{{ pull_secret_file }}"       # local file path on the Ansible control node, i.e. localhost (e.g., /tmp/pull-secret.txt)
+    dest: "{{ pull_secret_file }}"      # to the same path on the remote jumphost delegate
+    mode: '0644'
+
+# Set up azcli argument inclusion for Day-0
+- name: prepare_pullsecret | Confirm pull secret will be used via azcli
+  when: pull_secret_should_azcli
+  ansible.builtin.debug:
+    msg: "Pull secret will be passed to 'az aro create' via --pull-secret argument."
+    verbosity: 1
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging pull secret file existence
+  ansible.builtin.debug:
+    msg: "Pull secret file exists: {{ pull_secret_file_stat_exists | default(false) }}"
+    verbosity: 2
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging delegated task execution on remote
+  ansible.builtin.debug:
+    msg: "Executing on delegate: {{ delegation }} with pull secret file: {{ pull_secret_file }}"
+    verbosity: 2
+  delegate_to: "{{ delegation }}"
+
+# Set up Kubernetes secret for Day-2
+- name: prepare_pullsecret | Create Kubernetes secret from pull secret file
+  when: pull_secret_should_k8s
+  delegate_to: "{{ delegation }}"
+  block:
+    - name: prepare_pullsecret | Read and encode pull secret
+      set_fact:
+        pull_secret_data: "{{ lookup('file', pull_secret_file) | b64encode }}"
+
+    - name: prepare_pullsecret | Apply Kubernetes Secret to openshift-config
+      kubernetes.core.k8s:
+        state: present
+        definition:
+          apiVersion: v1
+          kind: Secret
+          metadata:
+            name: pull-secret
+            namespace: openshift-config
+          type: kubernetes.io/dockerconfigjson
+          data:
+            .dockerconfigjson: "{{ pull_secret_data }}"


Reviews:
user: login: cloudygreybeard
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T02:43:11Z
user: login: cloudygreybeard
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T21:45:02Z
user: login: ventifus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T16:16:23Z
user: login: cloudygreybeard
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T06:01:48Z

Review comments:
diff_hunk: @@ -0,0 +1,51 @@
+# Copy the pull secret to the delegate host if needed
+- name: prepare_pullsecret | Copy pull secret to delegate
+  when: pull_secret_should_copy
+  delegate_to: "{{ delegation }}"
+  ansible.builtin.copy:
+    src: "{{ pull_secret_file }}"       # local file path on the Ansible control node, i.e. localhost (e.g., /tmp/pull-secret.txt)
+    dest: "{{ pull_secret_file }}"      # to the same path on the remote jumphost delegate
+    mode: '0644'
+
+# Set up azcli argument inclusion for Day-0
+- name: prepare_pullsecret | Confirm pull secret will be used via azcli
+  when: pull_secret_should_azcli
+  ansible.builtin.debug:
+    msg: "Pull secret will be passed to 'az aro create' via --pull-secret argument."
+    verbosity: 1
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging pull secret file existence
+  ansible.builtin.debug:
+    msg: "Pull secret file exists: {{ pull_secret_file_stat_exists | default(false) }}"
+    verbosity: 2
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging delegated task execution on remote
+  ansible.builtin.debug:
+    msg: "Executing on delegate: {{ delegation }} with pull secret file: {{ pull_secret_file }}"
+    verbosity: 2
+  delegate_to: "{{ delegation }}"
+
+# Set up Kubernetes secret for Day-2
+- name: prepare_pullsecret | Create Kubernetes secret from pull secret file
+  when: pull_secret_should_k8s
+  delegate_to: "{{ delegation }}"
+  block:
+    - name: prepare_pullsecret | Read and encode pull secret
+      set_fact:
+        pull_secret_data: "{{ lookup('file', pull_secret_file) | b64encode }}"
+      delegate_to: localhost
path: ansible_collections/azureredhatopenshift/cluster/tasks/create_pullsecret.yaml
user: login: cloudygreybeard
body: `delegate_to: localhost`: May be unnecessary here given the block is delegated dynamically. Considering removal.
created_at: 2025-05-01T02:43:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,51 @@
+# Copy the pull secret to the delegate host if needed
+- name: prepare_pullsecret | Copy pull secret to delegate
+  when: pull_secret_should_copy
+  delegate_to: "{{ delegation }}"
+  ansible.builtin.copy:
+    src: "{{ pull_secret_file }}"       # local file path on the Ansible control node, i.e. localhost (e.g., /tmp/pull-secret.txt)
+    dest: "{{ pull_secret_file }}"      # to the same path on the remote jumphost delegate
+    mode: '0644'
+
+# Set up azcli argument inclusion for Day-0
+- name: prepare_pullsecret | Confirm pull secret will be used via azcli
+  when: pull_secret_should_azcli
+  ansible.builtin.debug:
+    msg: "Pull secret will be passed to 'az aro create' via --pull-secret argument."
+    verbosity: 1
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging pull secret file existence
+  ansible.builtin.debug:
+    msg: "Pull secret file exists: {{ pull_secret_file_stat_exists | default(false) }}"
+    verbosity: 2
+  delegate_to: localhost
+
+- name: prepare_pullsecret | Debugging delegated task execution on remote
+  ansible.builtin.debug:
+    msg: "Executing on delegate: {{ delegation }} with pull secret file: {{ pull_secret_file }}"
+    verbosity: 2
+  delegate_to: "{{ delegation }}"
+
+# Set up Kubernetes secret for Day-2
+- name: prepare_pullsecret | Create Kubernetes secret from pull secret file
+  when: pull_secret_should_k8s
+  delegate_to: "{{ delegation }}"
+  block:
+    - name: prepare_pullsecret | Read and encode pull secret
+      set_fact:
+        pull_secret_data: "{{ lookup('file', pull_secret_file) | b64encode }}"
+      delegate_to: localhost
path: ansible_collections/azureredhatopenshift/cluster/tasks/create_pullsecret.yaml
user: login: cloudygreybeard
body: Resolved in https://github.com/openshift/aro-e2e/pull/11/commits/3a1681ae30ede3a8947b895bda7526105e47ad31.
created_at: 2025-05-01T21:45:02Z
author_association: MEMBER
diff_hunk: @@ -152,6 +153,7 @@ cluster:
 			-e location=$(LOCATION) \
 			-e CLUSTERPREFIX=$(CLUSTERPREFIX) \
 			-e CLEANUP=$(CLEANUP) \
+			$(if $(IGNORED_ALERT_SEVERITIES),-e ignored_alert_severities='$(IGNORED_ALERT_SEVERITIES)') \
path: Makefile
user: login: ventifus
body: Question: does this need to be settable from `make cluster ...` or is setting this in inventory sufficient?
created_at: 2025-05-07T16:16:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -152,6 +153,7 @@ cluster:
 			-e location=$(LOCATION) \
 			-e CLUSTERPREFIX=$(CLUSTERPREFIX) \
 			-e CLEANUP=$(CLEANUP) \
+			$(if $(IGNORED_ALERT_SEVERITIES),-e ignored_alert_severities='$(IGNORED_ALERT_SEVERITIES)') \
path: Makefile
user: login: cloudygreybeard
body: I'm hoping to allow for either here, and I've slipped some guidance as comments into [ansible_collections/azureredhatopenshift/cluster/roles/smoketest/defaults/main.yaml](https://github.com/openshift/aro-e2e/pull/11/files#diff-e5d202aacf445fdcc281a1c9544460c831c4bf42ebd6a899e046b2a5e375c7b1), particularly:
> These can be overridden by:
> * Extra vars (`-e`)
> * Environment vars passed via Makefile
> * Vars defined in playbooks, inventory, or host/group vars

I've also just updated the Makefile comments and code hoping to add some clarity here (`#json-formatted array of severity strings to be ignored in Alertmanager smoke tests. If empty, playbook later defaults to '["none","info"]'`).
created_at: 2025-05-08T06:01:48Z
author_association: MEMBER

PR comments:
user: login: openshift-ci[bot]
created_at: 2025-04-30T22:53:02Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openshift/aro-e2e/pull/11#" title="Author self-approved">cloudygreybeard</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openshift%2Faro-e2e).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/openshift/aro-e2e/blob/main/OWNERS)~~ [cloudygreybeard]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: openshift-ci[bot]
created_at: 2025-05-09T00:00:48Z
body: @cloudygreybeard: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
ci/prow/e2e | 7667fe14bc8ced4a446770e6ee32405b6cfbecae | [link](https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_aro-e2e/11/pull-ci-openshift-aro-e2e-main-e2e/1920621171766726656) | false | `/test e2e`

[Full PR test history](https://prow.ci.openshift.org/pr-history?org=openshift&repo=aro-e2e&pr=11). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:cloudygreybeard).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: NONE
user: login: ventifus
created_at: 2025-05-12T18:17:46Z
body: /lgtm
author_association: CONTRIBUTOR