Repository: red-hat-storage/ocs-ci
Pull request #11610: add workaround for issues with uefi boot order
Author: login: dahorak
Labels: name: Verified
description: Mark when PR was verified and log provided
name: team/ecosystem
description: Ecosystem team related issues/PRs
name: size/L
description: PR that changes 100-499 lines
name: lgtm
Created: 2025-03-11T15:48:04Z | Merged: 2025-03-17T12:16:43Z

We are facing an issue with one of the baremetal servers, that UEFI boot order getting changed during OCP deployment and PXE boot is not on first place which leads to an issue during the next deployment.
This workaround will try to change the boot order using `efibootmgr` command and set PXE boot on the first place.


Diff:
diff --git a/conf/README.md b/conf/README.md
index d082e2edcea..f41de90c852 100644
--- a/conf/README.md
+++ b/conf/README.md
@@ -334,6 +334,8 @@ higher priority).
             * `node_network_configuration_policy_ip` - The ip address of NodeNetworkConfigurationPolicy CR
             * `node_network_configuration_policy_prefix_length` - The subnetmask of NodeNetworkConfigurationPolicy CR
             * `node_network_configuration_policy_destination_route` - The destination route of NodeNetworkConfigurationPolicy CR
+            * `fix_uefi_boot_order_first_option` - string identifying the PXE boot option which should be set to first place, if defined
+                (this is a workaround for UEFI boot order getting changed on some servers during the OCP deployment)
 * `hcp_version` - version of HCP client to be deployed on machine running the tests
 * `metallb_version` - MetalLB operator version to install
 * `deploy_acm_hub_cluster` - Deploy ACM hub cluster or not (Default: false)
diff --git a/ocs_ci/deployment/baremetal.py b/ocs_ci/deployment/baremetal.py
index 291e9a32c89..98809164c47 100644
--- a/ocs_ci/deployment/baremetal.py
+++ b/ocs_ci/deployment/baremetal.py
@@ -25,6 +25,7 @@
 from ocs_ci.ocs.node import get_nodes
 from ocs_ci.ocs.openshift_ops import OCP
 from ocs_ci.utility import ibmcloud_bm
+from ocs_ci.utility.baremetal import update_uefi_boot_order
 from ocs_ci.utility.bootstrap import gather_bootstrap
 from ocs_ci.utility.connection import Connection
 from ocs_ci.utility.csr import wait_for_all_nodes_csr_and_approve, approve_pending_csr
@@ -1097,6 +1098,17 @@ def deploy(self, log_cli_level="DEBUG"):
             # install the OCP cluster
             self.ai_cluster.install_cluster()
 
+            # workaround for UEFI boot order issue (make sure that PXE boot is on first place)
+            for machine in master_nodes + worker_nodes:
+                if self.srv_details[machine].get("fix_uefi_boot_order_first_option"):
+                    update_uefi_boot_order(
+                        first_option_string=self.srv_details[machine].get(
+                            "fix_uefi_boot_order_first_option"
+                        ),
+                        ocp_node=machine,
+                        ignore_error=True,
+                    )
+
         def create_dns_records(self):
             """
             Configure DNS records for api and ingress
@@ -1169,6 +1181,21 @@ def set_pxe_boot_and_reboot(self, machine):
                 machine (str): Machine Name
 
             """
+            # workaround for UEFI boot order issue (make sure that PXE boot is on first place)
+            if self.srv_details[machine].get("fix_uefi_boot_order_first_option"):
+                update_uefi_boot_order(
+                    first_option_string=self.srv_details[machine].get(
+                        "fix_uefi_boot_order_first_option"
+                    ),
+                    host={
+                        "host": self.srv_details[machine]["public_ip"],
+                        "user": "core",
+                        "private_key": os.path.expanduser(
+                            config.DEPLOYMENT["ssh_key_private"]
+                        ),
+                    },
+                    ignore_error=True,
+                )
             if self.srv_details[machine].get("mgmt_provider", "ipmitool") == "ipmitool":
                 secrets = [
                     self.srv_details[machine]["mgmt_username"],
diff --git a/ocs_ci/utility/baremetal.py b/ocs_ci/utility/baremetal.py
index fef2f9fd4b6..713b7aec283 100644
--- a/ocs_ci/utility/baremetal.py
+++ b/ocs_ci/utility/baremetal.py
@@ -1,4 +1,5 @@
 import logging
+import os
 
 import pyipmi
 import pyipmi.interfaces
@@ -9,6 +10,7 @@
 from ocs_ci.ocs.exceptions import UnexpectedBehaviour
 from ocs_ci.ocs.node import wait_for_nodes_status, get_worker_nodes, get_master_nodes
 from ocs_ci.ocs.ocp import OCP, wait_for_cluster_connectivity
+from ocs_ci.utility.connection import Connection
 from ocs_ci.utility.utils import TimeoutSampler, exec_cmd
 
 logger = logging.getLogger(__name__)
@@ -244,3 +246,110 @@ def get_nodes_ipmi_ctx(self, baremetal_machine):
                 )
                 node_ipmi_ctx.append(ipmi_ctx)
         return node_ipmi_ctx
+
+
+def update_uefi_boot_order(
+    first_option_string=None, ocp_node=None, host=None, ignore_error=False
+):
+    """
+    Update UEFI boot order (using efibootmgr) to set option containing first_option_string to first place.
+    Depending on the ocp_node and host parameters, it tries to access the node via oc debug command (if ocp_node
+    parameter is provided) or via ssh (as core user) if host (hostname or IP) is provided.
+    Parameter ocp_node have precedence before host parameter, which means that if both parameters are provided, it
+    will try to use only the oc debug access.
+
+    Args:
+        first_option_string (str): String identifying the boot option which should be set to first place
+        ocp_node (str): OCP Node name used to access the server via oc debug command
+        host (dict): configuration of the server to make ssh connection or None
+            the dict could contain following keys: host, user and private_key or password
+        ignore_error (bool): If True, do not raise an exception if the command fails for some reason.
+
+    Raises:
+        UnexpectedBehaviour: If parsing of output from efibootmgr command fails
+
+    """
+
+    def run_command(cmd):
+        if ocp_node:
+            ocp_obj = OCP()
+            out = ocp_obj.exec_oc_debug_cmd(
+                node=ocp_node, cmd_list=[cmd], namespace=constants.DEFAULT_NAMESPACE
+            )
+            logger.info(f"Command '{cmd}' output: {out}")
+            return out
+        elif host:
+            private_key = (
+                os.path.expanduser(host.get("private_key"))
+                if host.get("private_key")
+                else None
+            )
+            ssh = Connection(
+                host=host["host"],
+                user=host.get("user", "core"),
+                password=host.get("password"),
+                private_key=private_key,
+                stdout=True,
+            )
+            _, out, _ = ssh.exec_cmd(cmd)
+            return out
+
+    if not ocp_node and not host:
+        raise ValueError(
+            "update_uefi_boot_order: one of 'ocp_node' and 'host' parameters have to be provided"
+        )
+
+    try:
+        efi_cfg_str = run_command("efibootmgr")
+    except Exception as ex:
+        logger.info(f"Failed to run command 'efibootmgr': {ex}")
+        if ignore_error:
+            return
+        else:
+            raise ex
+    boot_order = None
+    selected_option = None
+    for line in efi_cfg_str.splitlines():
+        if line.startswith("BootOrder:"):
+            boot_order = line.split(":")[1].strip()
+        if line.startswith("Boot") and first_option_string in line:
+            selected_option = line.split()[0].rstrip("*").removeprefix("Boot")
+
+    if not boot_order:
+        msg = f"Unable to parse output from efibootmgr - BootOrder: line not found:\n{efi_cfg_str}"
+        logger.error(msg)
+        if ignore_error:
+            return
+        else:
+            raise UnexpectedBehaviour(msg)
+    if not selected_option:
+        msg = (
+            "Unable to parse output from efibootmgr - "
+            f"Boot line with first_option_string ('{first_option_string}') not found:\n{efi_cfg_str}"
+        )
+        logger.error(msg)
+        if ignore_error:
+            return
+        else:
+            raise UnexpectedBehaviour(msg)
+
+    boot_order_list = boot_order.split(",")
+    if boot_order_list[0] == selected_option:
+        logger.debug(
+            f"No change in UEFI boot order needed, '{selected_option}' is already on first place: "
+            f"{boot_order}"
+        )
+        return
+
+    boot_order_list.remove(selected_option)
+    boot_order_list.insert(0, selected_option)
+
+    cmd = f"sudo efibootmgr --bootorder {','.join(boot_order_list)}"
+    try:
+        efi_cfg_str = run_command(cmd)
+    except Exception as ex:
+        logger.info(f"Failed to run command '{cmd}': {ex}")
+        if ignore_error:
+            return
+        else:
+            raise ex


Reviews:
user: login: petr-balogh
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-14T11:02:59Z
user: login: dahorak
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T11:05:41Z
user: login: petr-balogh
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-14T12:07:05Z
user: login: amr1ta
body: looking good
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T10:13:31Z
user: login: jilju
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T10:14:33Z
user: login: petr-balogh
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-17T11:48:55Z
user: login: DanielOsypenko
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T11:59:22Z

Review comments:
diff_hunk: @@ -244,3 +246,110 @@ def get_nodes_ipmi_ctx(self, baremetal_machine):
                 )
                 node_ipmi_ctx.append(ipmi_ctx)
         return node_ipmi_ctx
+
+
+def update_uefi_boot_order(
+    first_option_string=None, ocp_node=None, host=None, ignore_error=False
+):
+    """
+    Update UEFI boot order (using efibootmgr) to set option containing first_option_string to first place.
+    Depending on the ocp_node and host parameters, it tries to access the node via oc debug command (if ocp_node
+    parameter is provided) or via ssh (as core user) if host (hostname or IP) is provided.
+    Parameter ocp_node have precedence before host parameter, which means that if both parameters are provided, it
+    will try to use only the oc debug access.
+
+    Args:
+        first_option_string (str): String identifying the boot option which should be set to first place
+        ocp_node (str): OCP Node name used to access the server via oc debug command
+        host (dict): configuration of the server to make ssh connection or None
+                the dict could contain following keys: host, user and private_key or password
path: ocs_ci/utility/baremetal.py
user: login: petr-balogh
body: I think you have one more indention here
created_at: 2025-03-14T11:02:23Z
author_association: MEMBER
diff_hunk: @@ -244,3 +246,110 @@ def get_nodes_ipmi_ctx(self, baremetal_machine):
                 )
                 node_ipmi_ctx.append(ipmi_ctx)
         return node_ipmi_ctx
+
+
+def update_uefi_boot_order(
+    first_option_string=None, ocp_node=None, host=None, ignore_error=False
+):
+    """
+    Update UEFI boot order (using efibootmgr) to set option containing first_option_string to first place.
+    Depending on the ocp_node and host parameters, it tries to access the node via oc debug command (if ocp_node
+    parameter is provided) or via ssh (as core user) if host (hostname or IP) is provided.
+    Parameter ocp_node have precedence before host parameter, which means that if both parameters are provided, it
+    will try to use only the oc debug access.
+
+    Args:
+        first_option_string (str): String identifying the boot option which should be set to first place
+        ocp_node (str): OCP Node name used to access the server via oc debug command
+        host (dict): configuration of the server to make ssh connection or None
+                the dict could contain following keys: host, user and private_key or password
path: ocs_ci/utility/baremetal.py
user: login: dahorak
body: fixed
created_at: 2025-03-14T11:05:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: dahorak
created_at: 2025-03-14T10:35:45Z
body: Verification job: https://url.corp.redhat.com/ff24aa2
The workaround were applied (the boot order were check) both before and after the OCP deployment:
```
...
2025-03-14 10:29:27  05:29:23 - MainThread - ocs_ci.utility.connection - INFO  - Executing cmd: efibootmgr on <IP>
2025-03-14 10:29:27  05:29:24 - MainThread - ocs_ci.utility.connection - INFO  - stdout: BootCurrent: 0000
2025-03-14 10:29:27  Timeout: 1 seconds
2025-03-14 10:29:27  BootOrder: 0005,0000,000A,0003,0006,0007,0008,0009
2025-03-14 10:29:27  Boot0000* Red Hat Enterprise Linux
2025-03-14 10:29:27  Boot0003* UEFI: Built-in EFI Shell
2025-03-14 10:29:27  Boot0005* (B24/D0/F0) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 10:29:27  Boot0006  (B24/D0/F1) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:....)
2025-03-14 10:29:27  Boot0007  (B24/D0/F2) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:....)
2025-03-14 10:29:27  Boot0008  (B24/D0/F3) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:....)
2025-03-14 10:29:27  Boot0009  USB CD
2025-03-14 10:29:27  Boot000A* UEFI: ATEN Virtual CDROM YS0J
2025-03-14 10:29:27  MirrorStatus: Platform does not support address range mirror
2025-03-14 10:29:27  DesiredMirroredPercentageAbove4G: 0.00
2025-03-14 10:29:27  DesiredMirrorMemoryBelow4GB: false
...
2025-03-14 11:23:27  06:23:22 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc debug nodes/baremetal6-01 --to-namespace=default  -- chroot /host /bin/bash -c "efibootmgr || echo 'CMD FAILED'; "
2025-03-14 11:23:27  + chmod o+rX -R /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth
2025-03-14 11:23:30  06:23:30 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Starting pod/baremetal6-01-debug-ncqrq ...
2025-03-14 11:23:30  To use host binaries, run `chroot /host`
2025-03-14 11:23:30  
2025-03-14 11:23:30  Removing debug pod ...
2025-03-14 11:23:30  
2025-03-14 11:23:30  06:23:30 - MainThread - ocs_ci.utility.baremetal - INFO  - Command 'efibootmgr' output: BootCurrent: 0001
2025-03-14 11:23:30  Timeout: 1 seconds
2025-03-14 11:23:30  BootOrder: 0001,0005,000A,0003,0006,0007,0008,0009
2025-03-14 11:23:30  Boot0001* Red Hat Enterprise Linux
2025-03-14 11:23:30  Boot0003* UEFI: Built-in EFI Shell
2025-03-14 11:23:30  Boot0005* (B24/D0/F0) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:30  Boot0006  (B24/D0/F1) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:30  Boot0007  (B24/D0/F2) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:30  Boot0008  (B24/D0/F3) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:30  Boot0009  USB CD
2025-03-14 11:23:30  Boot000A* UEFI: ATEN Virtual CDROM YS0J
2025-03-14 11:23:30  MirrorStatus: Platform does not support address range mirror
2025-03-14 11:23:30  DesiredMirroredPercentageAbove4G: 0.00
2025-03-14 11:23:30  DesiredMirrorMemoryBelow4GB: false
2025-03-14 11:23:30  
2025-03-14 11:23:30  06:23:30 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc debug nodes/baremetal6-01 --to-namespace=default  -- chroot /host /bin/bash -c "efibootmgr --bootorder 0005,0001,000A,0003,0006,0007,0008,0009 || echo 'CMD FAILED'; "
2025-03-14 11:23:31  06:23:31 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Starting pod/baremetal6-01-debug-9b5dw ...
2025-03-14 11:23:31  To use host binaries, run `chroot /host`
2025-03-14 11:23:31  
2025-03-14 11:23:31  Removing debug pod ...
2025-03-14 11:23:31  
2025-03-14 11:23:31  06:23:31 - MainThread - ocs_ci.utility.baremetal - INFO  - Command 'efibootmgr --bootorder 0005,0001,000A,0003,0006,0007,0008,0009' output: BootCurrent: 0001
2025-03-14 11:23:31  Timeout: 1 seconds
2025-03-14 11:23:31  BootOrder: 0005,0001,000A,0003,0006,0007,0008,0009
2025-03-14 11:23:31  Boot0001* Red Hat Enterprise Linux
2025-03-14 11:23:31  Boot0003* UEFI: Built-in EFI Shell
2025-03-14 11:23:31  Boot0005* (B24/D0/F0) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:31  Boot0006  (B24/D0/F1) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:31  Boot0007  (B24/D0/F2) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:31  Boot0008  (B24/D0/F3) UEFI PXE IPv4: Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T(MAC:...)
2025-03-14 11:23:31  Boot0009  USB CD
2025-03-14 11:23:31  Boot000A* UEFI: ATEN Virtual CDROM YS0J
2025-03-14 11:23:31  MirrorStatus: Platform does not support address range mirror
2025-03-14 11:23:31  DesiredMirroredPercentageAbove4G: 0.00
2025-03-14 11:23:31  DesiredMirrorMemoryBelow4GB: false
2025-03-14 11:23:31  
```
author_association: CONTRIBUTOR
user: login: dahorak
created_at: 2025-03-17T11:39:46Z
body: I have to push one more small fix - the `efibootmgr` has to be run with `sudo`.
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-17T11:59:27Z
body: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/red-hat-storage/ocs-ci/pull/11610#" title="Author self-approved">dahorak</a>*, *<a href="https://github.com/red-hat-storage/ocs-ci/pull/11610#pullrequestreview-2690198398" title="Approved">DanielOsypenko</a>*, *<a href="https://github.com/red-hat-storage/ocs-ci/pull/11610#pullrequestreview-2690166543" title="Approved">petr-balogh</a>*


The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=red-hat-storage%2Focs-ci).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/red-hat-storage/ocs-ci/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: dahorak
created_at: 2025-03-17T12:15:58Z
body: New verification job with the latest change: https://url.corp.redhat.com/630c831
author_association: CONTRIBUTOR
user: login: dahorak
created_at: 2025-12-15T11:47:51Z
body: /cherry-pick release-4.17
author_association: CONTRIBUTOR
user: login: dahorak
created_at: 2025-12-15T11:48:00Z
body: /cherry-pick release-4.16
author_association: CONTRIBUTOR
user: login: dahorak
created_at: 2025-12-15T11:48:04Z
body: /cherry-pick release-4.15
author_association: CONTRIBUTOR
user: login: openshift-cherrypick-robot
created_at: 2025-12-15T11:48:31Z
body: @dahorak: new pull request created: #13928

<details>

In response to [this](https://github.com/red-hat-storage/ocs-ci/pull/11610#issuecomment-3655200437):

>/cherry-pick release-4.17


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: COLLABORATOR
user: login: openshift-cherrypick-robot
created_at: 2025-12-15T11:48:39Z
body: @dahorak: new pull request created: #13929

<details>

In response to [this](https://github.com/red-hat-storage/ocs-ci/pull/11610#issuecomment-3655200900):

>/cherry-pick release-4.16


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: COLLABORATOR
user: login: openshift-cherrypick-robot
created_at: 2025-12-15T11:48:42Z
body: @dahorak: #11610 failed to apply on top of branch "release-4.15":
```
Patch is empty.
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To record the empty patch as an empty commit, run "git am --allow-empty".
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"

```

<details>

In response to [this](https://github.com/red-hat-storage/ocs-ci/pull/11610#issuecomment-3655201118):

>/cherry-pick release-4.15


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: COLLABORATOR
user: login: dahorak
created_at: 2025-12-15T12:23:42Z
body: /cherry-pick release-4.15
author_association: CONTRIBUTOR
user: login: openshift-cherrypick-robot
created_at: 2025-12-15T12:24:20Z
body: @dahorak: #11610 failed to apply on top of branch "release-4.15":
```
Applying: add workaround for issues with uefi boot order
Using index info to reconstruct a base tree...
M	conf/README.md
M	ocs_ci/deployment/baremetal.py
Falling back to patching base and 3-way merge...
Auto-merging ocs_ci/deployment/baremetal.py
Auto-merging conf/README.md
CONFLICT (content): Merge conflict in conf/README.md
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 add workaround for issues with uefi boot order

```

<details>

In response to [this](https://github.com/red-hat-storage/ocs-ci/pull/11610#issuecomment-3655336429):

>/cherry-pick release-4.15


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: COLLABORATOR