Repository: canonical/pycloudlib
Pull request #477: Add OCI example tests for OFED userspace tools and iperf3
Author: login: trentindav
Created: 2025-02-27T21:49:16Z | Merged: 2025-03-31T19:05:51Z

### Description

In the context of testing OFED on OCI instances I prepared a list of CLI tools that must be installed by the package.
I was suggested to add tests to validate that these commands are installed in the `example/oracle/oracle-example-cluster-test.py` . Note that these tests simply check that no error is returned, and that stdout includes a few expected keywords.
The CLI tools covered by the tests are:
- `mst`
- `mlxconfig`
- `mlxfwmanager`
- `flint`
- `mlxfwreset`

I also added a test to check what transmission performance is measured by `iperf3`, the expected throughput gives some room below the 50GBps supported by the physical interface.

### Additional Context and Relevant Issues

The original manual testing of the OFED packages comes from ATLA-29

### Test Steps

I manually executed the newly added tests on running instances where OFED and iperf3 were already installed. All tests pass.

```bash
$ tox -e integration-tests -- examples/oracle/oracle-example-cluster-test.py -k "TestOracleClusterOfedTools"
$ tox -e integration-tests -- examples/oracle/oracle-example-cluster-test.py -k "TestOracleClusterPerformance"
$ tox -e format
```


Diff:
diff --git a/VERSION b/VERSION
index 63c70000..d43e29f5 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-1!10.10.0
+1!10.10.1
diff --git a/examples/oracle/oracle-example-cluster-test.py b/examples/oracle/oracle-example-cluster-test.py
index 53105b84..605bb73b 100644
--- a/examples/oracle/oracle-example-cluster-test.py
+++ b/examples/oracle/oracle-example-cluster-test.py
@@ -3,6 +3,7 @@
 """Basic examples of various lifecycle with an OCI instance."""
 
 import logging
+import re
 import threading
 import time
 from datetime import datetime
@@ -51,7 +52,7 @@ def cluster() -> Generator[List[OciInstance], None, None]:
 class TestOracleClusterBasic:
     """Test basic functionalities of Oracle Cluster."""
 
-    def test_basic_ping_on_private_ips(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+    def test_basic_ping_on_private_ips(self, cluster: List[OciInstance]):
         """
         Test that cluster instances can ping each other on private IPs.
 
@@ -71,7 +72,7 @@ def test_basic_ping_on_private_ips(self, cluster: List[OciInstance]):  # pylint:
                     logger.info("Successfully pinged %s from %s", private_ip, instance.private_ip)
 
 
-def setup_mofed_iptables_rules(instance: OciInstance):
+def setup_mofed_iptables_rules(instance: OciInstance) -> OciInstance:
     """
     Set up IPTABLES rules for RDMA usage.
 
@@ -126,7 +127,73 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        test_cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance) -> str:
+    """
+    Get the PCI address of the second NIC on the instance (mlx5_1) which is used for RDMA.
+
+    The `mst status -v` command returns an output in the following format, where the
+    PCI address can be extracted from the column at index 2:
+
+    ```
+    $ sudo mst status -v | grep mlx5_1
+    ConnectX6DX(rev:0)      NA       4b:00.1   mlx5_1          net-ens300f1np1
+    ```
+
+    Args:
+        instance (OciInstance): The instance to get the PCI address from.
+
+    Returns:
+        str: The PCI address of the second NIC.
+
+    Raises:
+        ValueError: If a valid PCI address cannot be parsed from the mst output.
+    """
+    r = instance.execute("sudo mst status -v | grep mlx5_1")
+    if not r.ok or not r.stdout:
+        raise ValueError("Failed to retrieve PCI address: mst status command failed")
+
+    try:
+        pciaddr = r.stdout.split()[2]
+    except IndexError:
+        raise ValueError("Failed to retrieve the second column of the mst status output")
+
+    if not re.match(r".*[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9a-fA-F]$", pciaddr):
+        raise ValueError(f"Invalid PCI address format: {pciaddr}")
+
+    return pciaddr
 
 
 class TestOracleClusterRdma:
@@ -135,7 +202,7 @@ class TestOracleClusterRdma:
     @pytest.fixture(scope="class")
     def mofed_cluster(
         self,
-        cluster: List[OciInstance],  # pylint: disable=W0621
+        cluster: List[OciInstance],
     ) -> Generator[List[OciInstance], None, None]:
         """
         Configure cluster for RDMA testing.
@@ -144,20 +211,7 @@ def mofed_cluster(
             List[OciInstance]: RDMA-ready cluster instances.
         """
         ensure_image_is_rdma_ready(cluster[0])
-        for instance in cluster:
-            if instance.secondary_vnic_private_ip:
-                logger.info(
-                    "Instance %s already has a secondary VNIC, not attaching one.", instance.name
-                )
-                continue
-            logger.info("Creating a secondary VNIC on instance %s", instance.name)
-            # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
-            instance.add_network_interface(
-                nic_index=1,
-                subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
-            )
-            instance.configure_secondary_vnic()
-            setup_mofed_iptables_rules(instance)
+        ensure_second_vnics_ready(cluster)
 
         yield cluster
 
@@ -295,3 +349,146 @@ def start_server():
         )
         logger.info("ucx_perftest output: %s", r.stdout)
         assert r.ok, "Failed to run ucx_perftest"
+
+
+class TestOracleClusterOfedTools:
+    """
+    Test Nvidia tools included in OFED userspace package.
+
+    Validate that CLI tools included in OFED are installed and executable.
+    Only verify query commands to avoid affecting the physical NIC firmware.
+    """
+
+    def test_mst_status(self, cluster: List[OciInstance]):
+        """
+        Run mst status to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+
+        r = dut_instance.execute("sudo mst status")
+        logger.info("mst status output: %s", r.stdout)
+        assert r.ok, "Failed to run mst status"
+        assert "MST modules" in r.stdout
+        assert "PCI Devices" in r.stdout
+
+    def test_mlxconfig(self, cluster: List[OciInstance]):
+        """
+        Run mlxconfig to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxconfig -d {pci_addr} q")
+        logger.info("mlxconfig query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxconfig query"
+        assert "ConnectX" in r.stdout
+
+    def test_mlxfwmanager(self, cluster: List[OciInstance]):
+        """
+        Run mlxfwmanager to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwmanager -d {pci_addr} --query")
+        logger.info("mlxfwmanager query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxfwmanager query"
+        assert "ConnectX" in r.stdout
+        assert "Device Type:" in r.stdout
+        assert "Part Number:" in r.stdout
+
+    def test_flint(self, cluster: List[OciInstance]):
+        """
+        Run flint to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo flint -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
+        assert "Image type:" in r.stdout
+        assert "FW Version:" in r.stdout
+        assert "Product Version:" in r.stdout
+
+    def test_mlxfwreset(self, cluster: List[OciInstance]):
+        """
+        Run mlxfwreset to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwreset -d {pci_addr} q")
+        logger.info("mlxfwreset query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxfwreset query"
+        assert "3: Driver restart and PCI reset" in r.stdout
+        assert "0: Tool is the owner" in r.stdout
+
+
+class TestOracleClusterPerformance:
+    """Test traffic performance between Oracle Cluster instances."""
+
+    @pytest.fixture(scope="class")
+    def private_vnic_cluster(
+        self,
+        cluster: List[OciInstance],
+    ) -> Generator[List[OciInstance], None, None]:
+        """
+        Cluster with private VNIC pair.
+
+        Yields:
+            List[OciInstance]: Instances of cluster with private VNIC.
+        """
+        ensure_second_vnics_ready(cluster)
+
+        yield cluster
+
+    def test_iperf3(self, private_vnic_cluster: List[OciInstance]):
+        """
+        Test iperf3 between two instances.
+
+        This tests the following:
+        - iperf3 successfully runs between two instances
+        - iperf3 throughput is greater than the minimum threshold (45.0)
+
+        Args:
+            private_vnic_cluster (List[OciInstance]): Cluster using private VNICs
+        """
+        min_throughput = 28.0
+        server_instance = private_vnic_cluster[0]
+        client_instance = private_vnic_cluster[1]
+
+        def start_server():
+            """Start the iperf3 server on the "server_instance"."""
+            server_instance.execute("iperf3 -s -1")
+
+        server_thread = threading.Thread(target=start_server)
+        server_thread.start()
+
+        # Wait for iperf3 server to start before starting the client
+        time.sleep(5)
+        r = client_instance.execute(
+            f"iperf3 -c {server_instance.secondary_vnic_private_ip} -P 40 -Z | grep SUM"
+        )
+        iperf3_output = r.stdout
+        logger.info("iperf3 output: %s", iperf3_output)
+        assert r.ok, "Failed to run iperf3"
+
+        throughput = iperf3_output.splitlines()[-1].split()[5]
+        print("iperf3 measured throughput: %s" % throughput)
+        assert float(throughput) > min_throughput


Reviews:
user: login: MitchellAugustin
body: The iperf3 tests are not passing for me on an image with iperf3 installed. Aside from that, everything else works as expected and looks good.

If the iperf tests work on someone else's known, correct configuration, please record that here.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T23:38:45Z
user: login: MitchellAugustin
body: Davide has verified that the iperf3 tests work on his image, and I have verified that all other tests work as expected and look good to me, so I think this should be good to merge.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-02-28T15:34:36Z
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR adds OCI example tests for OFED userspace tools and iperf3 while refactoring RDMA test logic.  
- Introduces a new helper function (ensure_second_vnics_ready) to verify secondary VNICs are present and refactors the RDMA test fixture.  
- Adds tests to validate the installation and basic output checks for mst, mlxconfig, mlxfwmanager, flint, mlxfwreset, and iperf3 performance.

### Reviewed Changes

| File                                               | Description                                                            |
| -------------------------------------------------- | ---------------------------------------------------------------------- |
| examples/oracle/oracle-example-cluster-test.py     | Refactored RDMA tests; added new tests for OFED CLI tools and iperf3      |

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**examples/oracle/oracle-example-cluster-test.py:129**
* Typo detected in the skip message, 'beiing' should be corrected to 'being'.
```
pytest.skip("The image beiing used is not RDMA ready")
```
</details>


state: COMMENTED
author_association: NONE
submitted_at: 2025-03-03T15:30:59Z
user: login: a-dubs
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:00:15Z
user: login: trentindav
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T18:08:44Z
user: login: a-dubs
body: changes look great. thanks for working with me to improve some of the various docstrings and organization of the code. :blue_heart: 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T13:49:50Z
user: login: uhryniuk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T14:27:54Z
user: login: trentindav
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T16:05:30Z
user: login: uhryniuk
body: Sorry for delayed comments, finishing my review now!
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T20:44:11Z
user: login: trentindav
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T21:09:57Z
user: login: trentindav
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T12:40:34Z
user: login: uhryniuk
body: lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T14:05:43Z

Review comments:
diff_hunk: @@ -295,3 +312,142 @@ def start_server():
         )
         logger.info("ucx_perftest output: %s", r.stdout)
         assert r.ok, "Failed to run ucx_perftest"
+
+
+class TestOracleClusterOfedTools:
+    """
+    Test Nvidia tools included in OFED userspace package.
+
+    Validate that CLI tools included in OFED are installed and executable.
+    Only verify query commands to avoid affecting the physical NIC firmware.
+    """
+
+    def test_mst_status(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mst status to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+
+        r = dut_instance.execute("sudo mst status")
+        logger.info("mst status output: %s", r.stdout)
+        assert r.ok, "Failed to run mst status"
+        assert "MST modules" in r.stdout
+        assert "PCI Devices" in r.stdout
+
+    def test_mlxconfig(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxconfig to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxconfig -d {pci_addr} q")
+        logger.info("mlxconfig query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxconfig query"
+        assert "ConnectX" in r.stdout
+
+    def test_mlxfwmanager(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwmanager to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwmanager -d {pci_addr} --query")
+        logger.info("mlxfwmanager query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxfwmanager query"
+        assert "ConnectX" in r.stdout
+        assert "Device Type:" in r.stdout
+        assert "Part Number:" in r.stdout
+
+    def test_flint(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run flint to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo flint -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
+        assert "Image type:" in r.stdout
+        assert "FW Version:" in r.stdout
+        assert "Product Version:" in r.stdout
+
+    def test_mlxfwreset(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwreset to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwreset -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
path: examples/oracle/oracle-example-cluster-test.py
user: login: Copilot
body: The log message in the mlxfwreset test incorrectly refers to 'flint'; please update it to reflect 'mlxfwreset' for clarity.
```suggestion
        logger.info("mlxfwreset query output: %s", r.stdout)
        assert r.ok, "Failed to run mlxfwreset query"
```
created_at: 2025-03-03T15:30:58Z
author_association: NONE
diff_hunk: @@ -295,3 +312,142 @@ def start_server():
         )
         logger.info("ucx_perftest output: %s", r.stdout)
         assert r.ok, "Failed to run ucx_perftest"
+
+
+class TestOracleClusterOfedTools:
+    """
+    Test Nvidia tools included in OFED userspace package.
+
+    Validate that CLI tools included in OFED are installed and executable.
+    Only verify query commands to avoid affecting the physical NIC firmware.
+    """
+
+    def test_mst_status(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mst status to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+
+        r = dut_instance.execute("sudo mst status")
+        logger.info("mst status output: %s", r.stdout)
+        assert r.ok, "Failed to run mst status"
+        assert "MST modules" in r.stdout
+        assert "PCI Devices" in r.stdout
+
+    def test_mlxconfig(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxconfig to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxconfig -d {pci_addr} q")
+        logger.info("mlxconfig query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxconfig query"
+        assert "ConnectX" in r.stdout
+
+    def test_mlxfwmanager(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwmanager to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwmanager -d {pci_addr} --query")
+        logger.info("mlxfwmanager query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxfwmanager query"
+        assert "ConnectX" in r.stdout
+        assert "Device Type:" in r.stdout
+        assert "Part Number:" in r.stdout
+
+    def test_flint(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run flint to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo flint -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
+        assert "Image type:" in r.stdout
+        assert "FW Version:" in r.stdout
+        assert "Product Version:" in r.stdout
+
+    def test_mlxfwreset(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwreset to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwreset -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
path: examples/oracle/oracle-example-cluster-test.py
user: login: a-dubs
body: @trentindav holy crap it actually caught something :laughing: 
created_at: 2025-03-03T16:00:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -295,3 +312,142 @@ def start_server():
         )
         logger.info("ucx_perftest output: %s", r.stdout)
         assert r.ok, "Failed to run ucx_perftest"
+
+
+class TestOracleClusterOfedTools:
+    """
+    Test Nvidia tools included in OFED userspace package.
+
+    Validate that CLI tools included in OFED are installed and executable.
+    Only verify query commands to avoid affecting the physical NIC firmware.
+    """
+
+    def test_mst_status(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mst status to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+
+        r = dut_instance.execute("sudo mst status")
+        logger.info("mst status output: %s", r.stdout)
+        assert r.ok, "Failed to run mst status"
+        assert "MST modules" in r.stdout
+        assert "PCI Devices" in r.stdout
+
+    def test_mlxconfig(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxconfig to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxconfig -d {pci_addr} q")
+        logger.info("mlxconfig query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxconfig query"
+        assert "ConnectX" in r.stdout
+
+    def test_mlxfwmanager(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwmanager to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwmanager -d {pci_addr} --query")
+        logger.info("mlxfwmanager query output: %s", r.stdout)
+        assert r.ok, "Failed to run mlxfwmanager query"
+        assert "ConnectX" in r.stdout
+        assert "Device Type:" in r.stdout
+        assert "Part Number:" in r.stdout
+
+    def test_flint(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run flint to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo flint -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
+        assert "Image type:" in r.stdout
+        assert "FW Version:" in r.stdout
+        assert "Product Version:" in r.stdout
+
+    def test_mlxfwreset(self, cluster: List[OciInstance]):  # pylint: disable=W0621
+        """
+        Run mlxfwreset to confirm it is installed.
+
+        Args:
+            cluster (List[OciInstance]): cluster instances
+        """
+        dut_instance = cluster[0]
+        pci_addr = get_private_nic_pci_address(dut_instance)
+
+        r = dut_instance.execute(f"sudo mlxfwreset -d {pci_addr} q")
+        logger.info("flint query output: %s", r.stdout)
+        assert r.ok, "Failed to run flint query"
path: examples/oracle/oracle-example-cluster-test.py
user: login: trentindav
body: good it caught it! fixed it together with the "low confidence" suggestion
created_at: 2025-03-03T18:08:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +126,52 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance):
path: examples/oracle/oracle-example-cluster-test.py
user: login: uhryniuk
body: nit: missing return type annotation. 'str' is listed in the doc.
created_at: 2025-03-13T14:27:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +126,52 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance):
path: examples/oracle/oracle-example-cluster-test.py
user: login: trentindav
body: Nice catch, I am taking the opportunity to improve this and add the return type for other functions that have one. 
created_at: 2025-03-13T16:05:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +126,52 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        test_cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance) -> str:
+    """
+    Get the PCI address of the second NIC on the instance (mlx5_1) which is used for RDMA.
+
+    Args:
+        instance (OciInstance): The instance to get the PCI address from.
+
+    Returns:
+        str: The PCI address of the second NIC.
+    """
+    r = instance.execute("sudo mst status -v | grep mlx5_1")
+    pciaddr = r.stdout.split()[2]
path: examples/oracle/oracle-example-cluster-test.py
user: login: uhryniuk
body: Would you mind adding either a comment/pydoc or maybe some error handling around pciaddr parsing?

It'd be pretty tedious crawling through a traceback on an IndexError.
created_at: 2025-03-13T20:43:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +126,52 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        test_cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance) -> str:
+    """
+    Get the PCI address of the second NIC on the instance (mlx5_1) which is used for RDMA.
+
+    Args:
+        instance (OciInstance): The instance to get the PCI address from.
+
+    Returns:
+        str: The PCI address of the second NIC.
+    """
+    r = instance.execute("sudo mst status -v | grep mlx5_1")
+    pciaddr = r.stdout.split()[2]
path: examples/oracle/oracle-example-cluster-test.py
user: login: trentindav
body: right, will do that! I'll add an assertion on the validity of the retrieved pciaddr value with a meaningful error message
created_at: 2025-03-13T21:09:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -126,7 +126,52 @@ def ensure_image_is_rdma_ready(instance: OciInstance):
     r = instance.execute("ibstatus")
     if not r.stdout or not r.ok:
         logger.info("Infiniband status: %s", r.stdout + "\n" + r.stderr)
-        pytest.skip("The image beiing used is not RDMA ready")
+        pytest.skip("The image being used is not RDMA ready")
+
+
+def ensure_second_vnics_ready(test_cluster: List[OciInstance]):
+    """
+    Check if all cluster instances have a secondary VNIC and attach and configure one if not.
+
+    If the instance already has a secondary VNIC, it will skip the attachment process.
+
+    Otherwise, it will do the following to set up the secondary VNIC:
+        - Attach a secondary VNIC to the instance
+        - Configure the secondary VNIC using information from the IMDS
+        - Set up the iptables rules on the appropriate NIC for RDMA usage
+
+    Args:
+        test_cluster (List[OciInstance]): The cluster (list of instances) to check and configure.
+    """
+    for instance in test_cluster:
+        if instance.secondary_vnic_private_ip:
+            logger.info(
+                "Instance %s already has a secondary VNIC, not attaching one.", instance.name
+            )
+            continue
+        logger.info("Creating a secondary VNIC on instance %s", instance.name)
+        # create a secondary VNIC on the 2nd vnic on the private subnet for RDMA usage
+        instance.add_network_interface(
+            nic_index=1,
+            subnet_name="private subnet-mofed-vcn",  # use the private subnet for mofed testing
+        )
+        instance.configure_secondary_vnic()
+        setup_mofed_iptables_rules(instance)
+
+
+def get_private_nic_pci_address(instance: OciInstance) -> str:
+    """
+    Get the PCI address of the second NIC on the instance (mlx5_1) which is used for RDMA.
+
+    Args:
+        instance (OciInstance): The instance to get the PCI address from.
+
+    Returns:
+        str: The PCI address of the second NIC.
+    """
+    r = instance.execute("sudo mst status -v | grep mlx5_1")
+    pciaddr = r.stdout.split()[2]
path: examples/oracle/oracle-example-cluster-test.py
user: login: trentindav
body: just a heads-up that the change has been committed
created_at: 2025-03-19T12:40:34Z
author_association: CONTRIBUTOR

PR comments:
user: login: trentindav
created_at: 2025-02-28T14:43:39Z
body: I am not able to reproduce the `test_iperf3` failure. I spawned a new cluster network and executed these steps on both hosts before running the test:

1. Add second VNIC
2. install MOFED DKMS and userspace PPA 
3. `sudo apt install -y iperf3 rdmacm-utils ucx-utils`
4. reboot
5. run test adding the two instances OCIDs in `EXISTING_INSTANCE_IDS`

```bash
tox -e integration-tests -- examples/oracle/oracle-example-cluster-test.py -k "TestOracleClusterPerformance"
integration-tests: commands[0]> .tox/integration-tests/bin/python -m pytest --log-cli-level=INFO -svv examples/oracle/oracle-example-cluster-test.py -k TestOracleClusterPerformance
========================================================================================================================= test session starts ==========================================================================================================================
platform linux -- Python 3.12.3, pytest-8.3.4, pluggy-1.5.0 -- /home/davide/repo/tools/pycloudlib/.tox/integration-tests/bin/python
cachedir: .tox/integration-tests/.pytest_cache
rootdir: /home/davide/repo/tools/pycloudlib
configfile: pyproject.toml
plugins: xdist-3.6.1, mock-3.14.0, cov-6.0.0
collecting ... 
------------------------------------------------------------------------------------------------------------------------- live log collection --------------------------------------------------------------------------------------------------------------------------
INFO     oci.circuit_breaker:__init__.py:27 Default Auth client Circuit breaker strategy enabled
collected 12 items / 11 deselected / 1 selected                                                                                                                                                                                                                        

examples/oracle/oracle-example-cluster-test.py::TestOracleClusterPerformance::test_iperf3 
---------------------------------------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.cloud.OCI:cloud.py:354 No public key path provided, using: /home/davide/.ssh/id_ed25519.pub
INFO     oracle-example-cluster-test:oracle-example-cluster-test.py:143 Instance ocid1.instance.oc1.phx.anyhqljsniwq6syc5nsrp6gpjsdamswpjs6aluliyypm2v5k25h3ackuc3ia already has a secondary VNIC, not attaching one.
INFO     oracle-example-cluster-test:oracle-example-cluster-test.py:143 Instance ocid1.instance.oc1.phx.anyhqljsniwq6sycbiiymnuolvlp3hm7iryh5xr5rwzqh4vzz74gpkp43tea already has a secondary VNIC, not attaching one.
---------------------------------------------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------------------------------------------
INFO     pycloudlib.instance:instance.py:285 executing: sh -c 'iperf3 -s -1'
INFO     pycloudlib.instance:instance.py:106 Using ipv4 address: 129.146.167.132
INFO     paramiko.transport:transport.py:1944 Connected (version 2.0, client OpenSSH_9.6p1)
INFO     paramiko.transport:transport.py:1944 Authentication (publickey) successful!
INFO     pycloudlib.instance:instance.py:285 executing: sh -c 'iperf3 -c 10.0.1.99 -P 40 -Z | grep SUM'
INFO     pycloudlib.instance:instance.py:106 Using ipv4 address: 129.146.4.55
INFO     paramiko.transport:transport.py:1944 Connected (version 2.0, client OpenSSH_9.6p1)
INFO     paramiko.transport:transport.py:1944 Authentication (publickey) successful!
INFO     oracle-example-cluster-test:oracle-example-cluster-test.py:450 iperf3 output: [SUM]   0.00-1.00   sec  5.39 GBytes  46.3 Gbits/sec  3326             
[SUM]   1.00-2.00   sec  5.36 GBytes  46.0 Gbits/sec  2993             
[SUM]   2.00-3.00   sec  5.36 GBytes  46.0 Gbits/sec  2759             
[SUM]   3.00-4.00   sec  5.35 GBytes  46.0 Gbits/sec  2745             
[SUM]   4.00-5.00   sec  5.36 GBytes  46.0 Gbits/sec  2633             
[SUM]   5.00-6.00   sec  5.35 GBytes  46.0 Gbits/sec  2931             
[SUM]   6.00-7.00   sec  5.36 GBytes  46.0 Gbits/sec  2641             
[SUM]   7.00-8.00   sec  5.35 GBytes  46.0 Gbits/sec  2662             
[SUM]   8.00-9.00   sec  5.36 GBytes  46.0 Gbits/sec  2883             
[SUM]   9.00-10.01  sec  5.36 GBytes  45.8 Gbits/sec  2857             
[SUM]   0.00-10.01  sec  53.6 GBytes  46.0 Gbits/sec  28430             sender
[SUM]   0.00-10.01  sec  53.6 GBytes  46.0 Gbits/sec                  receiver
iperf3 measured throughput: 46.0
PASSED

=========================================================================================================================== warnings summary ===========================================================================================================================
examples/oracle/oracle-example-cluster-test.py: 132 warnings
  /home/davide/repo/tools/pycloudlib/.tox/integration-tests/lib/python3.12/site-packages/oci/base_client.py:77: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    return " " + str(datetime.utcnow()) + ": "

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================================================== 1 passed, 11 deselected, 132 warnings in 23.71s ============================================================================================================
  integration-tests: OK (24.44=setup[0.04]+cmd[24.41] seconds)
  congratulations :) (24.50 seconds)
```
author_association: CONTRIBUTOR
user: login: a-dubs
created_at: 2025-03-03T14:42:22Z
body: @MitchellAugustin thank you for reviewing this!
And thank you @trentindav for putting this PR up. love to see partner engineering team is taking to pycloudlib :raised_hands: 
I will review this later today and have some other CPC folks give this a look
author_association: CONTRIBUTOR
user: login: trentindav
created_at: 2025-03-14T13:03:41Z
body: @MitchellAugustin @a-dubs while validating the latest version of the tests I noticed that I cannot reach the same `iperf3` performance I consistently noticed when I first implemented them (I get ~30Gbps compared to the ~45Gbps previously measured). I am using the same custom image as I did during previous tests, so the origin of this difference may be a change in HW or network load.
I am reducing the accepted threshold to let  `TestOracleClusterPerformance::test_iperf3` pass, unless you believe it this is an issue worth investigating.
author_association: CONTRIBUTOR
user: login: MitchellAugustin
created_at: 2025-03-14T13:16:17Z
body: @trentindav 
> I am reducing the accepted threshold to let TestOracleClusterPerformance::test_iperf3 pass, unless you believe it this is an issue worth investigating

I think it is OK to lower the accepted threshold here since we aren't tailoring this test to reach line rate for any specific device.

In general though, to reach line rate via iperf3 at speeds above 40Gbps, you may need to use multiple iperf3 streams/processes. This is something that we need to do when testing the DGXes at 100Gbps+, since individual iperf3 threads aren't necessarily capable of rates that high.
author_association: MEMBER
user: login: trentindav
created_at: 2025-03-14T13:21:53Z
body: @MitchellAugustin  I am currently using the `-P` to start 40 parallel streams in the iperf3 client and this was sufficient to reach 45Gbps. I am also using the `-Z` (zerocopy) option which seemed to help getting more consistent results. 
author_association: CONTRIBUTOR
user: login: MitchellAugustin
created_at: 2025-03-14T13:26:14Z
body: > @MitchellAugustin I am currently using the `-P` to start 40 parallel streams in the iperf3 client and this was sufficient to reach 45Gbps. I am also using the `-Z` (zerocopy) option which seemed to help getting more consistent results.

ah sorry, somehow I missed that
author_association: MEMBER