Repository: Mellanox/libxlio
Pull request #350: [CI] migrate CI test steps to containers - Valgrind
Author: login: NirWolfer
Labels: name: Review effort 4/5
Created: 2025-05-07T13:48:23Z | Merged: 2025-05-15T06:12:15Z

## Description
Today we use benni09 static agent to run test/gtest/valgrind steps which is unscaleable since it can only run one pipeline at a time, causing delays in builds that can be stuck waiting for hours

The idea is to move these steps to containers, allowing running them in parallel as well as running multiple pipelines at the same time (depending on the capacity of the k8s cluster)

##### What
This commit moves Valgrind away from static agent to container running on k8s

##### Why ?
Issue: [HPCINFRA-3249](https://jirasw.nvidia.com/browse/HPCINFRA-3249)
Issue: [HPCINFRA-3559](https://jirasw.nvidia.com/browse/HPCINFRA-3559)

## Change type
What kind of change does this PR introduce?
- [ ] Bugfix
- [ ] Feature
- [ ] Code style update
- [ ] Refactoring (no functional changes, no api changes)
- [ ] Build related changes
- [x] CI related changes
- [ ] Documentation content changes
- [ ] Tests
- [ ] Other

## Check list
- [ ] Code follows the style de facto guidelines of this project
- [ ] Comments have been inserted in hard to understand places
- [ ] Documentation has been updated (if necessary)
- [ ] Test has been added (if possible)




Diff:
diff --git a/.ci/dockerfiles/Dockerfile.ubuntu22.04 b/.ci/dockerfiles/Dockerfile.ubuntu22.04
index 4404f2f34..533bc91b7 100644
--- a/.ci/dockerfiles/Dockerfile.ubuntu22.04
+++ b/.ci/dockerfiles/Dockerfile.ubuntu22.04
@@ -4,7 +4,22 @@ RUN apt-get update \
  && apt-get install -y libjson-c-dev \
  && apt-get clean && rm -rf /var/lib/apt/lists/*
 
-FROM build as style
+# Base for all tests (Gtest, Valgrind, test), installs utilities.
+FROM build AS tests
+RUN apt-get update && \
+    apt-get install -y \
+    net-tools unzip iproute2 wget \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Valgrind target, installs valgrind
+FROM tests AS vg
+RUN apt-get update && \
+    apt-get install -y \
+    valgrind \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Style target, installs clang-format-15
+FROM build AS style
 RUN apt-get update \
  && apt-get install -y clang-15 clang-format-15 \
  && update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-15 100 \
diff --git a/.ci/matrix_job.yaml b/.ci/matrix_job.yaml
index 4968ac6ca..8a6b59d12 100644
--- a/.ci/matrix_job.yaml
+++ b/.ci/matrix_job.yaml
@@ -12,8 +12,8 @@ kubernetes:
   cloud: il-ipp-blossom-prod
   nodeSelector: 'beta.kubernetes.io/os=linux'
   namespace: swx-media
-  limits: '{memory: 8Gi, cpu: 7000m}'
-  requests: '{memory: 8Gi, cpu: 7000m}'
+  limits: '{memory: 10Gi, cpu: 10000m}'
+  requests: '{memory: 10Gi, cpu: 10000m}'
 
 credentials:
   - {credentialsId: 'media_coverity_credentials', usernameVariable: 'XLIO_COV_USER', passwordVariable: 'XLIO_COV_PASSWORD'}
@@ -107,6 +107,24 @@ runs_on_dockers:
      build_args: '--no-cache --target static',
      category: 'tool'
      }
+# tests
+  - {
+     file: '.ci/dockerfiles/Dockerfile.ubuntu22.04',
+     arch: 'x86_64',
+     name: 'vg',
+     uri: 'xlio/$arch/ubuntu22.04/$name',
+     tag: '20250219',
+     build_args: '--no-cache --target vg',
+     category: 'tool',
+     annotations: [{ key: 'k8s.v1.cni.cncf.io/networks', value: 'sriov-cx6dx-p2' }],
+     limits: '{memory: 10Gi, cpu: 10000m, hugepages-2Mi: 10Gi, mellanox.com/sriov_cx6dx_p2: 1}',
+     requests: '{memory: 10Gi, cpu: 10000m, hugepages-2Mi: 10Gi, mellanox.com/sriov_cx6dx_p2: 1}',
+     caps_add: '[ IPC_LOCK, SYS_RESOURCE ]',
+     runAsUser: '0',
+     runAsGroup: '0',
+     cloud: swx-k8s-spray,
+     namespace: xlio-ci
+    }
 
 runs_on_agents:
   - {nodeLabel: 'beni09', category: 'base'}
@@ -142,6 +160,7 @@ steps:
   - name: Install Doca-host
     containerSelector:
       - "{category: 'base'}"
+      - "{name: 'vg'}"
     agentSelector:
       - "{nodeLabel: 'skip-agent'}"
     run: |
@@ -340,9 +359,9 @@ steps:
   - name: Valgrind
     enable: ${do_valgrind}
     containerSelector:
-      - "{name: 'skip-container'}"
+      - "{name: 'vg'}"
     agentSelector:
-      - "{nodeLabel: 'beni09'}"
+      - "{nodeLabel: 'skip-agent'}"
     run: |
       [ "x${do_valgrind}" == "xtrue" ] && action=yes || action=no
       env WORKSPACE=$PWD TARGET=${flags} jenkins_test_vg=${action} ./contrib/test_jenkins.sh
@@ -370,6 +389,11 @@ steps:
 
   - name: Artifacts
     enable: ${do_artifact}
+    containerSelector:
+      - "{category: 'base'}"
+      - "{name: 'vg'}"
+    agentSelector:
+      - "{nodeLabel: 'beni09'}"
     run: |
       ./.ci/artifacts.sh
     parallel: false
@@ -403,4 +427,4 @@ pipeline_stop:
 
 failFast: false
 
-taskName: '${flags}/${name}/${axis_index}'
+taskName: '${flags}/${arch}/${name}/${axis_index}'
diff --git a/contrib/jenkins_tests/globals.sh b/contrib/jenkins_tests/globals.sh
index b06b9ae7c..d9bcd5eb5 100755
--- a/contrib/jenkins_tests/globals.sh
+++ b/contrib/jenkins_tests/globals.sh
@@ -82,6 +82,17 @@ function do_archive()
 # Otherwise, return error code.
 # $1 - module name
 #
+
+function do_hugepages()
+{
+    if [[ (-f /.dockerenv || -f /run/.containerenv || -n "${KUBERNETES_SERVICE_HOST}") ]] && ! grep -q hugetlbfs /proc/mounts; then
+        mkdir -p /mnt/huge
+        mount -t hugetlbfs nodev /mnt/huge
+        grep hugetlbfs /proc/mounts
+        echo $?
+    fi
+}
+
 function do_module()
 {
     [ -z "$1" ] && return
diff --git a/contrib/jenkins_tests/vg.sh b/contrib/jenkins_tests/vg.sh
index 7a0cf118e..c41bcbbeb 100755
--- a/contrib/jenkins_tests/vg.sh
+++ b/contrib/jenkins_tests/vg.sh
@@ -2,6 +2,9 @@
 
 source $(dirname $0)/globals.sh
 
+# Fix hugepages for docker environments
+do_hugepages
+
 echo "Checking for valgrind ..."
 
 #do_module "tools/valgrind-3.12.0"
@@ -19,17 +22,29 @@ make $make_opt all
 make install
 rc=$?
 
-
-test_ip_list=""
-#if [ ! -z $(do_get_ip 'ib') ]; then
-#	test_ip_list="${test_ip_list} ib:$(do_get_ip 'ib')"
-#fi
-if [ ! -z "$(do_get_ip 'eth')" ]; then
-	test_ip_list="${test_ip_list} eth_ip4:$(do_get_ip 'eth')"
+if [[ -f /.dockerenv ]] || [[ -f /run/.containerenv ]] || [[ -n "${KUBERNETES_SERVICE_HOST}" ]]; then
+    if ip link show net1 > /dev/null 2>&1; then
+        test_ip_list="eth_ip4:$(ip -f inet addr show net1 | awk '/inet / {print $2}' | cut -d/ -f1)"
+        test_ip_list="${test_ip_list} eth_ip6:$(ip -f inet6 addr show net1 | grep global | awk '/inet6 / {print $2}' | cut -d/ -f1)"
+    else
+        echo "ERROR: net1 interface does not exist!"
+        exit 1
+    fi
+else
+	test_ip_list=""
+	if [ ! -z "$(do_get_ip 'eth')" ]; then
+		test_ip_list="${test_ip_list} eth_ip4:$(do_get_ip 'eth')"
+	fi
+	if [ ! -z "$(do_get_ip 'eth')" ]; then
+		test_ip_list="${test_ip_list} eth_ip6:$(do_get_ip 'inet6')"
+	fi
 fi
-if [ ! -z "$(do_get_ip 'eth')" ]; then
-	test_ip_list="${test_ip_list} eth_ip6:$(do_get_ip 'inet6')"
+
+if [ "$test_ip_list" == "eth_ip4: eth_ip6:" ] || [ -z "${test_ip_list}" ]; then
+	echo "ERROR: Cannot get IPv4/6 address of net1 interface!"
+	exit 1
 fi
+
 test_list="tcp:--tcp udp:"
 test_lib=${vg_dir}/install/lib/${prj_lib}
 test_lib_env="XLIO_MEM_ALLOC_TYPE=ANON XLIO_MEMORY_LIMIT=256MB XLIO_TX_WRE=2000 XLIO_RX_WRE=2000 XLIO_STRQ=off"


Reviews:
user: login: dpressle
body: I have general concern, when we run on beni we always got the logs for vg in beni tar.gz (even when all processes pass) on container we will only get logs when VG fails, i want you to check that we do get the logs when valgrind fails on container
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T06:03:33Z
user: login: dpressle
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-11T14:31:43Z
user: login: dpressle
body: See pr-agent review and improve suggestion
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T06:05:44Z
user: login: NirWolfer
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T07:26:38Z
user: login: dpressle
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-13T08:22:20Z

Review comments:
diff_hunk: @@ -4,7 +4,18 @@ RUN apt-get update \
  && apt-get install -y libjson-c-dev \
  && apt-get clean && rm -rf /var/lib/apt/lists/*
 
-FROM build as style
+FROM build AS tests
+RUN apt-get update && \
+    apt-get install -y \
+    net-tools unzip iproute2 wget \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+FROM tests AS vg
+RUN apt-get update && \
path: .ci/dockerfiles/Dockerfile.ubuntu22.04
user: login: dpressle
body: add new space line between these targets
created_at: 2025-05-08T06:01:18Z
author_association: COLLABORATOR
diff_hunk: @@ -4,7 +4,18 @@ RUN apt-get update \
  && apt-get install -y libjson-c-dev \
  && apt-get clean && rm -rf /var/lib/apt/lists/*
 
-FROM build as style
+FROM build AS tests
path: .ci/dockerfiles/Dockerfile.ubuntu22.04
user: login: dpressle
body: can you add some comment/description for each new target you added?
created_at: 2025-05-08T06:01:38Z
author_association: COLLABORATOR
diff_hunk: @@ -82,6 +82,17 @@ function do_archive()
 # Otherwise, return error code.
 # $1 - module name
 #
+
+function do_hugepages()
+{
+    if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then
path: contrib/jenkins_tests/globals.sh
user: login: dpressle
body: I dont think this is enough for Blossom, see how we check for k8s env in vg.sh line 25
created_at: 2025-05-13T06:05:23Z
author_association: COLLABORATOR
diff_hunk: @@ -82,6 +82,17 @@ function do_archive()
 # Otherwise, return error code.
 # $1 - module name
 #
+
+function do_hugepages()
+{
+    if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then
path: contrib/jenkins_tests/globals.sh
user: login: NirWolfer
body: added the rest of the checks
`if [[ (-f /.dockerenv || -f /run/.containerenv || -n "${KUBERNETES_SERVICE_HOST}") && ! grep -q hugetlbfs /proc/mounts ]]; then`
created_at: 2025-05-13T07:26:38Z
author_association: COLLABORATOR

PR comments:
user: login: NirWolfer
created_at: 2025-05-07T16:15:45Z
body: bot:retest
author_association: COLLABORATOR
user: login: dpressle
created_at: 2025-05-12T21:44:32Z
body: /review
author_association: COLLABORATOR
user: login: pr-review-bot-app[bot]
created_at: 2025-05-12T21:45:06Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 4 🔵🔵🔵🔵⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/Mellanox/libxlio/pull/350/files#diff-bd20dae0683f0461e07269794e1958b1642755191eac9977317668ac74536568R86-R94'><strong>Possible Issue</strong></a>

The `do_hugepages` function assumes that the `hugetlbfs` mount is not present in Docker environments and attempts to mount it. This could lead to unexpected behavior or errors if the mount already exists or if permissions are insufficient.
</summary>

```shell
function do_hugepages()
{
    if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then
        mkdir -p /mnt/huge
        mount -t hugetlbfs nodev /mnt/huge
        grep hugetlbfs /proc/mounts
        echo $?
    fi
}
```

</details>

<details><summary><a href='https://github.com/Mellanox/libxlio/pull/350/files#diff-33b6c4544dea5b541abe7fa2e8c3c91dce8ba1cef74b61481865f517ac77e950R25-R40'><strong>Error Handling</strong></a>

The logic for determining `test_ip_list` in container environments may fail silently if `do_get_ip` does not return valid IPs. Consider adding error handling or logging to ensure issues are detected.
</summary>

```shell
if [[ -f /.dockerenv ]] || [[ -f /run/.containerenv ]] || [[ -n "${KUBERNETES_SERVICE_HOST}" ]]; then
	test_ip_list="eth_ip4:$(ip -f inet addr show net1 | awk '/inet / {print $2}' | cut -d/ -f1)"
	test_ip_list="${test_ip_list} eth_ip6:$(ip -f inet6 addr show net1 | grep global | awk '/inet6 / {print $2}' | cut -d/ -f1)"
else
	test_ip_list=""
	if [ ! -z "$(do_get_ip 'eth')" ]; then
		test_ip_list="${test_ip_list} eth_ip4:$(do_get_ip 'eth')"
	fi
	if [ ! -z "$(do_get_ip 'eth')" ]; then
		test_ip_list="${test_ip_list} eth_ip6:$(do_get_ip 'inet6')"
	fi
fi

if [ "$test_ip_list" == "eth_ip4: eth_ip6:" ] || [ -z "${test_ip_list}" ]; then
	echo "ERROR: Cannot get IPv4/6 address of net1 interface!"
	exit 1
```

</details>

<details><summary><a href='https://github.com/Mellanox/libxlio/pull/350/files#diff-b4437ff6080723c747d363225b9c61172d642414929dbd3c21c8fa783e053fccR7-R19'><strong>Resource Usage</strong></a>

The new Dockerfile stages for Valgrind and tests significantly increase memory and CPU limits. Ensure these limits are justified and do not exceed the capacity of the Kubernetes cluster.
</summary>

```txt
# Base for all tests (Gtest, Valgrind, test), installs utilities.
FROM build AS tests
RUN apt-get update && \
    apt-get install -y \
    net-tools unzip iproute2 wget \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Valgrind target, installs valgrind
FROM tests AS vg
RUN apt-get update && \
    apt-get install -y \
    valgrind \
    && apt-get clean && rm -rf /var/lib/apt/lists/*
```

</details>

</td></tr>
</table>

author_association: NONE
user: login: dpressle
created_at: 2025-05-12T21:49:16Z
body: /improve
author_association: COLLABORATOR
user: login: pr-review-bot-app[bot]
created_at: 2025-05-12T21:49:33Z
body: ## PR Code Suggestions ✨

<!-- c415375 -->

<table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=2>Possible issue</td>
<td>



<details><summary>Validate network interface existence</summary>

___


**Validate the existence of the <code>net1</code> interface before attempting to retrieve IP <br>addresses to prevent potential errors if the interface does not exist.**

[contrib/jenkins_tests/vg.sh [26-27]](https://github.com/Mellanox/libxlio/pull/350/files#diff-33b6c4544dea5b541abe7fa2e8c3c91dce8ba1cef74b61481865f517ac77e950R26-R27)

```diff
-test_ip_list="eth_ip4:$(ip -f inet addr show net1 | awk '/inet / {print $2}' | cut -d/ -f1)"
-test_ip_list="${test_ip_list} eth_ip6:$(ip -f inet6 addr show net1 | grep global | awk '/inet6 / {print $2}' | cut -d/ -f1)"
+if ip link show net1 > /dev/null 2>&1; then
+    test_ip_list="eth_ip4:$(ip -f inet addr show net1 | awk '/inet / {print $2}' | cut -d/ -f1)"
+    test_ip_list="${test_ip_list} eth_ip6:$(ip -f inet6 addr show net1 | grep global | awk '/inet6 / {print $2}' | cut -d/ -f1)"
+else
+    echo "ERROR: net1 interface does not exist!"
+    exit 1
+fi
```
<details><summary>Suggestion importance[1-10]: 9</summary>

__

Why: The suggestion adds a critical validation step to check if the `net1` interface exists before attempting to retrieve IP addresses. This prevents potential runtime errors and ensures the script behaves predictably in environments where `net1` may not be present.


</details></details></td><td align=center>High

</td></tr><tr><td>



<details><summary>Fix incorrect <code>grep</code> usage logic</summary>

___


**Ensure the <code>grep</code> command is correctly used to check for <code>hugetlbfs</code> in <code>/proc/mounts</code> by <br>removing the unnecessary negation and properly handling the command's exit status.**

[contrib/jenkins_tests/globals.sh [88-93]](https://github.com/Mellanox/libxlio/pull/350/files#diff-bd20dae0683f0461e07269794e1958b1642755191eac9977317668ac74536568R88-R93)

```diff
-if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then
+if [[ -f /.dockerenv && ! grep -q hugetlbfs /proc/mounts ]]; then
     mkdir -p /mnt/huge
     mount -t hugetlbfs nodev /mnt/huge
     grep hugetlbfs /proc/mounts
     echo $?
 fi
```
<details><summary>Suggestion importance[1-10]: 8</summary>

__

Why: The suggestion correctly identifies and fixes the improper use of `grep` with a negation, ensuring the logic works as intended. This improves the reliability of the code when checking for `hugetlbfs` in `/proc/mounts`.


</details></details></td><td align=center>Medium

</td></tr></tr></tbody></table>


author_association: NONE
user: login: dpressle
created_at: 2025-05-13T08:02:05Z
body: /ask what is the difference between these lines of code
if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then
if [[ -f /.dockerenv && ! grep -q hugetlbfs /proc/mounts ]]; then
author_association: COLLABORATOR
user: login: dpressle
created_at: 2025-05-13T08:04:41Z
body: /ask "what is the difference between these lines of code: if [[ -f /.dockerenv && ! $(grep -q hugetlbfs /proc/mounts) ]]; then if [[ -f /.dockerenv && ! grep -q hugetlbfs /proc/mounts ]]; then"
author_association: COLLABORATOR