Repository: kernelci/kernelci-core
Pull request #2865: Add fault injection template
Author: login: crazoes
Created: 2025-04-30T08:56:52Z | Merged: 2025-05-12T10:08:14Z

Add a basic fault injection test template which will be currently used to add a suspend/resume test.


Diff:
diff --git a/config/core/rootfs-configs.yaml b/config/core/rootfs-configs.yaml
index 6b5a50f7f6..f2d5b8149b 100644
--- a/config/core/rootfs-configs.yaml
+++ b/config/core/rootfs-configs.yaml
@@ -117,6 +117,15 @@ rootfs_configs:
     script: "scripts/bookworm-cros-ec-tests.sh"
     test_overlay: ""
 
+  bookworm-fault-injection:
+    rootfs_type: debos
+    debian_release: bookworm
+    arch_list:
+      - amd64
+      - arm64
+      - armhf
+    script: "scripts/bookworm-fault-injection.sh"
+
   bookworm-gst-fluster:
     rootfs_type: debos
     debian_release: bookworm
diff --git a/config/rootfs/debos/scripts/bookworm-fault-injection.sh b/config/rootfs/debos/scripts/bookworm-fault-injection.sh
new file mode 100755
index 0000000000..f23350ffc5
--- /dev/null
+++ b/config/rootfs/debos/scripts/bookworm-fault-injection.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+set -ex
+
+PACKAGES_INSTALL="\
+      ca-certificates \
+      wget \
+"
+
+apt-get install --no-install-recommends -y ${PACKAGES_INSTALL}
+
+# Fetch and install failcmd
+wget https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/testing/fault-injection/failcmd.sh -O /usr/local/bin/failcmd
+chmod +x /usr/local/bin/failcmd
+
+# Basic validation
+if [ ! -x /usr/local/bin/failcmd ]; then
+    echo "failcmd installation failed" >&2
+    exit 1
+fi
+
+apt-get remove --purge -y ${PACKAGES_INSTALL}
+apt-get autoremove --purge -y
+apt-get clean
diff --git a/config/runtime/tests/fault-injection.jinja2 b/config/runtime/tests/fault-injection.jinja2
new file mode 100644
index 0000000000..d168e41528
--- /dev/null
+++ b/config/runtime/tests/fault-injection.jinja2
@@ -0,0 +1,32 @@
+- test:
+    timeout:
+      minutes: {{ job_timeout|default(20) }}
+    definitions:
+    - from: inline
+      name: "fault-injection-{{ test_name }}"
+      path: inline/fault-injection.yaml
+      repository:
+        metadata:
+          format: Lava-Test Test Definition 1.0
+          name: fault-injection
+          description: |
+             Fault injection: {{ test_name }}
+             Type: {{ fail_type|default('failslab') }} | Iterations: {{ iterations|default(5) }}
+          os:
+          - debian
+          scope:
+          - functional
+        run:
+          steps:
+            - |
+              for i in $(seq 1 {{ iterations|default(5) }}); do
+                echo "Iteration $i: {{ test_name }}"
+                FAILCMD_TYPE={{ fail_type|default('failslab') }} \
+                  /usr/local/bin/failcmd \
+                  --probability={{ probability|default(100) }} \
+                  --times={{ times|default(100) }} \
+                  --interval={{ interval|default(100) }} \
+                  --verbose=2 \
+                  "{{ test_command }}"
+                sleep {{ sleep_time|default(10) }}
+              done


Reviews:
user: login: laura-nao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T08:10:36Z
user: login: laura-nao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T08:12:51Z
user: login: nuclearcat
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T08:44:01Z
user: login: laura-nao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T09:34:52Z

Review comments:
diff_hunk: @@ -0,0 +1,32 @@
+- test:
+    timeout:
+      minutes: {{ job_timeout|default(20) }}
+    definitions:
+    - from: inline
+      name: "fault-injection-{{ test_name }}"
+      path: inline/fault-injection.yaml
+      repository:
+        metadata:
+          format: Lava-Test Test Definition 1.0
+          name: fault-injection
+          description: "Fault injection test: {{ test_name }}"
+          os:
+          - debian
+          scope:
+          - functional
+        run:
+          steps:
+            - wget {{ failcmd_url }} -O /tmp/failcmd.sh
path: config/runtime/tests/fault-injection.jinja2
user: login: laura-nao
body: We're going to fetch the script from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/fault-injection/failcmd.sh right? if the URL is not going to change, I guess there's no need to make it a parameter.

Side note - the script could also be installed in the rootfs as an overlay. However, since it's specific for this type of tests and there's no other dependency required (except for wget, which is already installed on most rootfs images already available in KernelCI) I think getting the script with wget is fine in this case. @pawiecz @nuclearcat any thought on this?


created_at: 2025-05-01T08:10:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,32 @@
+- test:
+    timeout:
+      minutes: {{ job_timeout|default(20) }}
+    definitions:
+    - from: inline
+      name: "fault-injection-{{ test_name }}"
+      path: inline/fault-injection.yaml
+      repository:
+        metadata:
+          format: Lava-Test Test Definition 1.0
+          name: fault-injection
+          description: "Fault injection test: {{ test_name }}"
+          os:
+          - debian
+          scope:
+          - functional
+        run:
+          steps:
+            - wget {{ failcmd_url }} -O /tmp/failcmd.sh
+            - chmod +x /tmp/failcmd.sh
path: config/runtime/tests/fault-injection.jinja2
user: login: laura-nao
body: Maybe we can wrap the wget and chmod commands in a LAVA test set, so it's easier to spot a failure if it happens during the "setup" phase. So something like:
```
- lava-test-set start setup
- <wget command>
- <chmod command>
- lava-test-set stop setup
```
created_at: 2025-05-01T08:12:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,32 @@
+- test:
+    timeout:
+      minutes: {{ job_timeout|default(20) }}
+    definitions:
+    - from: inline
+      name: "fault-injection-{{ test_name }}"
+      path: inline/fault-injection.yaml
+      repository:
+        metadata:
+          format: Lava-Test Test Definition 1.0
+          name: fault-injection
+          description: "Fault injection test: {{ test_name }}"
+          os:
+          - debian
+          scope:
+          - functional
+        run:
+          steps:
+            - wget {{ failcmd_url }} -O /tmp/failcmd.sh
path: config/runtime/tests/fault-injection.jinja2
user: login: nuclearcat
body: kernel.org really hate this days if you fetch from them scripts in CI many times, most likely they will block IP. It is recommended to fetch and install in rootfs using overlay as you said.
created_at: 2025-05-01T08:44:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,32 @@
+- test:
+    timeout:
+      minutes: {{ job_timeout|default(20) }}
+    definitions:
+    - from: inline
+      name: "fault-injection-{{ test_name }}"
+      path: inline/fault-injection.yaml
+      repository:
+        metadata:
+          format: Lava-Test Test Definition 1.0
+          name: fault-injection
+          description: "Fault injection test: {{ test_name }}"
+          os:
+          - debian
+          scope:
+          - functional
+        run:
+          steps:
+            - wget {{ failcmd_url }} -O /tmp/failcmd.sh
path: config/runtime/tests/fault-injection.jinja2
user: login: laura-nao
body: Ack - then @crazoes we're going to need a new entry in the `rootfs-configs.yaml` file for a new rootfs for fault injection tests, that includes the failcmd script. I suggest copying the existing config for `bookworm` and adjust as needed. Either we create a script that gets run during the rootfs build and fetches the failcmd script (`script:` property), or we store the script as an overlay (`test_overlay:` property). I'd probably go with the former, as it would allow us to pick up any upstream script update simply by triggering a new rootfs build (fetching the script with wget in this case would be reasonable, as we don't trigger rootfs builds that often).

As we're adding a new rootfs config, it might be worth moving some of the test logic to a dedicated script too (via the `test_overlay:` property). This will simplify the jinja2 template and make it easier to add a result parser later if we need it.
created_at: 2025-05-01T09:34:52Z
author_association: CONTRIBUTOR

PR comments:
user: login: laura-nao
created_at: 2025-05-12T07:44:38Z
body: LGTM; the rootfs changes were also verified on staging: https://storage.staging.kernelci.org/rootfs/bookworm-fault-injection/20250507.0/amd64/
author_association: CONTRIBUTOR
user: login: nuclearcat
created_at: 2025-05-12T10:07:55Z
body: Excellent, thank you very much for verifying, merging
author_association: MEMBER