Repository: openstack-k8s-operators/install_yamls
Pull request #1039: [OSPRH-15341] Use a job instead of oc debug for 'make crc_storage'
Author: login: abays
Labels: name: lgtm
name: approved
Created: 2025-03-28T14:24:34Z | Merged: 2025-03-28T22:54:15Z

Recent analysis of Prow jobs has revealed that most of our failures are due to this error (or the equivalent in `crc_storage_cleanup`):

```
Removing debug pod ...
error: unable to upgrade connection: container container-00 not found in pod oko-12-bdbms-master-0-debug-7b4gk_openstack-kuttl-tests
make[3]: *** [Makefile:613: crc_storage] Error 1 
```

This PR attempts to use a Kubernetes `Job` to create and clean-up the CRC storage instead, which will hopefully be more reliable.


Diff:
diff --git a/scripts/create-pv.sh b/scripts/create-pv.sh
index 7338fe12..e58340a0 100644
--- a/scripts/create-pv.sh
+++ b/scripts/create-pv.sh
@@ -14,7 +14,12 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 set -ex
+
+SCRIPTPATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+. "${SCRIPTPATH}/storage_common.sh"
+
 PV_NUM=${PV_NUM:-12}
+TIMEOUT=${TIMEOUT:-300s}
 
 released=$(oc get pv -o json | jq -r '.items[] | select(.status.phase | test("Released")).metadata.name')
 
@@ -22,11 +27,13 @@ for name in $released; do
     oc patch pv -p '{"spec":{"claimRef": null}}' $name
 done
 
-NODE_NAMES=$(oc get node -o name -l node-role.kubernetes.io/worker)
+NODE_NAMES=$(oc get node -o template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -l node-role.kubernetes.io/worker)
 if [ -z "$NODE_NAMES" ]; then
     echo "Unable to determine node name with 'oc' command."
     exit 1
 fi
 for node in $NODE_NAMES; do
-    oc debug $node -T -- chroot /host /usr/bin/bash -c "for i in `seq -w -s ' ' $PV_NUM`; do echo \"creating dir /mnt/openstack/pv\$i on $node\"; mkdir -p /mnt/openstack/pv\$i; done"
+    . "${SCRIPTPATH}/storage_apply.sh" "${node}" "create"
 done
+
+oc wait job -n "${NAMESPACE}" -l install-yamls.crc.storage --for condition=Complete --timeout "${TIMEOUT}"
diff --git a/scripts/delete-pv.sh b/scripts/delete-pv.sh
index 15c83b2b..ce6c4dc4 100644
--- a/scripts/delete-pv.sh
+++ b/scripts/delete-pv.sh
@@ -14,13 +14,20 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 set -ex
+
+SCRIPTPATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+. "${SCRIPTPATH}/storage_common.sh"
+
 PV_NUM=${PV_NUM:-12}
+TIMEOUT=${TIMEOUT:-300s}
 
-NODE_NAMES=$(oc get node -o name -l node-role.kubernetes.io/worker)
+NODE_NAMES=$(oc get node -o template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -l node-role.kubernetes.io/worker)
 if [ -z "$NODE_NAMES" ]; then
     echo "Unable to determine node name with 'oc' command."
     exit 1
 fi
 for node in $NODE_NAMES; do
-    oc debug $node -T -- chroot /host /usr/bin/bash -c "for i in `seq -w -s ' ' $PV_NUM`; do echo \"deleting dir /mnt/openstack/pv\$i on $node\"; rm -rf /mnt/openstack/pv\$i; done"
+    . "${SCRIPTPATH}/storage_apply.sh" "${node}" "delete"
 done
+
+oc wait job -n "${NAMESPACE}" -l install-yamls.crc.storage --for condition=Complete --timeout "${TIMEOUT}"
diff --git a/scripts/storage_apply.sh b/scripts/storage_apply.sh
new file mode 100755
index 00000000..24647447
--- /dev/null
+++ b/scripts/storage_apply.sh
@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2025 Red Hat Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+set -ex
+
+NODE=${1:-"crc"}
+OPERATION=${2:-"create"}
+
+oc delete -n "${NAMESPACE}" job "crc-storage-${NODE}" --ignore-not-found
+
+cat << EOF | oc apply -f -
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: crc-storage-${NODE}
+  namespace: ${NAMESPACE}
+  labels:
+    install-yamls.crc.storage: ""
+spec:
+  template:
+    spec:
+      containers:
+      - name: storage
+        image: bash:latest
+        env:
+          - name: PV_NUM
+            value: "${PV_NUM}"
+        command: ["bash"]
+        args: ["/usr/local/bin/crc-storage.sh"]
+        securityContext:
+          privileged: true
+          allowPrivilegeEscalation: true
+          runAsUser: 0
+          seccompProfile:
+            type: RuntimeDefault
+        volumeMounts:
+          - mountPath: /usr/local/bin/crc-storage.sh
+            name: crc-storage
+            readOnly: true
+            subPath: ${OPERATION}-storage.sh
+          - name: node-mnt
+            mountPath: /mnt/nodeMnt
+      nodeSelector:
+        kubernetes.io/hostname: ${NODE}
+        node-role.kubernetes.io/worker: ""
+      restartPolicy: Never
+      securityContext:
+        runAsUser: 0
+      serviceAccount: crc-storage
+      volumes:
+        - configMap:
+            defaultMode: 493
+            items:
+              - key: ${OPERATION}-storage.sh
+                path: ${OPERATION}-storage.sh
+            name: crc-storage
+          name: crc-storage
+        - name: node-mnt
+          hostPath:
+            path: /mnt
+            type: Directory
+  backoffLimit: 10
+EOF
diff --git a/scripts/storage_common.sh b/scripts/storage_common.sh
new file mode 100755
index 00000000..6170013b
--- /dev/null
+++ b/scripts/storage_common.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Copyright 2025 Red Hat Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+set -ex
+OPERATION=${1:-"create"}
+
+cat << EOF | oc apply -f -
+apiVersion: v1
+data:
+  create-storage.sh: |
+    #!/bin/bash
+
+    for i in \`seq -w -s ' ' \${PV_NUM}\`; do
+      echo "creating dir /mnt/openstack/pv\$i on host"
+      mkdir -p /mnt/nodeMnt/openstack/pv\$i
+    done
+  delete-storage.sh: |
+    #!/bin/bash
+
+    for i in \`seq -w -s ' ' \${PV_NUM}\`; do
+      echo "deleting dir /mnt/openstack/pv\$i on host"
+      rm -rf /mnt/nodeMnt/openstack/pv\$i
+    done
+kind: ConfigMap
+metadata:
+  name: crc-storage
+  namespace: ${NAMESPACE}
+EOF
+
+cat << EOF | oc apply -f -
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: crc-storage
+  namespace: ${NAMESPACE}
+EOF
+
+cat << EOF | oc apply -f -
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: crc-storage-role
+  namespace: ${NAMESPACE}
+rules:
+- apiGroups:
+  - security.openshift.io
+  resourceNames:
+  - anyuid
+  - privileged
+  resources:
+  - securitycontextconstraints
+  verbs:
+  - use
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  - jobs
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+  - update
+  - patch
+  - delete
+EOF
+
+cat << EOF | oc apply -f -
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: crc-storage-rolebinding
+  namespace: ${NAMESPACE}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: crc-storage-role
+subjects:
+- kind: ServiceAccount
+  name: crc-storage
+  namespace: ${NAMESPACE}
+EOF


Reviews:
user: login: olliewalsh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T16:08:37Z
user: login: abays
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T16:09:56Z
user: login: abays
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T16:53:59Z
user: login: dprince
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T18:09:26Z
user: login: abays
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T18:26:31Z
user: login: dprince
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T20:33:54Z

Review comments:
diff_hunk: @@ -22,11 +26,13 @@ for name in $released; do
     oc patch pv -p '{"spec":{"claimRef": null}}' $name
 done
 
-NODE_NAMES=$(oc get node -o name -l node-role.kubernetes.io/worker)
+NODE_NAMES=$(oc get node -o template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -l node-role.kubernetes.io/worker)
 if [ -z "$NODE_NAMES" ]; then
     echo "Unable to determine node name with 'oc' command."
     exit 1
 fi
 for node in $NODE_NAMES; do
-    oc debug $node -T -- chroot /host /usr/bin/bash -c "for i in `seq -w -s ' ' $PV_NUM`; do echo \"creating dir /mnt/openstack/pv\$i on $node\"; mkdir -p /mnt/openstack/pv\$i; done"
+    . "${SCRIPTPATH}/storage_apply.sh" "${node}" "create"
 done
+
+oc wait job -n openstack -l install-yamls.crc.storage --for condition=Complete
path: scripts/create-pv.sh
user: login: olliewalsh
body: IIUC default timeout is 30s, maybe worth extending it?
created_at: 2025-03-28T16:07:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -22,11 +26,13 @@ for name in $released; do
     oc patch pv -p '{"spec":{"claimRef": null}}' $name
 done
 
-NODE_NAMES=$(oc get node -o name -l node-role.kubernetes.io/worker)
+NODE_NAMES=$(oc get node -o template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -l node-role.kubernetes.io/worker)
 if [ -z "$NODE_NAMES" ]; then
     echo "Unable to determine node name with 'oc' command."
     exit 1
 fi
 for node in $NODE_NAMES; do
-    oc debug $node -T -- chroot /host /usr/bin/bash -c "for i in `seq -w -s ' ' $PV_NUM`; do echo \"creating dir /mnt/openstack/pv\$i on $node\"; mkdir -p /mnt/openstack/pv\$i; done"
+    . "${SCRIPTPATH}/storage_apply.sh" "${node}" "create"
 done
+
+oc wait job -n openstack -l install-yamls.crc.storage --for condition=Complete
path: scripts/create-pv.sh
user: login: abays
body: Good point
created_at: 2025-03-28T16:09:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -22,11 +26,13 @@ for name in $released; do
     oc patch pv -p '{"spec":{"claimRef": null}}' $name
 done
 
-NODE_NAMES=$(oc get node -o name -l node-role.kubernetes.io/worker)
+NODE_NAMES=$(oc get node -o template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -l node-role.kubernetes.io/worker)
 if [ -z "$NODE_NAMES" ]; then
     echo "Unable to determine node name with 'oc' command."
     exit 1
 fi
 for node in $NODE_NAMES; do
-    oc debug $node -T -- chroot /host /usr/bin/bash -c "for i in `seq -w -s ' ' $PV_NUM`; do echo \"creating dir /mnt/openstack/pv\$i on $node\"; mkdir -p /mnt/openstack/pv\$i; done"
+    . "${SCRIPTPATH}/storage_apply.sh" "${node}" "create"
 done
+
+oc wait job -n openstack -l install-yamls.crc.storage --for condition=Complete
path: scripts/create-pv.sh
user: login: abays
body: Done
created_at: 2025-03-28T16:53:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2025 Red Hat Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+set -ex
+
+NODE=${1:-"crc"}
+OPERATION=${2:-"create"}
+
+oc delete -n openstack job "crc-storage-${NODE}" --ignore-not-found
+
+cat << EOF | oc apply -f -
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: crc-storage-${NODE}
+  namespace: openstack
path: scripts/storage_apply.sh
user: login: dprince
body: should we run this in the openstack-operators namespace? Or maybe even in its own "openstack-crc-storage" namespace to keep it isolated. I typically run crc_storage very early and won't always have this pre-created.
created_at: 2025-03-28T18:09:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2025 Red Hat Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+set -ex
+
+NODE=${1:-"crc"}
+OPERATION=${2:-"create"}
+
+oc delete -n openstack job "crc-storage-${NODE}" --ignore-not-found
+
+cat << EOF | oc apply -f -
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: crc-storage-${NODE}
+  namespace: openstack
path: scripts/storage_apply.sh
user: login: abays
body: We could give it its own namespace, yes.  But as it works right now the `namespace` target is a prereq for both `make crc_storage` and `make crc_storage_cleanup`, which creates the `openstack` namespace [1] [2].

[1] https://github.com/openstack-k8s-operators/install_yamls/blob/3ab4f516006065c6e7147374835b039210bf352e/Makefile#L602
[2] https://github.com/openstack-k8s-operators/install_yamls/blob/3ab4f516006065c6e7147374835b039210bf352e/Makefile#L609
created_at: 2025-03-28T18:26:30Z
author_association: CONTRIBUTOR

PR comments:
user: login: softwarefactory-project-zuul[bot]
created_at: 2025-03-28T19:31:57Z
body: Build failed (check pipeline).  Post `recheck` (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6a2caad04e484a609612e256c3d0f739

:x: [openstack-k8s-operators-content-provider ](https://softwarefactory-project.io/zuul/t/rdoproject.org/build/9b13f4de9fb249cf9923a9e3dc087752) **FAILURE** in 11m 49s
:warning: [install-yamls-crc-podified-edpm-baremetal ](https://softwarefactory-project.io/zuul/t/rdoproject.org/build/d76a7df12ec64899b50a7fb753f57360) SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
:warning: [podified-multinode-edpm-deployment-crc ](https://softwarefactory-project.io/zuul/t/rdoproject.org/build/8c21e2fe85ad4b07a0cbab38f0bc014f) SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

author_association: NONE
user: login: abays
created_at: 2025-03-28T20:32:25Z
body: recheck
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-28T20:34:01Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openstack-k8s-operators/install_yamls/pull/1039#" title="Author self-approved">abays</a>*, *<a href="https://github.com/openstack-k8s-operators/install_yamls/pull/1039#pullrequestreview-2726675744" title="Approved">dprince</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openstack-k8s-operators%2Finstall_yamls).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/openstack-k8s-operators/install_yamls/blob/main/OWNERS)~~ [abays,dprince]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR