Chapter 11. Troubleshooting When troubleshooting OpenShift sandboxed containers, you can open a support case and provide debugging information using the must-gather tool. If you are a cluster administrator, you can also review logs on your own, enabling a more detailed level of logs. 11.1. Collecting data for Red Hat Support When opening a support case, it is helpful to provide debugging information about your cluster to Red Hat Support. The must-gather tool enables you to collect diagnostic information about your OpenShift Container Platform cluster, including virtual machines and other data related to OpenShift sandboxed containers. For prompt support, supply diagnostic information for both OpenShift Container Platform and OpenShift sandboxed containers. Using the must-gather tool The oc adm must-gather CLI command collects the information from your cluster that is most likely needed for debugging issues, including: Resource definitions Service logs By default, the oc adm must-gather command uses the default plugin image and writes into ./must-gather.local . Alternatively, you can collect specific information by running the command with the appropriate arguments as described in the following sections: To collect data related to one or more specific features, use the --image argument with an image, as listed in a following section. For example: USD oc adm must-gather --image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel9:1.8.1 To collect the audit logs, use the -- /usr/bin/gather_audit_logs argument, as described in a following section. For example: USD oc adm must-gather -- /usr/bin/gather_audit_logs Note Audit logs are not collected as part of the default set of information to reduce the size of the files. When you run oc adm must-gather , a new pod with a random name is created in a new project on the cluster. The data is collected on that pod and saved in a new directory that starts with must-gather.local . This directory is created in the current working directory. For example: NAMESPACE NAME READY STATUS RESTARTS AGE ... openshift-must-gather-5drcj must-gather-bklx4 2/2 Running 0 72s openshift-must-gather-5drcj must-gather-s8sdh 2/2 Running 0 72s ... Optionally, you can run the oc adm must-gather command in a specific namespace by using the --run-namespace option. For example: USD oc adm must-gather --run-namespace <namespace> --image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel9:1.8.1 11.2. Collecting log data The following features and objects are associated with OpenShift sandboxed containers: All namespaces and their child objects that belong to OpenShift sandboxed containers resources All OpenShift sandboxed containers custom resource definitions (CRDs) You can collect the following component logs for each pod running with the kata runtime: Kata agent logs Kata runtime logs QEMU logs Audit logs CRI-O logs 11.2.1. Enabling debug logs for CRI-O runtime You can enable debug logs by updating the logLevel field in the KataConfig CR. This changes the log level in the CRI-O runtime for the worker nodes running OpenShift sandboxed containers. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure Change the logLevel field in your existing KataConfig CR to debug : USD oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}' Monitor the kata-oc machine config pool until the value of UPDATED is True , indicating that all worker nodes are updated: USD oc get mcp kata-oc Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE kata-oc rendered-kata-oc-169 False True False 3 1 1 0 9h Verification Start a debug session with a node in the machine config pool: USD oc debug node/<node_name> Change the root directory to /host : # chroot /host Verify the changes in the crio.conf file: # crio config | egrep 'log_level Example output log_level = "debug" 11.2.2. Viewing debug logs for components Cluster administrators can use the debug logs to troubleshoot issues. The logs for each node are printed to the node journal. You can review the logs for the following OpenShift sandboxed containers components: Kata agent Kata runtime ( containerd-shim-kata-v2 ) virtiofsd QEMU only generates warning and error logs. These warnings and errors print to the node journal in both the Kata runtime logs and the CRI-O logs with an extra qemuPid field. Example of QEMU logs Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.587116986Z" level=info msg="Start logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.607339014Z" level=error msg="qemu-kvm: -machine q35,accel=kvm,kernel_irqchip=split,foo: Expected '=' after parameter 'foo'" name=containerd-shim-v2 pid=2241647 qemuPid=2241693 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.60890737Z" level=info msg="Stop logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu The Kata runtime prints Start logging QEMU when QEMU starts, and Stop Logging QEMU when QEMU stops. The error appears in between these two log messages with the qemuPid field. The actual error message from QEMU appears in red. The console of the QEMU guest is printed to the node journal as well. You can view the guest console logs together with the Kata agent logs. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-admin role. Procedure To review the Kata agent logs and guest console logs, run the following command: USD oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g "reading guest console" To review the Kata runtime logs, run the following command: USD oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata To review the virtiofsd logs, run the following command: USD oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t virtiofsd To review the QEMU logs, run the following command: USD oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g "qemuPid=\d+" Additional resources Gathering data about your cluster in the OpenShift Container Platform documentation

Commands:
oc adm must-gather --image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel9:1.8.1
oc adm must-gather -- /usr/bin/gather_audit_logs
NAMESPACE NAME READY STATUS RESTARTS AGE openshift-must-gather-5drcj must-gather-bklx4 2/2 Running 0 72s openshift-must-gather-5drcj must-gather-s8sdh 2/2 Running 0 72s
oc adm must-gather --run-namespace <namespace> --image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel9:1.8.1
oc patch kataconfig <kataconfig> --type merge --patch '{"spec":{"logLevel":"debug"}}'
oc get mcp kata-oc
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE kata-oc rendered-kata-oc-169 False True False 3 1 1 0 9h
oc debug node/<node_name>
chroot /host
crio config | egrep 'log_level
log_level = "debug"
Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.587116986Z" level=info msg="Start logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.607339014Z" level=error msg="qemu-kvm: -machine q35,accel=kvm,kernel_irqchip=split,foo: Expected '=' after parameter 'foo'" name=containerd-shim-v2 pid=2241647 qemuPid=2241693 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu Mar 11 11:57:28 openshift-worker-0 kata[2241647]: time="2023-03-11T11:57:28.60890737Z" level=info msg="Stop logging QEMU (qemuPid=2241693)" name=containerd-shim-v2 pid=2241647 sandbox=d1d4d68efc35e5ccb4331af73da459c13f46269b512774aa6bde7da34db48987 source=virtcontainers/hypervisor subsystem=qemu
oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g "reading guest console"
oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata
oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t virtiofsd
oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g "qemuPid=\d+"