Change:
majority of tests Since 01-25 No response No response No response /sig node
Last passing - First failing -
Majority of tests failing with
Passing metadata Failing metadata
/cc
/assign
More investigation -- containerd version was bumped from passing run to failing run (from ) passing version: failing version:
I'm trying to compare the env setup to which is same except not on COS m97 which is passing fine. The failures in this version seem to be networking/connectivity related, lots of errors of connection not made and agnhost issues...
/triage accepted
Did some more digging on this run - I picked a single failure to investigate deeper - So one of the storage tests is attempting to exec a command on the node but the container is exited. The failure is here - To run the host exec commands it appears the test creates a . runs . The pod the test created was called . The audit logs have the pod spec for this pod - The pod appears to be scheduled on node. Here are the logs related to the pod from kubelet searching by pod name and UID It appears the pod starts successfully but then abruptly it is killed The restarts a few times and this pattern repeats. One thing stands out to me right before the container is killed, there is the following log saying that cpuset controller is missing...
I was able to repro the test failures locally now, with running it as follows: After investigating the cgroup error above, the issue became more clear -- the test is no longer configuring containerd correctly. Specifically, containerd is no longer configured to use the systemd cgroup driver. As background, the test is expected to pick up the containerd configuration from the containerd script and the systemd cgroup setting by using this . However, reproing the test, the containerd config is not picking up this change. Looking at bit deeper, it looks like the issue is that the containerd configuration is being overriden. Specifically, first the containerd setup scripts runs and writes the containerd config file () and then the k8s scripts runs and overwrites the config as part of calling . As a result any special config in the first containerd setup script ends up by overriden and lost. This is due to the changes introduced in the following two PRs: The current behavior of k8s setup scripts is now we always call . Prior to , was only called if was set to : The related issue is that prior to , was set to . By setting to , the script did not call and the containerd config was preserved (since ). I think the larger issue is that prior to , there were two settings and for on . was used in jobs where containerd had to be installed manually and was used to indicate that the built in containerd version of the base OS will be used. Due to the combination of and , there is no longer distinction between and and the containerd config now is always being overridden by the k8s scripts.
Here is one way that it can be fixed: alternatively we can consider to revert the two PRs above but since they are quite large we might hit some conflicts, so it may be easier to rollforward with fix.
Last run is back to green, yay! <img width="1295" alt="image" src="https://user-"

Discussion:
# stop docker if it is present as we want to use just containerd log-wrap 'StopDocker' systemctl stop docker || echo "unable to stop docker" fi <del> log-wrap 'SetupContainerd' setup-containerd </del> <ins> if [[ ! -e "/etc/profile.d/containerd_env.sh" ]]; then log-wrap 'SetupContainerd' setup-containerd else echo "Skipping SetupContainerd step because containerd has already been setup by containerd's configure.sh script" fi </ins> log-start 'SetupKubePodLogReadersGroupDir' if [[ -n "${KUBE_POD_LOG_READERS_GROUP:-}" ]]; then