Change:
pull-kubernetes-conformance-kind-ga-only-parallel [sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance] After is merged. It relies on events which are not reliable as discussed . No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is not failing that much looking at the testgrid. Is it actually an issue?
Not so often, but it does happen. So, I think that it is worth fixing.
still flaking -
flaking conformance tests are concerning and should take priority over new conformance test additions
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I still see this test failing , also in periodic jobs
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale seems like it's is still failing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
still flake /remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/assign Namanl2001
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
please unassign if you are not working on it. Seems to be still flaking
It still flakes but not that much. I checked the history: adjusted the timeout from 30s to 1m. changed it not to fail for watching Delete Event, and just print it like below: This time, we see the above adding warning and then failed. Looking into kubelet/container runtime log. The pod is just deleted after 1 min, maybe 1min10s. I'm afraid that we have to adjust the timeout of the delete pod again to fix it. Is a valid pod deletion timeout? I think most deletion timeout of pod is 5min, however this is too long after pod with .
but why it takes so long kubelet to delete a pod? what are all of those ""Failed to delete cgroup paths" messages? can this be hiding some problem with runc or the kubelet?
Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: E0919 13:14:56. 359 ] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod53d19839-496a-4be8-8964-] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" pod="pods-900/pod-test" podUID=-496a-4be8-8964- Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56. 359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:14:56 kinder-external-etcd-worker-2 containerd[220]: time="2023-09-19T13:14:56.925525892Z" level=info msg="StartContainer for "e3b435116d2640d7c8166332e1c4bf6722f97ac1401035cbe443704c8b442efd" returns successfully" Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56. 359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:15:01 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:15:01. 359 ] "Cleaned up orphaned pod volumes dir" podUID=-496a-4be8-8964- path="/var/lib/kubelet/pods/-496a-4be8-8964-/volumes" It means timeout. Another failure is not for the same reason.
/joke
I wish I could clean mirrors for a living. It's just something I can see myself doing. <detailsIn response to : /joke Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is starting to be worrisome
another failure
was another occurrence of this I think. I'm seeing the timeout error for deleting the pod in the build logs: And then the error: It has the same sanbok not found error in the kind-worker logs: Along with the same cgroup error in the logs: So it's probably an instance of the first case mentioned caused by timeout.
Flake in . Delete timeout is 1 minute here. It still need more investigation why this is not deleted in 1 min.(heavy load or logic problems.)
Flake twice in yersterday.

Discussion:
"golang.org/x/net/websocket" v1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

return false, nil }) if err != nil { <del> p, _ := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.Logf("Pod: %+v", p) </del> <ins> framework.Logf("failed to see event that pod is created: %v", err) </ins> } <del> framework.ExpectNoError(err, "failed to see Pod %v in namespace %v running", testPod.ObjectMeta.Name, testNamespaceName) </del> <ins> p, err := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get Pod %v in namespace %v", testPodName, testNamespaceName) framework.ExpectEqual(p.Status.Phase, v1.PodRunning, "failed to see Pod %v in namespace %v running", p.ObjectMeta.Name, testNamespaceName) </ins> ginkgo.By("patching the Pod with a new Label and updated data") podPatch, err := json.Marshal(v1.Pod{

} return false, nil }) <del> framework.ExpectNoError(err, "failed to see %v event", watch.Modified) </del> <ins> if err != nil { framework.Logf("failed to see %v event: %v", watch.Modified, err) } </ins> ginkgo.By("getting the Pod and ensuring that it's patched") pod, err := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{})

} return false, nil }) <del> framework.ExpectNoError(err, "failed to see %v event", watch.Deleted) </del> <ins> if err != nil { framework.Logf("failed to see %v event: %v", watch.Deleted, err) } _, err = f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.ExpectError(err, "pod %v found in namespace %v, but it should be deleted", testPodName, testNamespaceName) framework.ExpectEqual(apierrors.IsNotFound(err), true, fmt.Sprintf("expected IsNotFound error, got %#v", err)) </ins> }) })