Code change:
Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not? more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)

Related discussion:
text: } // DeletePodWithWaitByName deletes the named and namespaced pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWaitByName(c clientset.Interface, podName, podNamespace string) error { e2elog.Logf("Deleting pod %q in namespace %q", podName, podNamespace) <del> deletionPolicy := metav1.DeletePropagationForeground err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{ // If the pod is the owner of some resources (like ephemeral inline volumes), // then we want to be sure that those are also gone before we return. // Blocking pod deletion via metav1.DeletePropagationForeground achieves that. PropagationPolicy: &deletionPolicy, }) </del> <ins> err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{}) </ins> if err != nil { if apierrors.IsNotFound(err) { return nil // assume pod was already deleted