Code change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [] [sig-node] Pods Extended [] Delete Grace Period should be submitted and removed [Conformance] Since when has it been failing: The ipv6 jobs have flakiness, but since I had 3 flakies in a row with this test Testgrid link: (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5&sort-by-flakiness= Reason for failure: It fails because is not able to delete de pod in 30 seconds, seems that this is the cause of that Nov 30 07:41:07 kind-worker kubelet[677]: W1130 07:41:07. 677 ] Failed to process watch event {EventType:0 Name:/docker/f35390ea97c6bae92f61d7d297fe7eacb4cfb3f2effafb23f3aca00e1a9d2341/kubepods/besteffort/podd8d966f8-2a03-4529-9d70-/243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d WatchSource:0}: task 243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d not found: not found Anything else we need to know: The job view seems to report the same test failures twice but the log output shows different timestamps
/sig node
/cc
interestingly, the serial ipv6 job doesn't look flaky: (IPv6),%20master%20(dev)&width=5&include-filter-by-regex=Delete%20Grace%20Period
this one is failing quite a lot in the 1.17 windows job - could someone confirm if it's a bug or a test issue? Thanks
or is it actually a completely different failure on windows? Since it's being tracked here as well
will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close
Closing this issue. <detailsIn response to : will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
by the way, this one is unrelated to the windows one :-)
ACK, thank you :)
Two failures seen on after the related PR merged, that was only a cleanup, not a root cause. /reopen
Reopened this issue. <detailsIn response to : two failures seen on after the related PR merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-testing /kind flake
Those labels are not set on the issue: <detailsIn response to : /remove-kind failing-testing /kind flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-test
I think that I'm making progress trying to deflake the ipv6 tests I have submitted PR to fix the portmap plugin issue that should be able to improve the situation
Retitling because I don't think this specific test failure is IPv6 specific anymore
triage: flaking in pull-kubernetes-e2e-gce: /priority important-soon This is the fourth flakiest test for pull-kubernetes-e2e-gce
debugging in
bumped priority, a conformance test must not be flaky. this is now the top flake in the e2e suite.
cc
debugging log messages merged 2019-12-19T08:38:10Z, will check for output in next failure
debug output from failure:
debug output from success:
failure in
based on my reading of this, the timeline was: 00:02:23. - we got a pod running 00:02:23. - deleted it with a 30 second grace period 00:02:54. - the kubelet finished handling the delete request (30.18 seconds) 00:02:54. - test failed with a timeout (30.25 seconds) That seems like extremely tight tolerances, especially since the grace period refers to termination of the pod process, not removal from the API. I would be inclined to allow up to 2x the grace period to observe complete removal from the API.
opened to allow time for the kubelet to remove the terminated pod from the API
top flake according to
Opened to tag as [Flaky] and kick out of [Conformance]
Test is no longer run in blocking presubmit. - who owns resolving this and reintroducing the test before 1.18 is released? Investigation with combined timeline at shows that the kubelet stopped the container and destroyed it within ~15 seconds (observed deletion at 06:20:11., destroyed container at 06:20:46.), but then took another 30 before actually deleting the pod from the API (06:21:12.).
To evaluate the effectiveness of a potential fix, I would recommend opening a PR that removes the tag along with the proposed fix, and repeatedly on that PR to accumulate 10-20 test runs.
/assign assigning sig-node leads for visibility/routing
/milestone v1.19
/milestone v1.18 we need an ack from sig-leads whether this is release blocking or not
This issue is opened for 2 releases in a row. I guess the flake is not a release-blocking. In addition, there are many PRs linked to this issue, and many attempts happened to make the situation better. However, we should keep this issue open and make sure that the flake will disappear. WDYT? cc
The test has been flaky for multiple releases, but it was marked flaky and completely removed mid-1.18, which is new. I'm wanting to make sure that absence of any test signal around this is known/acceptable to sig-node.
Links to the investigations I was able to do: combined e2e/api/kubelet log from failure with extended timeout:
Added to agenda for 3/24 sig-node call

Related discussion:
text: ginkgo.By("verifying the kubelet observed the termination notice") <ins> // allow up to 3x grace period (which allows process termination) // for the kubelet to remove from api.  need to follow-up on if this // latency between termination and reportal can be isolated further. </ins> start := time.Now() <del> err = wait.Poll(time.Second*5, time.Second*30, func() (bool, error) { </del> <ins> err = wait.Poll(time.Second*5, time.Second*30*3, func() (bool, error) { </ins> podList, err := e2ekubelet.GetKubeletPods(f.ClientSet, pod.Spec.NodeName) if err != nil { framework.Logf("Unable to retrieve kubelet pods for node %v: %v", pod.Spec.NodeName, err)