jiebi/CodeConvo huggingface dataset:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] is failing since June 29th 2021 (but the first failure was different than more recent ones). Commit , infra commit , . Other three tests are failing since July 9th 2021. Commit , infra commit , . Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: There is another open issues for failing tests in this job: Failures reported there are about missing metrics, and were occuring before unmounting problems. So I opened a new issue, even though it's the same SIG. These tests don't look related to me, but I don't have enough context, maybe they somehow are. /sig storage /kind failing-test /milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.
<detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.
was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".
After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.
Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).
last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-gce-cos-k8sbeta-default Networking Granular Checks: Services should function for service endpoints using hostNetwork Triage looks like it's mostly flaking on this job only: Examples /sig network <!-- Please see the deflaking doc () for more guidance! --
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Thread:
/assign
This is fixed by , let's wait some runs to confirm before closing
/milestone clear (since this has been deemed to be non release-blocking)
/triage accepted
/close
Closing this issue. <detailsIn response to : This is fixed by , let's wait some runs to confirm before closing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<img width="1312" alt="image" src=""
This looks like a test you a few days ago: /assign Can you look into these failures?
GitHub didn't allow me to assign the following users: chengjoey. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This looks like a test you a few days ago: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
As mentioned in the previous comments, there should be only one expected state in the end. From the logs of these failed tests, after stopping the container, the phase changes to Failed. I think this does not meet expectations.
Thanks to for revert my previous PR. I think the pod final state should always be in one state. From the current e2e test results, the final state may be Succeed or Failed, seems pod.Status.Phase is empty. I think this seems to be wrong. I will study whether the node status (sufficient memory) and other factors interfere with the final state. Thanks again to for his help.
Add the log with phase "Succeed": The normal update status steps are 1-2-3-4, but the result of failure tests is that step 2 is executed last, which seems a bit strange. Because phase is empty
feel free to reopen the PR. But let's make sure to run the tests that this reported as failing before we merge. Thank you for your help on improving test coverage!
The links to the failed E2E tests in the overview of this issue all show the Pod phase as , but is there a link available somewhere for cases where it's empty string?
In fact, these failed tests are all empty strings, because not terminated: should be followed by phase Because my original test was written like this as I commented earlier, different order of updating status may lead to different final state
The Pod's final state is shown as in the dumped result immediately after the test fails. The referenced inside - is that coming from the Pod information fetched within ? Could it be that it's showing up as an empty string because it's not able to access the value externally?
In the tests submitted previously, and in my local(minnikube) tests, the pod status will change from to again see this log step 3 So I think the order of state changes affects the final result, and this is not very stable
I think this is a separate issue, so as I mentioned in , I thought it might be fine to wait for either the or phase. However, after looking at , it seems that we may need to resolve this unstable final result first. WDYT?
If we can explain different results, it is fine. I don't want unexplained check for either condition in our tests
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [sig-auth] [Feature:NodeAuthorizer] A node shouldn't be able to create another node Since when has it been failing: N/A Testgrid link: N/A Reason for failure: The test creates a new ` node, but it does not delete it after the test ends: kubectl get nodes NAME STATUS ROLES AGE VERSION 1954k8s000 Ready agent 24d v1.14.1 1954k8s001 Ready agent 24d v1.14.1 foo NotReady <none4m3s k8s-master--0 Ready master 24d v1.14.1 Anything else we need to know: /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .
and show that we run this job every few minutes, but it never passes. I'm going to send out a PR to disable this job until someone has the time to investigate and resolve.
It's failing before tests even start:
At this point I don't think it's a flake, we needed flannel for a release and it's less important now so I'm not investigating why the cluster isn't coming up soon. I do believe if the cluster comes up a majority if not all of the tests will pass, and the ones that don't will flake for known reasons.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This is probably just a documentation / example issue. I'm a real noob so I can just tell this is an issue, but I can't fix the docs through a PR, otherwise I'd do it happily. Hope it's not a dupe. Following the guide at with IPVS. Using the following script: This generates the following YAML (only the interesting part): You see that the line has an extra comma. It's instead of (extra comma after second argument) This generates the following log: Removing the extra comma fixes everything. Setup a cluster with Kube-DNS IPVS mode Generate the nodelocaldns Deploy the manifest Pods are crashing Kubernetes version v1.21.0 Cloud provider or hardware configuration: Bare Metal with HA setup (behind an external LB, Kube-proxy as IPVS) OS (e.g: ): CentOS Linux 8 Kernel (e.g. ): Linux 4.18.0-240.10.1.el83.x8664 SMP Mon Jan 18 17:05:51 UTC 2021 x8664 x8664 x86_64 GNU/LinuxInstall tools: none, a classic kubeadm initNetwork plugin and version (if this is a network-related bug): Not relevant, but Canal (Calico + Flannel) Others: nothing. Love you all for making K8S. Thank you a billion times.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
, it seems like that don't get a value return. i saw command in Can you try this again and check if can kubedns get a value?
/area dns /sig network /assign Documentation here is not changed since Sep 2019 and needs to be updated as well.
More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR