jiebi/CodeConvo huggingface dataset:
Code change:
return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

Related discussion:
text: The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.
Code change:
return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

Related discussion:
text: I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.
Code change:
kubelet.masterServiceNamespace = metav1.NamespaceDefault kubelet.serviceLister = testServiceLister{} kubelet.serviceHasSynced = func() bool { return true } <ins> kubelet.nodeHasSynced = func() bool { return true } </ins> kubelet.nodeLister = testNodeLister{ nodes: []*v1.Node{ {

Related discussion:
text: What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.
Code change:
kubelet.masterServiceNamespace = metav1.NamespaceDefault kubelet.serviceLister = testServiceLister{} kubelet.serviceHasSynced = func() bool { return true } <ins> kubelet.nodeHasSynced = func() bool { return true } </ins> kubelet.nodeLister = testNodeLister{ nodes: []*v1.Node{ {

Related discussion:
text: Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
return fmt.Errorf("PersistentVolumeClaims %v not all in phase %s within %v", pvcNames, phase, timeout) } <ins> // findAvailableNamespaceName random namespace name starting with baseName. func findAvailableNamespaceName(baseName string, c clientset.Interface) (string, error) { var name string err := wait.PollImmediate(Poll, 30*time.Second, func() (bool, error) { name = fmt.Sprintf("%v-%v", baseName, randomSuffix()) _, err := c.CoreV1().Namespaces().Get(name, metav1.GetOptions{}) if err == nil { // Already taken return false, nil } if apierrs.IsNotFound(err) { return true, nil } Logf("Unexpected error while getting namespace: %v", err) return false, nil }) return name, err } </ins> // CreateTestingNS should be used by every test, note that we append a common prefix to the provided test name. // Please see NewFramework instead of using this directly. func CreateTestingNS(baseName string, c clientset.Interface, labels map[string]string) (*v1.Namespace, error) {

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Code change:
endpointsAvailableForLB := !allEndpointsTerminating && !allEndpointsNonServing proxier.deleteExistingLoadBalancer(hns, svcInfo.winProxyOptimization, &svcInfo.hnsID, sourceVip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), hnsEndpoints, queriedLoadBalancers) <del> if endpointsAvailableForLB { </del> <ins> // clusterIPEndpoints is the endpoint list used for creating ClusterIP loadbalancer. clusterIPEndpoints := hnsEndpoints if svcInfo.internalTrafficLocal { // Take local endpoints for clusterip loadbalancer when internal traffic policy is local. clusterIPEndpoints = hnsLocalEndpoints } </ins> <del> // clusterIPEndpoints is the endpoint list used for creating ClusterIP loadbalancer. clusterIPEndpoints := hnsEndpoints if svcInfo.internalTrafficLocal { // Take local endpoints for clusterip loadbalancer when internal traffic policy is local. clusterIPEndpoints = hnsLocalEndpoints } </del> <ins> if len(clusterIPEndpoints) > 0 { </ins> // If all endpoints are terminating, then no need to create Cluster IP LoadBalancer // Cluster IP LoadBalancer creation

Related discussion:
text: I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured
Code change:
} mountComplete := false <del> err = wait.Poll(5*time.Second, 10*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(1*time.Second, 2*time.Minute, func() (bool, error) { </ins> err := b.mounter.MountSensitive(source, dir, "cifs", mountOptions, sensitiveMountOptions) mountComplete = true return true, err

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When starting pods with AFS volumes, there is an initial delay of five seconds until the pod is transitioning from the "Scheduled" state. The reason for this is that currently the volume mounting happens inside a wait.Poll which will initially wait a specified interval (currently 5 seconds) before execution. Event log for comparison with and without AFS volume attached (seconds since event): What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /priority important-soon /sig cloud-provider /area provider/azure
see
Code change:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Related discussion:
text: The test flakes very occasionally. Below are the findings of the circumstances of the flake. During two pods are created in quick succession. The test flakes when the second Pod fails to come up. The second pod fails to come up with error: Upon exploring the logs we discovered that there was only 1 Attach call made at the beginning, and one Detach call upon timeout of the test [See timestamps]: We tracked down the error to the likely culprit in func : based on the error message returned. This is likely caused by (string) being empty [uncomfirmed], which is what the Kubelet would get from the DevicePath field on the Node object. This DevicePath field is normally on Attach and removed on Detach of the volume. The DevicePath is clearly present on the Node when the first pod is being brought up. But it is possible that it is somehow missing or not able to be retrieved by the time the second Pod is starting up (and continues to be unavailable exhausting the 15m timeout on the test) even though no Detach operation was performed on the volume in the meantime (according to logs). We are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test. This flake was originally seen in and the test logs can be found here: https://k8s- Environment: GCE Kubernetes version (use ):Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ): /kind flake /kind bug /sig storage /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale A check in triage shows that this failing test is still a problem, and in fact has become more of a problem since October 4th. Has there been any progress on a fix?
Code change:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Related discussion:
text: /priority important-soon
/assign for triage
We should find the root cause and fix for 1.13
/assign Patrick can you PTAL at what is causing flakiness in this test.
I will have a look. Right now the log output of the sidecar containers and the CSI driver are not captured, which makes debugging a bit hard. In PR I am adding code which includes those message in the test log. Let me finish that PR and once it is merged, debugging the next test flake might provide some better insight into what went wrong.
Hey, this issue is considered a blocker for 1.13beta. Is there anything we can do to get it resolved sooner, so that it doesn't delay beta next week?
is trying to instrument the e2e tests to get logs from the csi driver containers so that we can debug. He has tried running the test locally ~100 times and didn't see it flake.
here's one recent one you can check out:
Michelle Au <: I had a look, but something is odd about that one. The error for the second pod is there: But under there is no further information about node bootstrap-e2e-gce-mz-minion-group-3kn6. Shouldn't that be available?
"Patrick Ohly" Here's a better example where the kublet log with the error is available: That one also has the log of the external-attacher, but that doesn't tell me much. I'm now comparing a "bad" run with a "good" local run. One observation is that in the "good" run, operationExecutor.UnmountDevice is completed before the second usage, including the "Volume detached for volume" step in Then operationExecutor.VerifyControllerAttachedVolume is done again for the second pod, exactly as it was done for the first: In the "bad" run, operationExecutor.MountVolume is invoked before that "Volume detached for volume" and then runs with an empty device path, without going through VerifyControllerAttachedVolume again: So to me it looks like a race between two conflicting operations. Does that make sense? How to fix it is less obvious. The relevant code seems to be this in Any suggestions?
This seems inline with the observations that made initially. We should trace where DevicePath is getting set.
Code change:
klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Related discussion:
text: Could there be a race with AD controller or external attacher if they try to detach the volume? is another race around this area. Unclear if it's the same yet.
Michelle Au <: No idea. Even if it isn't, it's at least related to the same piece of code. Probably worth fixing both together. Can we get someone who has worked in this area to fix these issues? Today is the first day that I've ever seen this code, so I'm a bit afraid that it'll take me a long time to get it right, and I'm worried about regressions.
I've asked to help investigate
It is not a blocker for beta or freeze. Can be addressed in stabilization. Let's keep the 1.13 tag on it. And based on progress by stabilization we can reassess if it is even a 1.13 blocker (at this point I don't believe it is).
The in the actual state of the world gets reset during unmounting:
That removes the device path looks wrong to me. I've submitted PR which changes that. Let's see how much that breaks...
/assign The conclusion in the CSI working group was that while my analysis and the resulting fix might be correct, further debugging and a second pair of eyes would be good. Let me reassign, because I have done what I could for this issue.
The RT considers this a blocker now, because flakes are happening often enough to interfere with getting clear signal on several jobs. Even if the underlying problem isn't that serious.
/assign I will help triage as well.
Ack Thanks
Instead of which may have negative side effects for non-CSI volumes, let's do this instead: Modify the CSI method in to ignore the passed in (which is populated from devicePath. Instead let's have that method generate the from scratch, e.g. attachID := getAttachmentName(volID, driverName, string(nodeName)) This will remove any CSI dependency on devicePath and not effect other volume plugins.
I am trying to twist my head around the code. IMO it affects also AWS that uses devicePath stored in and it should be reproducible there too. So far I was unsuccessful. In the logs it's clearly visible that UnmountDevice succeeded. So why kubelet continues with MountVolume instead of MountDevice? That's wrong.