jiebi/CodeConvo huggingface dataset:
Code change:
isMigratedOnNode := mpaSet.Has(pluginName) <del> if isMigratedOnNode { installed := false driverName, err := csiMigratedPluginManager.GetCSINameFromInTreeName(pluginName) if err != nil { return isMigratedOnNode, err } for _, driver := range csiNode.Spec.Drivers { if driver.Name == driverName { installed = true break } } if !installed { return true, fmt.Errorf("in-tree plugin %s is migrated on node %s but driver %s is not installed", pluginName, string(nodeName), driverName) } } </del> return isMigratedOnNode, nil }

Related discussion:
text: After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.
Code change:
isMigratedOnNode := mpaSet.Has(pluginName) <del> if isMigratedOnNode { installed := false driverName, err := csiMigratedPluginManager.GetCSINameFromInTreeName(pluginName) if err != nil { return isMigratedOnNode, err } for _, driver := range csiNode.Spec.Drivers { if driver.Name == driverName { installed = true break } } if !installed { return true, fmt.Errorf("in-tree plugin %s is migrated on node %s but driver %s is not installed", pluginName, string(nodeName), driverName) } } </del> return isMigratedOnNode, nil }

Related discussion:
text: But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran
Code change:
if err != nil { return err } <ins> if existingRole.AggregationRule != nil { // the old role already moved to an aggregated role, so there are no custom rules to migrate at this point return nil } </ins> glog.V(1).Infof("migrating %v to %v", existingRole.Name, newName) existingRole.Name = newName existingRole.ResourceVersion = "" // clear this so the object can be created.

Related discussion:
text: /kind bug What happened: I deployed a fresh custom HA kubernetes v1.10.2 cluster with 3 nodes (all controller and worker currently, more workers planned). I then created a namespace called "stummi" and a rolebinding with "user=stummi clusterrole=admin" in this namespace. User stummi (successfully authenticated by TLS client certificate) was not able to list pods or anything else in his namespace: While trying to figure things out, I downgraded to v1.8.11 (a version without cluster role aggregation, that worked) and also tried v1.11 alpha2 (which did not work). I never purged my etcd and I sadly do not have logs from before the different versions. Shortly after posting on slack and before writing this ticket, I dumped the cluster roles in question: Note the rules. What you expected to happen: I expected user stummi to be able to do most things in his namespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I posted this problem on slack and helped me and wanted this issue opened. /sig auth The cluster is deployed with ansible and I can give the playbooks when required. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: 3 virtual machines (2 cores, 8GB memory) OS (e.g. from /etc/os-release): Debian Stretch Kernel (e.g. ): Install tools: custom ansible playbooks Others: Please let me know if you need more information. I'm happy to provide everything you need to fix this :)
/assign
Code change:
"net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

Related discussion:
text: In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?
Code change:
"net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

Related discussion:
text: : I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!
Code change:
"net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

Related discussion:
text: </details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)
Code change:
return false, nil, nil }, }, <ins> { name:         "API watch error happen", volID:        "vol-005", attachID:     getAttachmentName("vol-005", testDriver, nodeName), shouldFail:   true, watcherError: true, reactor: func(action core.Action) (handled bool, ret runtime.Object, err error) { if action.Matches("get", "volumeattachments") { return true, makeTestAttachment(getAttachmentName("vol-005", testDriver, nodeName), nodeName, "vol-005"), nil } return false, nil, nil }, }, </ins> } for _, tc := range testCases {

Related discussion:
text: A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.
Code change:
return false, nil, nil }, }, <ins> { name:         "API watch error happen", volID:        "vol-005", attachID:     getAttachmentName("vol-005", testDriver, nodeName), shouldFail:   true, watcherError: true, reactor: func(action core.Action) (handled bool, ret runtime.Object, err error) { if action.Matches("get", "volumeattachments") { return true, makeTestAttachment(getAttachmentName("vol-005", testDriver, nodeName), nodeName, "vol-005"), nil } return false, nil, nil }, }, </ins> } for _, tc := range testCases {

Related discussion:
text: if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
"testing" "github.com/imdario/mergo" <ins> "k8s.io/kubernetes/pkg/client/restclient" </ins> clientcmdapi "k8s.io/kubernetes/pkg/client/unversioned/clientcmd/api" )

Related discussion:
text: Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in
Code change:
targetNumNodes := int32(framework.TestContext.CloudConfig.NumNodes - 1) ginkgo.By(fmt.Sprintf("decreasing cluster size to %d", targetNumNodes)) err = framework.ResizeGroup(group, targetNumNodes) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> err = framework.WaitForGroupSize(group, targetNumNodes) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> err = e2enode.WaitForReadyNodes(c, framework.TestContext.CloudConfig.NumNodes-1, 10*time.Minute) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> targetNodes, err := e2enode.GetReadySchedulableNodes(c) framework.ExpectNoError(err) framework.ExpectEqual(len(targetNodes.Items), int(targetNumNodes))

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.
Code change:
"fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

Related discussion:
text: All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.
Code change:
"fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

Related discussion:
text: The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.