jiebi/CodeConvo huggingface dataset:
Code change:
DeleteFunc: c.deleteAPIService, }) <del> serviceInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc:    c.addService, UpdateFunc: c.updateService, DeleteFunc: c.deleteService, }) </del> c.syncFn = c.sync return c

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects
Code change:
DisableAcceleratorUsageMetrics:                 {Default: true, PreRelease: featuregate.Beta}, HPAContainerMetrics:                            {Default: false, PreRelease: featuregate.Alpha}, RootCAConfigMap:                                {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.22 <del> SizeMemoryBackedVolumes:                        {Default: false, PreRelease: featuregate.Alpha}, </del> <ins> SizeMemoryBackedVolumes:                        {Default: true, PreRelease: featuregate.Beta}, </ins> ExecProbeTimeout:                               {Default: true, PreRelease: featuregate.GA}, // lock to default and remove after v1.22 based on KEP #1972 update KubeletCredentialProviders:                     {Default: false, PreRelease: featuregate.Alpha}, GracefulNodeShutdown:                           {Default: true, PreRelease: featuregate.Beta},

Related discussion:
text: When master opens up for , we need to add this back in: /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
note to self / 1.21 and 1.22 release notes team -- let's make sure the PR body on the original PR is copied over to the new PR. will think that the old release note and documentation has been covered in 1.21, but we will (I will update this comment with link) manually override it to not show up in 1.21 release notes.
Code change:
return nil } <del> // handleDockercfgContent serializes a dockercfg json file func handleDockercfgContent(username, password, email, server string) ([]byte, error) { </del> <ins> // handleDockerCfgJsonContent serializes a ~/.docker/config.json file func handleDockerCfgJsonContent(username, password, email, server string) ([]byte, error) { </ins> dockercfgAuth := credentialprovider.DockerConfigEntry{ Username: username, Password: password, Email:    email, } <del> dockerCfg := credentialprovider.DockerConfigJson{ </del> <ins> dockerCfgJson := credentialprovider.DockerConfigJson{ </ins> Auths: map[string]credentialprovider.DockerConfigEntry{server: dockercfgAuth}, } <del> return json.Marshal(dockerCfg) </del> <ins> return json.Marshal(dockerCfgJson) </ins> }

Related discussion:
text: /kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.
Code change:
return nil } <del> // handleDockercfgContent serializes a dockercfg json file func handleDockercfgContent(username, password, email, server string) ([]byte, error) { </del> <ins> // handleDockerCfgJsonContent serializes a ~/.docker/config.json file func handleDockerCfgJsonContent(username, password, email, server string) ([]byte, error) { </ins> dockercfgAuth := credentialprovider.DockerConfigEntry{ Username: username, Password: password, Email:    email, } <del> dockerCfg := credentialprovider.DockerConfigJson{ </del> <ins> dockerCfgJson := credentialprovider.DockerConfigJson{ </ins> Auths: map[string]credentialprovider.DockerConfigEntry{server: dockercfgAuth}, } <del> return json.Marshal(dockerCfg) </del> <ins> return json.Marshal(dockerCfgJson) </ins> }

Related discussion:
text: However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret
Code change:
// } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

Related discussion:
text: On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.
Code change:
// } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

Related discussion:
text: It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.
Code change:
// } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

Related discussion:
text: - that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.
Code change:
# TODO(pjh): move the logging agent code below into a separate # module; it was put here temporarily to avoid disrupting the file layout in # the K8s release machinery. <del> $LOGGINGAGENT_VERSION = '1.7.3' </del> <ins> $LOGGINGAGENT_VERSION = '1.7.5' </ins> $LOGGINGAGENT_ROOT = 'C:fluent-bit' $LOGGINGAGENT_SERVICE = 'fluent-bit' $LOGGINGAGENT_CMDLINE = '*fluent-bit.exe*'

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted
Code change:
"//pkg/api:go_default_library", "//pkg/api/v1/ref:go_default_library", "//pkg/kubelet/apis/cri/v1alpha1/runtime:go_default_library", <del> "//pkg/kubelet/events:go_default_library", </del> "//pkg/kubelet/util/format:go_default_library", "//pkg/kubelet/util/ioutils:go_default_library", "//pkg/util/hash:go_default_library",

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub
Code change:
return err } // Unmount the device from the device mount point. <del> klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { </del> <ins> notMnt, err := detacher.mounter.IsLikelyNotMountPoint(deviceMountPath) if err != nil { </ins> return err } <del> klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </del> <ins> if !notMnt { klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err } klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) } </ins> <del> klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err </del> <ins> // Get devicePath from deviceMountPath if devicePath is empty if devicePath == "" { rbdImageInfo, err := getRbdImageInfo(deviceMountPath) if err != nil { return err } found := false devicePath, found = getRbdDevFromImageAndPool(rbdImageInfo.pool, rbdImageInfo.name) if !found { klog.Warningf("rbd: can't found devicePath for %v. Device is already unmounted, Image %v, Pool %v", deviceMountPath, rbdImageInfo.pool, rbdImageInfo.name) } } if devicePath != "" { klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err } klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </ins> } <del> klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </del> err = os.Remove(deviceMountPath) if err != nil { return err

Related discussion:
text: Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'm experiencing this issue. From what I can read above, if the rbd mount happens before the node_exporter starts; we'll always experience status 16?
We're getting this issue. This is because the mounted volume is mounted on Cadvisor container too. So the /dev/rbdX file is still in use. But still didn't find any solution to fix.
rbd volume will stuck when failed as will always returen
My solution: container use a volumeMount. make a script file to be executed every a minute:
Code change:
nsFlag := fmt.Sprintf("--namespace=%v", ns) ginkgo.By("executing a command with run and attach with stdin") <del> runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "cat && echo 'stdin closed'"). WithStdinData("abcd1234"). </del> <ins> // We wait for a non-empty line so we know kubectl has attached runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "while [ -z "$s" ]; do read s; sleep 1; done; echo read:$s && cat && echo 'stdin closed'"). WithStdinData("valuenabcd1234"). </ins> ExecOrDie() <del> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test", 1*time.Minute, g) gomega.Expect(err).To(gomega.BeNil()) // NOTE: we cannot guarantee our output showed up in the container logs before stdin was closed, so we have // to loop test. err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, 1*time.Second) { e2elog.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test") } logOutput := framework.RunKubectlOrDie(nsFlag, "logs", runTestPod.Name) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) gomega.Expect(err).To(gomega.BeNil()) </del> <ins> gomega.Expect(runOutput).To(gomega.ContainSubstring("read:value")) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) </ins> gomega.Expect(c.BatchV1().Jobs(ns).Delete("run-test", nil)).To(gomega.BeNil())

Related discussion:
text: <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this