jiebi/CodeConvo huggingface dataset:
Change:
allErrs = append(allErrs, field.Required(idxPath, "must specify `addresses` or `notReadyAddresses`")) } for addr := range ss.Addresses { <del> allErrs = append(allErrs, validateEndpointAddress(&ss.Addresses[addr], idxPath.Child("addresses").Index(addr), ipToNodeName)...) </del> <ins> allErrs = append(allErrs, validateEndpointAddress(&ss.Addresses[addr], idxPath.Child("addresses").Index(addr))...) </ins> } for addr := range ss.NotReadyAddresses { <del> allErrs = append(allErrs, validateEndpointAddress(&ss.NotReadyAddresses[addr], idxPath.Child("notReadyAddresses").Index(addr), ipToNodeName)...) </del> <ins> allErrs = append(allErrs, validateEndpointAddress(&ss.NotReadyAddresses[addr], idxPath.Child("notReadyAddresses").Index(addr))...) </ins> } for port := range ss.Ports { allErrs = append(allErrs, validateEndpointPort(&ss.Ports[port], len(ss.Ports) > 1, idxPath.Child("ports").Index(port))...)

Discussion:
/kind bug What happened: We have noticed that one of the node-exporters is down. It turned out that service has "old" endpoint still. After digging deeper i see following errors in controller manager Turns out controller manager can not update endpoints for node-exporter service, because of this error. In endpoint list is see right IPs for worker nodes, but wrong IP for master node (that's why we detected the problem). Context: two worker nodes were recreated with new hostnames (generated randomly), but they use same ip address one master node were recreated with new hostname and new ip I've tried to manually update hostname and got same error, seems nodeName is not updatable field. It also affects all other endpoints, we were forced to recreated all pods (so they will change IPs) to be able to fix endpoints reconciliation. What you expected to happen: controller manager successfully update endpoints after node recreated with same IP, but different hostname. How to reproduce it (as minimally and precisely as possible): 1 master, 1 worker, service that points to pods with host network. k8s API service (keep it shut down) worker node with same IP and different host name to recreate pod on worker and in controller manager should appear error above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): CoreOS 1688.5.3 Kernel (e.g. ): 4.14.32-coreos Install tools: own tool () Others:
/sig node
what is the use case of changing node name?
/sig api-machinery
We have ephemeral immutable nodes in our clusters. So upgrade or any other operation will cause node recreation. Hostname generated automatically, but the IP address depends on the underlay hypervisor node, where node will be scheduled. So it's possible that node stays on the same hypervisor and uses same ip address.
/cc
/assign
/cc do you think it's reasonable to loose endpoints validation?
/sig network this API is owned by sig network
Change:
if err != nil { glog.Fatalf("FAILED: unable to get pods to list: %v", err) } <del> if err := wait.Poll(time.Second, time.Second*10, podsOnMinions(c, *pods)); err != nil { </del> <ins> if err := wait.Poll(time.Second, time.Second*30, podsOnMinions(c, *pods)); err != nil { </ins> glog.Fatalf("FAILED: pods never started running %v", err) }

Discussion:
https://travis- Setting as P0 since scheduling pods is pretty basic...
Looks like it's the "podBackoff" thing. IMO exponential backoff makes little sense for this; there's either pods that permanently won't schedule, or there's pods having some temporary problem. The former we should give up on and the latter we should retry at some fixed interval.
Actually it's not that because there's no error messages. Hm...
I think fixed this for the moment. But looking at the Travis logs the problem isn't slowness, it's a race-- the pods show up immediately after the 10 second timeout. Probably something is caching something and not updating for 10 seconds.
Actually, not fixed yet: https://travis-
Just extending timeouts isn't sufficient to fix this, or should have worked.
Could this be the problem "fixed" by
I'm betting it was actually fixed by: That's definitely a race/flake inducing bug. (but I'm not 100% positive) --brendan , bgrant0607 wrote:
Closing this as I think we found the cause, and we haven't seen it in a while.
Change:
for vc, zoneToHostsMap := range vcToZoneHostsMap { for zone, hosts := range zoneToHostsMap { commonDatastores := retrieveCommonDatastoresAmongHosts(hosts, vcToHostDatastoresMap[vc]) <del> if vcToZoneDatastoresMap[vc] == nil { vcToZoneDatastoresMap[vc] = make(map[string][]string) </del> <ins> if nm.vcToZoneDatastoresMap[vc] == nil { nm.vcToZoneDatastoresMap[vc] = make(map[string][]string) </ins> } <del> vcToZoneDatastoresMap[vc][zone] = commonDatastores </del> <ins> nm.vcToZoneDatastoresMap[vc][zone] = commonDatastores </ins> } } <del> framework.Logf("Zone to datastores map : %+v", vcToZoneDatastoresMap) </del> <ins> framework.Logf("Zone to datastores map : %+v", nm.vcToZoneDatastoresMap) </ins> return nil }

Discussion:
Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted
Change:
} // getZoneByProviderIDorName will attempt to get the zone of node using its providerID <del> // then it's name. If both attempts fail, an error is returned func getZoneByProviderIDOrName(ctx context.Context, zones cloudprovider.Zones, node *v1.Node) (cloudprovider.Zone, error) { zone, err := zones.GetZoneByProviderID(ctx, node.Spec.ProviderID) </del> <ins> // then it's name. If both attempts fail, an error is returned. func getZoneByProviderIDOrName(ctx context.Context, zones cloudprovider.Zones, providerID, nodeName string) (cloudprovider.Zone, error) { zone, err := zones.GetZoneByProviderID(ctx, providerID) </ins> if err != nil { providerIDErr := err <del> zone, err = zones.GetZoneByNodeName(ctx, types.NodeName(node.Name)) </del> <ins> zone, err = zones.GetZoneByNodeName(ctx, types.NodeName(nodeName)) </ins> if err != nil { return cloudprovider.Zone{}, fmt.Errorf("Zone: Error fetching by providerID: %v Error fetching by NodeName: %v", providerIDErr, err) }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re:
Change:
ObservedGeneration:    actualPDB.Generation, } actualStatus := actualPDB.Status <del> if !reflect.DeepEqual(actualStatus, expectedStatus) { </del> <ins> if !apiequality.Semantic.DeepEqual(actualStatus, expectedStatus) { </ins> debug.PrintStack() t.Fatalf("PDB %q status mismatch.  Expected %+v but got %+v.", key, expectedStatus, actualStatus) }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref
Change:
configs[Agnhost] = Config{e2eRegistry, "agnhost", "2.0"} configs[Alpine] = Config{dockerLibraryRegistry, "alpine", "3.7"} configs[AuthenticatedAlpine] = Config{gcAuthenticatedRegistry, "alpine", "3.7"} <ins> configs[AuthenticatedWindowsNanoServer] = Config{gcAuthenticatedRegistry, "windows-nanoserver", "v1"} </ins> configs[APIServer] = Config{e2eRegistry, "sample-apiserver", "1.10"} configs[AppArmorLoader] = Config{e2eRegistry, "apparmor-loader", "1.0"} configs[AuditProxy] = Config{e2eRegistry, "audit-proxy", "1.0"}

Discussion:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: All sig-windows jobs (master): Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this PR merged: Testgrid link: Reason for failure: The mentioned PR removed hardcoded image names from tests, centralizing them into , but there were a few omissions: entry was written for. registry set for ` is because that registry contains the Windows images needed for all the Kubernetes E2E tests. Anything else we need to know: /sig testing /sig windows /area conformance /priority important-soon
Change:
params["name"] = GetFlagString(cmd, "service-name") } if _, found := params["selector"]; !found { <ins> rc, err := client.ReplicationControllers(namespace).Get(args[0]) checkErr(err) </ins> params["selector"] = kubectl.MakeLabels(rc.Spec.Selector) } if GetFlagBool(cmd, "create-external-load-balancer") {

Discussion:
expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.
Change:
"http://metadata.google.internal/computeMetadata/v1/instance/tags?wait_for_change=true&timeout_sec=0", "http://metadata.google.internal/computeMetadata/v1/instance/tags?wait_for_change=true&last_etag=d34db33f", } <del> legacySuccessEndpoints = []string{ // Discovery "http://metadata.google.internal/0.1/meta-data", "http://metadata.google.internal/computeMetadata/v1beta1", // Allowed API versions. "http://metadata.google.internal/0.1/meta-data/", "http://metadata.google.internal/computeMetadata/v1beta1/", // Service account token endpoints. "http://metadata.google.internal/0.1/meta-data/service-accounts/default/acquire", "http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token", // Known query params. "http://metadata.google.internal/0.1/meta-data/service-accounts/default/acquire?scopes", } </del> noKubeEnvEndpoints = []string{ // Check that these don't get a recursive result. "http://metadata.google.internal/computeMetadata/v1/instance/?recursive%3Dtrue",   // urlencoded

Discussion:
Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign
Change:
// Defines a time budget that can be spend on waiting for not-ready watchers // while dispatching event before shutting them down. <del> dispatchTimeoutBudget *timeBudget </del> <ins> dispatchTimeoutBudget timeBudget </ins> // Handling graceful termination. stopLock sync.RWMutex

Discussion:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review
Change:
// Register registers a collectable metric but uses the global registry func Register(c metrics.Registerable) error { err := defaultRegistry.Register(c) <del> // sideload global prom registry as fallback prometheus.Register(c) </del> return err } // MustRegister registers registerable metrics but uses the global registry. func MustRegister(cs ...metrics.Registerable) { defaultRegistry.MustRegister(cs...) <del> // sideload global prom registry as fallback for _, c := range cs { prometheus.Register(c) } </del> } // RawMustRegister registers prometheus collectors but uses the global registry, this

Discussion:
What happened: CI failed when deprecated metrics. Refer to The reason is that a hidden metrics won't be created thus can't be registered to prometheus's registry. I guess the other PR failed for the same reason. What you expected to happen: How to reproduce it (as minimally and precisely as possible): When starting the cluster, panic happens Anything else we need to know?: In my option, we need to remove backoff from legacyregistry. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig instrumentation
Discoveries after run go test ( by ) after removing backoff and deprecating some metrics: The kubelet's metrics are protected by the unit tests, and they work well . The only block is , that is I mean register to legacy registry and gather from Prometheus.
By backoff do you mean the fallback call to prometheus.Register in the various legacyregistry register functions?
Yes. I opened a PR and see what would happen. (I guess it should be fine now)
Change:
list = &metav1.APIGroupList{ Groups: []metav1.APIGroup{ { <ins> Name: "apps", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "apps/v1beta1", Version: "v1beta1"}, {GroupVersion: "apps/v1beta2", Version: "v1beta2"}, {GroupVersion: "apps/v1beta3", Version: "v1beta3"}, {GroupVersion: "apps/v1beta4", Version: "v1beta4"}, {GroupVersion: "apps/v1beta5", Version: "v1beta5"}, {GroupVersion: "apps/v1beta6", Version: "v1beta6"}, {GroupVersion: "apps/v1beta7", Version: "v1beta7"}, {GroupVersion: "apps/v1beta8", Version: "v1beta8"}, {GroupVersion: "apps/v1beta9", Version: "v1beta9"}, {GroupVersion: "apps/v1beta10", Version: "v1beta10"}, }, }, { </ins> Name: "extensions", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "extensions/v1beta1", Version: "v1beta1"}, {GroupVersion: "extensions/v1beta2", Version: "v1beta2"}, <ins> {GroupVersion: "extensions/v1beta3", Version: "v1beta3"}, {GroupVersion: "extensions/v1beta4", Version: "v1beta4"}, {GroupVersion: "extensions/v1beta5", Version: "v1beta5"}, {GroupVersion: "extensions/v1beta6", Version: "v1beta6"}, {GroupVersion: "extensions/v1beta7", Version: "v1beta7"}, {GroupVersion: "extensions/v1beta8", Version: "v1beta8"}, {GroupVersion: "extensions/v1beta9", Version: "v1beta9"}, {GroupVersion: "extensions/v1beta10", Version: "v1beta10"}, </ins> }, }, },

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened