jiebi/CodeConvo huggingface dataset:
Code change:
}, }, Command: []string{"/bin/sh"}, <del> Args:    []string{"-c", "python ./data_download.py && time -p python ./wide_deep.py --model_type=wide_deep --train_epochs=300 --epochs_between_evals=300 --batch_size=32561"}, </del> <ins> Args:    []string{"-c", "time -p python ./wide_deep.py --model_type=wide_deep --train_epochs=300 --epochs_between_evals=300 --batch_size=32561"}, </ins> } containers = append(containers, ctn)

Related discussion:
text: Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.
Code change:
serviceLister:                           serviceLister, serviceHasSynced:                        serviceHasSynced, nodeLister:                              nodeLister, <ins> nodeHasSynced:                           nodeHasSynced, </ins> masterServiceNamespace:                  masterServiceNamespace, streamingConnectionIdleTimeout:          kubeCfg.StreamingConnectionIdleTimeout.Duration, recorder:                                kubeDeps.Recorder,

Related discussion:
text: What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.
Code change:
serviceLister:                           serviceLister, serviceHasSynced:                        serviceHasSynced, nodeLister:                              nodeLister, <ins> nodeHasSynced:                           nodeHasSynced, </ins> masterServiceNamespace:                  masterServiceNamespace, streamingConnectionIdleTimeout:          kubeCfg.StreamingConnectionIdleTimeout.Duration, recorder:                                kubeDeps.Recorder,

Related discussion:
text: Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
} output, err := json.Marshal(agg) require.NoError(t, err) <del> // Content-type is "aggregated" discovery format. w.Header().Set("Content-Type", AcceptV2Beta1) </del> <ins> // Content-type is "aggregated" discovery format. Add extra parameter // to ensure we are resilient to these extra parameters. w.Header().Set("Content-Type", AcceptV2Beta1+"; charset=utf-8") </ins> w.WriteHeader(status) w.Write(output) }))

Related discussion:
text: We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :
Code change:
if kl.kubeClient == nil { return kl.initialNode(context.TODO()) } <ins> // if we have a valid kube client, we wait for initial lister to sync if !kl.nodeHasSynced() { err := wait.PollImmediate(time.Second, maxWaitForAPIServerSync, func() (bool, error) { return kl.nodeHasSynced(), nil }) if err != nil { return nil, fmt.Errorf("nodes have not yet been read at least once, cannot construct node object") } } </ins> return kl.nodeLister.Get(string(kl.nodeName)) }

Related discussion:
text: What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.
Code change:
if kl.kubeClient == nil { return kl.initialNode(context.TODO()) } <ins> // if we have a valid kube client, we wait for initial lister to sync if !kl.nodeHasSynced() { err := wait.PollImmediate(time.Second, maxWaitForAPIServerSync, func() (bool, error) { return kl.nodeHasSynced(), nil }) if err != nil { return nil, fmt.Errorf("nodes have not yet been read at least once, cannot construct node object") } } </ins> return kl.nodeLister.Get(string(kl.nodeName)) }

Related discussion:
text: Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Code change:
for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.LoadBalancerIPMode, tc.ipModeEnabled)() <del> s := core.LoadBalancerStatus{} tc.tweakLBStatus(&s) errs := ValidateLoadBalancerStatus(&s, field.NewPath("status")) </del> <ins> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.AllowServiceLBStatusOnNonLB, tc.nonLBAllowed)() status := core.LoadBalancerStatus{} tc.tweakLBStatus(&status) spec := core.ServiceSpec{Type: core.ServiceTypeLoadBalancer} if tc.tweakSvcSpec != nil { tc.tweakSvcSpec(&spec) } errs := ValidateLoadBalancerStatus(&status, field.NewPath("status"), &spec) </ins> if len(errs) != tc.numErrs { t.Errorf("Unexpected error list for case %q(expected:%v got %v) - Errors:n %v", tc.name, tc.numErrs, len(errs), errs.ToAggregate()) }

Related discussion:
text: xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
expectError: false, }, { <del> name: "subpath-mounting-unix-socket", </del> <ins> name: "mount-unix-socket", </ins> prepare: func(base string) ([]string, string, string, error) { volpath, subpathMount := getTestPaths(base) mounts := []string{subpathMount}

Related discussion:
text: Subpath unix socket tests can flake because of long socket path. On Linux the path limit should be 108 chars or less. /sig storage
Code change:
<ins> # Rollback workflow Build it in this directory. Make sure you have etcd dependency ready. Last time we use etcd v3.0.7. ``` $ go build . ``` Run it: ``` $ ./rollback2 --data-dir $ETCD_DATA_DIR --ttl 1h ``` This will rollback KV pairs from v3 into v2. If a key was attached to a lease before, it will be created with given TTL (default to 1h). On success, it will print at the end: ``` Finished successfully ``` Repeat this on all etcd members. You can do simple check on keys (if any exists): ``` etcdctl ls / ``` Important Note ------ This tool isn't recommended to use if any problem comes up in etcd3 backend. Please report bugs and we will fix it soon. If it's still preferred to run this tool, please backup all your data beforehand. This tool will also back up datadir to same path with ".rollback.backup" suffix. Caveats: - The tool doesn't preserve versions of keys. - If any v2 data exists before rollback, they will be wiped out. - v3 data only exists in the backup after successful rollback. [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/cluster/images/etcd/rollback/README.md?pixel)]() </ins>  No newline at end of file

Related discussion:
text: If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is different with the failure after the revert. We(release signal team) plan to open an issue for the flake later. (Probably, we should open that one and then close on behalf of that new issue later.)
Feel free to close it then. However, it failed 3 times out of last 4 runs. It's more a failure than a flake in my opinion. If doesn't help/ not appropriate, I'd suggest to revert installer and image changes
comment is clear about the root cause, we have a problem installing the drivers, can you redirect this issue if you think is a bug?
After was merged, there are 6 failure in last 24 run. Tab stats: 8 of 10 (80.0%) recent columns passed (258 of 262 or 98.5% cells) The Flaky rate is lower.
Code change:
return nil case watch.Error: <del> watcher.Stop() watcherClosed = true // start another cycle return c.waitForVolumeDetachmentInternal(volumeHandle, attachID, timer, timeout) </del> <ins> klog.Warningf("waitForVolumeDetachmentInternal received watch error: %v", event) </ins> } case <-timer.C:

Related discussion:
text: A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.
Code change:
return nil case watch.Error: <del> watcher.Stop() watcherClosed = true // start another cycle return c.waitForVolumeDetachmentInternal(volumeHandle, attachID, timer, timeout) </del> <ins> klog.Warningf("waitForVolumeDetachmentInternal received watch error: %v", event) </ins> } case <-timer.C:

Related discussion:
text: if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details