jiebi/CodeConvo huggingface dataset:
Code change:
func (p *PriorityQueue) MoveAllToActiveQueue() { p.lock.Lock() defer p.lock.Unlock() <ins> // There is a chance of errors when adding pods to other queues, // we make a temporary slice to store the pods, // since the probability is low, we set its len to 0 addErrorPods := make([]*framework.PodInfo, 0) </ins> for _, pInfo := range p.unschedulableQ.podInfoMap { pod := pInfo.Pod if p.isPodBackingOff(pod) { if err := p.podBackoffQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the backoff queue: %v", pod.Name, err) <ins> addErrorPods = append(addErrorPods, pInfo) </ins> } } else { if err := p.activeQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the scheduling queue: %v", pod.Name, err) <ins> addErrorPods = append(addErrorPods, pInfo) </ins> } } } p.unschedulableQ.clear() <ins> // Adding pods that we could not move to Active queue or Backoff queue back to the Unschedulable queue for _, podInfo := range addErrorPods { p.unschedulableQ.addOrUpdate(podInfo) } </ins> p.moveRequestCycle = p.schedulingCycle p.cond.Broadcast() }

Related discussion:
text: What happened: We will now always delete the pods from unschedulable queue no matter the Add operations succeed or not in function. What you expected to happen: Delete pods from unschedulable queue only when add operations succeeds
/sig scheduling
/assign
Will the pods be to the queues again somewhere if we delete them by mistake here ? It seems that can not make sure to add them again.
I am not sure if we will add them back again somewhere. If not, not only but also need to be updated. I have already send a PR here:
Code change:
if len(q.nominatedPods.nominatedPods) != 1 { t.Errorf("Expected nomindatePods to have one element: %v", q.nominatedPods) } <del> if q.unschedulableQ.get(&unschedulablePod) != &unschedulablePod { </del> <ins> if getUnschedulablePod(q, &unschedulablePod) != &unschedulablePod { </ins> t.Errorf("Pod %v was not found in the unschedulableQ.", unschedulablePod.Name) } }

Related discussion:
text: Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling
Code change:
// GetCustomResourceListerCollectionDeleter returns the ListerCollectionDeleter of // the given crd. func (r *crdHandler) GetCustomResourceListerCollectionDeleter(crd *apiextensions.CustomResourceDefinition) (finalizer.ListerCollectionDeleter, error) { <del> info, err := r.getOrCreateServingInfoFor(crd) </del> <ins> info, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name) </ins> if err != nil { return nil, err } return info.storages[info.storageVersion].CustomResource, nil } <del> func (r *crdHandler) getOrCreateServingInfoFor(crd *apiextensions.CustomResourceDefinition) (*crdInfo, error) { </del> <ins> // getOrCreateServingInfoFor gets the CRD serving info for the given CRD UID if the key exists in the storage map. // Otherwise the function fetches the up-to-date CRD using the given CRD name and creates CRD serving info. func (r *crdHandler) getOrCreateServingInfoFor(uid types.UID, name string) (*crdInfo, error) { </ins> storageMap := r.customStorage.Load().(crdStorageMap) <del> if ret, ok := storageMap[crd.UID]; ok { </del> <ins> if ret, ok := storageMap[uid]; ok { </ins> return ret, nil } r.customStorageLock.Lock() defer r.customStorageLock.Unlock() <ins> // Get the up-to-date CRD when we have the lock, to avoid racing with updateCustomResourceDefinition. // If updateCustomResourceDefinition sees an update and happens later, the storage will be deleted and // we will re-create the updated storage on demand. If updateCustomResourceDefinition happens before, // we make sure that we observe the same up-to-date CRD. crd, err := r.crdLister.Get(name) if err != nil { return nil, err } </ins> storageMap = r.customStorage.Load().(crdStorageMap) if ret, ok := storageMap[crd.UID]; ok { return ret, nil

Related discussion:
text: Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :
Code change:
else echo "Detecting nodes in the cluster" detect-node-names &> /dev/null <del> node_names=( "${NODE_NAMES[@]}" ) </del> <ins> if [[ -n "${NODE_NAMES:-}" ]]; then node_names=( "${NODE_NAMES[@]}" ) fi </ins> fi if [[ "${#node_names[@]}" == 0 ]]; then

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In https://k8s- failed due to . Afterwards crashed as follows: What you expected to happen: Graceful exit.
/assign
Code change:
// 3. While a volume is being unmounted, add it back to the desired state of world t.Logf("UnmountDevice called") generatedVolumeName, err = dsw.AddPodToVolume( <del> podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */) </del> <ins> podName, pod, volumeSpecCopy, volumeSpec.Name(), "" /* volumeGidValue */) </ins> dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName}) return nil }

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : (PR with no go code changes, should not have failed any unit tests) links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assigne
/assign
/assign tried to fix it in
Code change:
klog.V(4).Infof("GetHostsInZone %v returning: %v", zoneFailureDomain, hosts) return hosts, nil } <ins> func (nm *NodeManager) SetNodeLister(nodeLister corelisters.NodeLister) { nm.nodeLister = nodeLister } func (nm *NodeManager) SetNodeGetter(nodeGetter coreclients.NodesGetter) { nm.nodeGetter = nodeGetter } </ins>

Related discussion:
text: I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.
Code change:
klog.V(4).Infof("GetHostsInZone %v returning: %v", zoneFailureDomain, hosts) return hosts, nil } <ins> func (nm *NodeManager) SetNodeLister(nodeLister corelisters.NodeLister) { nm.nodeLister = nodeLister } func (nm *NodeManager) SetNodeGetter(nodeGetter coreclients.NodesGetter) { nm.nodeGetter = nodeGetter } </ins>

Related discussion:
text: /assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.
Code change:
} if !proxier.lbWhiteListCIDRSet.isEmpty() || !proxier.lbWhiteListIPSet.isEmpty() { // link kube-services chain -> kube-fire-wall chain <del> args := []string{"-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain)} if _, err := proxier.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, kubeServicesChain, args...); err != nil { glog.Errorf("Failed to ensure that ipset %s chain %s jumps to %s: %v", proxier.lbIngressSet.Name, kubeServicesChain, KubeFireWallChain, err) </del> <ins> args := []string{ "-A", string(kubeServicesChain), "-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain), </ins> } <ins> writeLine(proxier.natRules, args...) </ins> if !proxier.lbWhiteListCIDRSet.isEmpty() { args = append(args[:0], "-A", string(KubeFireWallChain),

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.
Code change:
command := app.NewProxyCommand() <del> utilflag.InitFlags() </del> <ins> // TODO: once we switch everything over to Cobra commands, we can go back to calling // utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the // normalize func and add the go flag set by hand. pflag.CommandLine.SetNormalizeFunc(utilflag.WordSepNormalizeFunc) pflag.CommandLine.AddGoFlagSet(goflag.CommandLine) // utilflag.InitFlags() </ins> logs.InitLogs() defer logs.FlushLogs()

Related discussion:
text: There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.
Code change:
public List<InetAddress> getSeeds() { List<InetAddress> list = new ArrayList<InetAddress>(); <del> String host = getEnvOrDefault("KUBERNETES_API_HOST","https://kubernetes.default.svc.cluster.local"); </del> <ins> //String host = "https://kubernetes.default.svc.cluster.local"; String proto = "https://"; String host = getEnvOrDefault("KUBERNETES_PORT_443_TCP_ADDR", "kubernetes.default.svc.cluster.local"); String port = getEnvOrDefault("KUBERNETES_PORT_443_TCP_PORT", "443"); </ins> String serviceName = getEnvOrDefault("CASSANDRA_SERVICE", "cassandra"); String podNamespace = getEnvOrDefault("POD_NAMESPACE", "default"); String path = String.format("/api/v1/namespaces/%s/endpoints/", podNamespace);

Related discussion:
text: What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !
Code change:
"//federation/pkg/dnsprovider/providers/aws/route53/stubs:go_default_library", "//federation/pkg/dnsprovider/rrstype:go_default_library", "//vendor/github.com/aws/aws-sdk-go/aws:go_default_library", <ins> "//vendor/github.com/aws/aws-sdk-go/aws/awserr:go_default_library", </ins> "//vendor/github.com/aws/aws-sdk-go/aws/request:go_default_library", "//vendor/github.com/aws/aws-sdk-go/aws/session:go_default_library", "//vendor/github.com/aws/aws-sdk-go/service/route53:go_default_library",

Related discussion:
text: /kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:
Code change:
Name: stats.Attributes.Metadata.Name, // The StartTime in the summary API is the container creation time. StartTime: metav1.NewTime(time.Unix(0, container.CreatedAt)), <del> // Work around heapster bug. https://github.com/kubernetes/kubernetes/issues/54962 // TODO(random-liu): Remove this after heapster is updated to newer than 1.5.0-beta.0. CPU: &statsapi.CPUStats{ UsageNanoCores: proto.Uint64(0), }, Memory: &statsapi.MemoryStats{ RSSBytes: proto.Uint64(0), }, Rootfs: &statsapi.FsStats{}, </del> <ins> CPU:       &statsapi.CPUStats{}, Memory:    &statsapi.MemoryStats{}, Rootfs:    &statsapi.FsStats{}, </ins> // UserDefinedMetrics is not supported by CRI. } if stats.Cpu != nil {

Related discussion:
text: In , we merged a work around for heapster panic We should get rid of the work around after heapster version is updated. /cc /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Code change:
// Check that container has restarted ginkgo.By("Waiting for container to restart") restarts := int32(0) <del> err = wait.PollImmediate(10*time.Second, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(10*time.Second, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value.