jiebi/CodeConvo huggingface dataset:
Change:
mountName := fmt.Sprintf("path%d", i) volumes[i].Name = mountName if src == "" { <del> volumes[i].VolumeSource.EmptyDir = &v1.EmptyDirVolumeSource{Medium: v1.StorageMediumMemory} </del> <ins> volumes[i].VolumeSource.EmptyDir = &v1.EmptyDirVolumeSource{} </ins> } else { volumes[i].VolumeSource.HostPath = &v1.HostPathVolumeSource{ Path: src,

Discussion:
Noticing this is in our gce 5k-node cluster tests. It has been continuously failing for 4 runs now already: https://k8s- https://k8s- (and 39 and 40) The e2e test concerned is - Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content And it is failing due to: cc
cc - Could one of you PTAL or redirect as apt?
FTR, it is passing on small scale - https://k8s- I took a quick look at the and the following PR catches my attention at first sight: test: e2e: support NFS test on overlayfs
Do we have some umbrella issue for failing correctness tests on large scale atm?
Not really, we have , though.
does look very suspicious. Before, the NFS server test image would use the container image layer for its data, and this change makes it use emptydir tmpfs instead. for the large scale tests, do you guys have default memory limits for pods? That could potentially explain why it's failing on large scale tests and not the regular tests. A possible fix could be to change it to use regular emptydir so it's not subject to any memory limits.
FMU we're not setting any memory limits for pods explicitly and it's the same e2e test running on both the large cluster and the small cluster. I'm not sure if there's sth (like an admission plugin) which secretly sets memory limits for pods based on cluster size - I'd be quite surprised if that's the case.
Hmm or it could be that in the scale test, the nodes are more loaded, so resources could be more constrained. In any case, I think it's better for this test to not write to tmpfs, because it can write gigs of data.
I would be surprised a bit if that was the case - as this test was passing for quite some time (https://k8s-). So mostly unless sth changed around this e2e test or with storage/kubernetes, it doesn't seem to be so. Anyhow, not writing gigs of data to tmpfs does sound good to me :) One more thing to note is we're using the following config for our large cluster:
Yah so this test got changed from using disk to using memory by , which is when this test started failing.

So I suspect it is related to either a memory limit or memory pressure.
That seems to have . Thanks
More references FTR: https://k8s- https://k8s- https://k8s-
Change:
// GetDatastoresInZone returns all the datastores in the specified zone func (nm *NodeMapper) GetDatastoresInZone(vc string, zone string) []string { <del> return vcToZoneDatastoresMap[vc][zone] </del> <ins> nm.nodeInfoRWLock.RLock() defer nm.nodeInfoRWLock.RUnlock() return nm.vcToZoneDatastoresMap[vc][zone] </ins> } // GetNodeInfo returns NodeInfo for given nodeName func (nm *NodeMapper) GetNodeInfo(nodeName string) *NodeInfo { <del> return nameToNodeInfo[nodeName] </del> <ins> nm.nodeInfoRWLock.RLock() defer nm.nodeInfoRWLock.RUnlock() return nm.nameToNodeInfo[nodeName] </ins> } // SetNodeInfo sets NodeInfo for given nodeName. This function is not thread safe. Users need to handle concurrency. func (nm *NodeMapper) SetNodeInfo(nodeName string, nodeInfo *NodeInfo) { <del> nameToNodeInfo[nodeName] = nodeInfo </del> <ins> nm.nodeInfoRWLock.Lock() defer nm.nodeInfoRWLock.Unlock() nm.nameToNodeInfo[nodeName] = nodeInfo </ins> }

Discussion:
Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted
Change:
"k8s.io/apimachinery/pkg/util/uuid" utilversion "k8s.io/apimachinery/pkg/util/version" clientset "k8s.io/client-go/kubernetes" <ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" </ins> "k8s.io/kubernetes/test/e2e/framework" e2eevents "k8s.io/kubernetes/test/e2e/framework/events" e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet"

Discussion:
What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign
Change:
return clientConfig } <ins> // toRawKubePersistentConfigLoader binds config flag values to config overrides // Returns a persistent clientConfig for propagation. func (f *ConfigFlags) toRawKubePersistentConfigLoader() clientcmd.ClientConfig { f.lock.Lock() defer f.lock.Unlock() if f.clientConfig == nil { f.clientConfig = f.toRawKubeConfigLoader() } return f.clientConfig } </ins> // ToDiscoveryClient implements RESTClientGetter. // Expects the AddFlags method to have been called. // Returns a CachedDiscoveryInterface using a computed RESTConfig.

Discussion:
/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!
Change:
type DesiredStateOfWorldPopulator interface { Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) <del> // ReprocessPod removes the specified pod from the list of processedPods // (if it exists) forcing it to be reprocessed. This is required to enable </del> <ins> // ReprocessPod sets value for the specified pod in processedPods // to false, forcing it to be reprocessed. This is required to enable </ins> // remounting volumes on pod updates (volumes like Downward API volumes // depend on this behavior to ensure volume content is updated). ReprocessPod(podName volumetypes.UniquePodName)

Discussion:
It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!
Change:
selfLinkPrefix := "" switch crd.Spec.Scope { case apiextensions.ClusterScoped: <del> selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version) + "/" </del> <ins> selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version) + "/" + crd.Status.AcceptedNames.Plural + "/" </ins> case apiextensions.NamespaceScoped: selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version, "namespaces") + "/" }

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a custom resource definition with -this is important - scope "Cluster". Then I created an element of that kind. Then I fetched the element and saw that the "metadata.selfLink" attribute had an incorrect value. The path did not contain the custom resource name. Trying to fetch the element using API results in 404 response. What you expected to happen: The "metadata.selfLink" of an instance of a custom resource should be correct. How to reproduce it (as minimally and precisely as possible): Prepare a custom resource definition with scope: Cluster (important! with scope Namespaced the selfLink is correct) in a file (slightly modified standard example from here: ) Create it: Now prepare a custom object in a file my- with this content: And create it: Now get the custom object list: The result is: As you can see the selfLink is . This is wrong, it should be . Using the selfLink as it is causes 404 when fetching the resource via API. Anything else we need to know?: This only happens when the custom resource definition has scope: Cluster. Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: 2017 Macbook Pro OS (e.g. from /etc/os-release): macOS High Sierra 10.13.3 Kernel (e.g. ): Darwin Kernel Version 17.4.0 Install tools: minikube 0.24.1
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Created the PR:
/cc
Change:
// enough space as per the garbage collection policy. GarbageCollect() error <ins> // Start async garbage collection of images. Start() error </ins> // TODO(vmarmol): Have this subsume pulls as well. }

Discussion:
I'm trying to start 100-node cluster. I got the following error in kubelet: E0505 08:31:55. 5066 ] Failed to create kubelet: failed to initialize image manager: failed to start image manager: dial unix connection refused After ssh-ing I checked that docker is working by running . I guess it must be some kind of a race where kubelet starts before docker is ready. When I start 3-node cluster it works just fine. This happened to me 3 times in a row. Please add this to 1.0 milestone if necessary.
I've just exactly the same problem (when creating a 100-node cluster)
This is a bug in our initialization logical. We initialize image manager which needs to talk to docker at very early stage before docker is up yet. Later kubelet does a wait to make sure docker is up. We should either delay the initialization of image manager or handle docker unavailable better. Kubelet will be restarted by monit eventually, but this definitely slow down node initialization. I mark this for 1.0. Should be a simple fix if we just simply delay the initialization of image manager.
I'll go ahead and fix it (since I am mainly to blame :) )
Changed to P0 because asked us to only add P0 issues to v1.0 milestone.
Change:
go.etcd.io/etcd/pkg/v3 v3.5.13 // indirect go.etcd.io/etcd/raft/v3 v3.5.13 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect <del> go.opentelemetry.io/otel/metric v1.28.0 // indirect </del> go.opentelemetry.io/proto/otlp v1.3.1 // indirect go.uber.org/multierr v1.11.0 // indirect golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect

Discussion:
Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:
Change:
}) ginkgo.It("The containers terminated by OOM killer should have the reason set to OOMKilled", func() { <ins> cfg, configErr := getCurrentKubeletConfig(context.TODO()) framework.ExpectNoError(configErr) if utilfeature.DefaultFeatureGate.Enabled(features.NodeSwap) { // If Swap is enabled, we should test OOM with LimitedSwap. // UnlimitedSwap allows for workloads to use unbounded swap which // makes testing OOM challenging. // We are not able to change the default for these conformance tests, // so we will skip these tests if swap is enabled. if cfg.MemorySwap.SwapBehavior == "" || cfg.MemorySwap.SwapBehavior == "UnlimitedSwap" { ginkgo.Skip("OOMKiller should not run with UnlimitedSwap") } } </ins> ginkgo.By("Waiting for the pod to be failed") err := e2epod.WaitForPodTerminatedInNamespace(context.TODO(), f.ClientSet, testCase.podSpec.Name, "", f.Namespace.Name) framework.ExpectNoError(err, "Failed waiting for pod to terminate, %s/%s", f.Namespace.Name, testCase.podSpec.Name)

Discussion:
The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm
Change:
type azureToken struct { token       adal.Token <ins> environment string </ins> clientID    string tenantID    string apiserverID string

Discussion:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
// Close the queue. func (f *DeltaFIFO) Close() { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> f.closed = true f.cond.Broadcast() }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile: