jiebi/CodeConvo huggingface dataset:
Code change:
// if you set fire time at X, you can get the bookmark within (X-1,X+1) period. // This is NOT thread-safe. type watcherBookmarkTimeBuckets struct { <ins> lock            sync.Mutex </ins> watchersBuckets map[int64][]*cacheWatcher startBucketID   int64 clock           clock.Clock

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc
Code change:
Describe("guestbook", func() { var guestbookPath = filepath.Join(testContext.RepoRoot, "examples/guestbook") <del> if testContext.Provider != "gce" && testContext.Provider != "gke" { By(fmt.Sprintf("Skipping guestbook, uses createExternalLoadBalancer, a (gce|gke) feature")) } </del> It("should create and stop a working application", func() { <ins> if !providerIs("gce", "gke") { By(fmt.Sprintf("Skipping guestbook, uses createExternalLoadBalancer, a (gce|gke) feature")) return } </ins> defer cleanup(guestbookPath, frontendSelector, redisMasterSelector, redisSlaveSelector) By("creating all guestbook components")

Related discussion:
text: I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.
Code change:
"//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/strategicpatch:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/version:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good
Code change:
return state } <ins> func updateRunningPodAndContainerMetrics(pods []*kubecontainer.Pod) { // Set the number of running pods in the parameter metrics.RunningPodCount.Set(float64(len(pods))) // intermediate map to store the count of each "container_state" containerStateCount := make(map[string]int) for _, pod := range pods { containers := pod.Containers for _, container := range containers { // update the corresponding "container_state" in map to set value for the gaugeVec metrics containerStateCount[string(container.State)]++ } } for key, value := range containerStateCount { metrics.RunningContainerCount.WithLabelValues(key).Set(float64(value)) } } </ins> func (pr podRecords) getOld(id types.UID) *kubecontainer.Pod { r, ok := pr[id] if !ok {

Related discussion:
text: Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /sig instrumentation /assign
GitHub didn't allow me to assign the following users: solodov. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
recommends that we generate those metrics in the GenericPLEG during the relist. We can generate the data directly from the pods call there ().
I would be interested to work on this issue if no one else is working on it currently
Go for it!
/assign
Code change:
// Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

Related discussion:
text: TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
// Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

Related discussion:
text: pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.
Code change:
// Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

Related discussion:
text: But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and
Code change:
if err != nil { return currentRevision, updateRevision, currentStatus, err } <del> // update the set's status err = ssc.updateStatefulSetStatus(ctx, set, currentStatus) if err != nil { return currentRevision, updateRevision, currentStatus, err } klog.V(4).InfoS("StatefulSet pod status", "statefulSet", klog.KObj(set), "replicas", currentStatus.Replicas, "readyReplicas", currentStatus.ReadyReplicas, "currentReplicas", currentStatus.CurrentReplicas, "updatedReplicas", currentStatus.UpdatedReplicas) </del> klog.V(4).InfoS("StatefulSet revisions", "statefulSet", klog.KObj(set), "currentRevision", currentStatus.CurrentRevision,

Related discussion:
text: I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
Code change:
} } <ins> if oldCRD.UID != newCRD.UID { r.removeStorage_locked(oldCRD.UID) } </ins> storageMap := r.customStorage.Load().(crdStorageMap) oldInfo, found := storageMap[newCRD.UID] if !found {

Related discussion:
text: Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).
Code change:
} } <ins> if oldCRD.UID != newCRD.UID { r.removeStorage_locked(oldCRD.UID) } </ins> storageMap := r.customStorage.Load().(crdStorageMap) oldInfo, found := storageMap[newCRD.UID] if !found {

Related discussion:
text: Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :
Code change:
// this is ok because we know exactly how we want to be serialized TypeMeta: metav1.TypeMeta{APIVersion: batchv1.SchemeGroupVersion.String(), Kind: "Job"}, ObjectMeta: metav1.ObjectMeta{ <del> Name:        o.Name, Annotations: annotations, Labels:      cronJob.Spec.JobTemplate.Labels, OwnerReferences: []metav1.OwnerReference{ { APIVersion: batchv1.SchemeGroupVersion.String(), Kind:       "CronJob", Name:       cronJob.GetName(), UID:        cronJob.GetUID(), }, }, </del> <ins> Name:            o.Name, Annotations:     annotations, Labels:          cronJob.Spec.JobTemplate.Labels, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cronJob, batchv1.SchemeGroupVersion.WithKind("CronJob"))}, </ins> }, Spec: cronJob.Spec.JobTemplate.Spec, }

Related discussion:
text: Given a cronjob created with a scedule of "0 6 *" that failed at last run (status.lastSuccessfulTime is now "before" status.lastScheduleTime). Then I trigger the cronjob by hand and this jobs ends successfully. However status.lastSuccessfulTime is not updated, thus making it difficult to monitor failed cronjobs. status.lastSuccessfulTime should be updated with a timestamp of the succeeded job endTime Install the attached files: Wait until cronjob has actually trigged, test with to see that lastScheduleTime and lastSuccessfulTime are populated ok (as expected). Then apply the fail config-map, now the job will not complete OK: Again wait until cronjob has trigged automatically, observe to see that lastScheduleTime is updated but lastSuccessfulTime is not. Then apply the "OK" configmap and then create the job by hand: when the job has finished and completed ok, observe that lastSuccessfulTime is still not updated. Wait until next schedule and observe that the field is now updated. No response <details</details<detailsTested on kind cluster on Mac </details<details</details<details</details<details</details<details</details
Had to upload the files as .txt files as github wouldn't accept .yaml or .yml files
(Double-check: I may read wrong messagess ) /sig apps
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : When I tried without the prefix in the job name, the status is updated. /sig apps /area job not sure if this is a bug or an expected behavior. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area workload-api/cronjob
I have just tested in my kind cluster and neither with the same naming pattern as the normal jobs (which apparently isn't epoch) or with epoch - this will not trigger a update of the field. I have also tested it on another cluster (rke2 based) and here I'm not able to get the field updated no matter which naming pattern I use.
Sorry for the mistake. I can repro the problem you have and not related to the prefix. /cc
/assign
/remove-sig apps /sig cli
/triage accepted