jiebi/CodeConvo huggingface dataset:
Change:
bs, err := gce.GetRegionBackendService(bsName, gce.region) assert.NoError(t, err) <del> // Check that the instances are updated newNodes, err := createAndInsertNodes(gce, newNodeNames, vals.ZoneName) newIgLinks, err := gce.ensureInternalInstanceGroups(igName, newNodes) backends := backendsFromGroupLinks(newIgLinks) </del> <ins> // Check that the Backends reflect the new InstanceGroups backends := backendsFromGroupLinks(newIGLinks) </ins> assert.Equal(t, bs.Backends, backends) }) }

Discussion:
Seeing several of the tests in ./pkg/cloudprovider/providers/gce flake 1 in 5-10 runs when run locally (I am logged in to gcloud). Is it possible one of the APIs is not correctly shimmed and is fetching real data?
you were the last folks in here - why would this and other internal_tests in here be flaking?
Happens in both master and 1.11 release branches
/cc cheftako
Change:
software-properties-common  lsb-release <ins> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 release=$(lsb_release -cs) if [ "$release" == "focal" ]; then release="bionic"; fi </ins> # Add the Docker apt-repository (as we install containerd from there) curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg  | apt-key add - add-apt-repository  "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")  <del> $(lsb_release -cs) stable" </del> <ins> $release stable" </ins> # Install containerd from Docker repo apt-get update && 

Discussion:
Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:42.774] scp: /var/log/konnectivity- No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: *: No such file or directory W0501 09:18:42.780] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
I am seeing the same error listed above on successful runs as well so I think the issue might be elsewhere
/assign

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:22.072] scp: /var/log/konnectivity- No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.073] scp: *: No such file or directory W0501 09:18:22.075] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
/assign
Change:
nodeExpansionRequired   bool disableAttach           bool disableResizingOnDriver bool <ins> simulatedCSIDriverError expansionStatus </ins> expectFailure           bool }{ { <del> name:                  "should expand volume without restarting pod if nodeExpansion=off", nodeExpansionRequired: false, </del> <ins> name:                    "should expand volume without restarting pod if nodeExpansion=off", nodeExpansionRequired:   false, simulatedCSIDriverError: expansionSuccess, }, { name:                    "should expand volume by restarting pod if attach=on, nodeExpansion=on", nodeExpansionRequired:   true, simulatedCSIDriverError: expansionSuccess, </ins> }, { <del> name:                  "should expand volume by restarting pod if attach=on, nodeExpansion=on", nodeExpansionRequired: true, </del> <ins> name:                    "should not have staging_path missing in node expand volume pod if attach=on, nodeExpansion=on", nodeExpansionRequired:   true, simulatedCSIDriverError: expansionFailedMissingStagingPath, </ins> }, { <del> name:                  "should expand volume by restarting pod if attach=off, nodeExpansion=on", disableAttach:         true, nodeExpansionRequired: true, </del> <ins> name:                    "should expand volume by restarting pod if attach=off, nodeExpansion=on", disableAttach:           true, nodeExpansionRequired:   true, simulatedCSIDriverError: expansionSuccess, </ins> }, { name:                    "should not expand volume if resizingOnDriver=off, resizingOnSC=on", disableResizingOnDriver: true, expectFailure:           true, <ins> simulatedCSIDriverError: expansionSuccess, </ins> }, } for _, t := range tests {

Discussion:
When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted
Change:
namespace: kube-system labels: k8s-app: elasticsearch-logging <del> version: v7.4.2 </del> <ins> version: v7.4.3 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: serviceName: elasticsearch-logging

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: es-image as part of fluent-elasticsearch cluster addon does not build What you expected to happen: in the es-image folder should work How to reproduce it (as minimally and precisely as possible): cd cluster/addons/fluentd-elasticsearch/es-image make Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
/sig instrumentation
Change:
claimsToBind:     []*v1.PersistentVolumeClaim{}, podVolumesByNode: map[string]*scheduling.PodVolumes{}, }, <del> wantFilterStatus: framework.NewStatus(framework.Error, `could not find v1.PersistentVolume "pv-a"`), </del> <ins> wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, `pvc(s) bound to non-existent pv(s)`), </ins> }, }

Discussion:
What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point
Change:
// ControlPlaneComponent holds settings common to control plane component of the cluster type ControlPlaneComponent struct { // ExtraArgs is an extra set of flags to pass to the control plane component. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> // TODO: This is temporary and ideally we would like to switch all components to // use ComponentConfig + ConfigMaps. ExtraArgs map[string]string

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm
Change:
} // Pod should fail <del> testPodFailSubpath(f, pod, "SubPath `..`: must not contain '..'") </del> <ins> testPodFailSubpath(f, pod) </ins> }) /* Testname: var-expansion-subpath-with-absolute-path Description: Make sure a container's subpath can not be set using an <del> expansion of environment variables when absoluete path is supplied. </del> <ins> expansion of environment variables when absolute path is supplied. </ins> */ It("should fail substituting values in a volume subpath with absolute path [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion][Slow]", func() {

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Looks like the newly tests are failing: https://k8s-
/assign Also random question to why is the same test run 4 times?
GitHub didn't allow me to assign the following users: kevtaylor. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally
Reiterating the mentions to trigger a notification: <detailsIn response to : can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suggest unless anyone can define why the events do not appear, we disable that part of the test and just check for error. If you are okay with that please reopen the bug and I'll put in a new fix
it looks like node alpha suite is having some other issues right now that are preventing the tests from being run. So it's not clear right now if the fix helped or not.
it failed again. I did a comparison of a locally running one and the alpha suite. In the log file https://k8s- there are these references But in the locally running tests I don't see that step This running And the events running locally
/reopen
Change:
value.RecordCacheMiss() } uid := string(uuid.NewUUID()) <ins> klog.V(6).InfoS("Decrypting content using envelope service", "uid", uid, "key", string(dataCtx.AuthenticatedData())) </ins> key, err := t.envelopeService.Decrypt(ctx, uid, &DecryptRequest{ Ciphertext:  encryptedObject.EncryptedDEK, KeyID:       encryptedObject.KeyID,

Discussion:
Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.
Change:
stopCh: make(chan struct{}), stopWg: sync.WaitGroup{}, } <ins> // See startCaching method for explanation and where this is unlocked. </ins> cacher.usable.Lock() <del> // See startCaching method for why explanation on it. watchCache.SetOnReplace(func() { cacher.usable.Unlock() }) </del> watchCache.SetOnEvent(cacher.processEvent) stopCh := cacher.stopCh

Discussion:
cc
This is still hapening. Assigning Isaac for triage.
Three times in the last three days, BeforeSuite failed (filed ). Otherwise, just Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) is just totally borked (filed ). what prevented you from triaging this yourself? It'd be good to know so I'm less in the critical path.
Sorry, I read this wrong. I was looking at the gke-serial suite. Triaging now.
It looks like massive failures are correlated extremely strongly with failure of Etcd failure [Disruptive] should recover from SIGKILL Every time that test passes, everything else (except ) passes; every time it fails, the whole suite goes south. Reassigning to for triage.
sorry for delegating the work to you, but I completely didn't have time to do it and nobody else hasn't volunteered to do it for a while. it seems you are the author of the test, so assigning you.
No problem Just wanted to make sure there wasn't some information missing.
I'm looking at this today
It looks like the token controller is not recovering from etcd failing. The test namespace is also stuck in a terminating phase. I'll continue to investigate
Actually lot's of watch related things are broken. I can create/delete objects in the api but watch is broken. Trying to establish a watch in this state causes: Restarting the apiserver fixes the problem. I can start a watch again: cc
Not particularly useful apiserver logs Also none of the components are trying to reestablish watches so I'm assuming that they still have watches open but are not receiving anything.
startCaching is trying to lock here, but something is holding the lock. I'm looking through a stack trace right now for the offender. This is exactly where I'd expect a problem to be from the description of "bringing etcd down and up repeatedly".
Ah I think what happened is, when etcd stays down, a List never successfully finishes, and therefore it never calls the OnReplace function, and therefore the lock is never unlocked, and therefore startCaching can't exit.

There's lots in apiserver log to support this: Additionally I manually looked through the stack trace and confirmed that all of the stuck calls to Watch are stuck waiting to RLock and not some point afterwards.
This is pretty bad, etcd going down for just a bit could freeze up apiserver until it is rebooted.
I think I can make a fix.
Thanks for looking into this.
Yeah, I repro'd in a UT. Fix coming shortly.
I assigned to but if anyone wants it merged sooner than monday, feel free to review.
Many thanks to for handing me a broken server to look at, it made this super easy to find.
Change:
go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful v0.42.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 <ins> go.opentelemetry.io/otel/metric v1.28.0 </ins> go.opentelemetry.io/otel/sdk v1.28.0 go.opentelemetry.io/otel/trace v1.28.0 go.opentelemetry.io/proto/otlp v1.3.1

Discussion:
Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks: