jiebi/CodeConvo huggingface dataset:
We should drop the bit once in favor of a once gcp has a replacement. xref: /assign /milestone v1.24 /triage accepted /sig auth
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /lifecycle frozen
I'm in the release 1.24 bug triage shadow. Does this issue still target release 1.24? If yes, would you please update the kind and priority fields for us to track it better?
Yes, definitely targeting 1.24 /priority important-soon /kind cleanup deprecation
would you please link the PR that solves this issue and update the status?
There is none yet. Waiting on GKE folks.
addresses this issue.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted
In my CSI controller I have enabled the feature gate HonorPVReclaimPolicy for csi-provisioner. Based on the documentation Finalizers can be on a PersistentVolume to ensure that PersistentVolumes having Delete reclaim policy are deleted only after the backing storage are deleted. The newly introduced finalizers and external- are only to dynamically provisioned volumes. I can see the finalizer set on the PV while creating static Persistent Volume. PV deletion is stuck due to this finalizer apiVersion: v1 kind: PersistentVolume metadata: creationTimestamp: "2023-11-08T11:39:24Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-11-08T11:54:31Z" finalizers: external- name: csi-infoscale-pv PV should be deleted when PVC is deleted. Persistent Volume yaml apiVersion: v1 kind: PersistentVolume metadata: name: csi-infoscale-pv spec: storageClassName: csi-infoscale-sc-prod persistentVolumeReclaimPolicy: Delete capacity: storage: 2Gi accessModes: ReadWriteOnce csi: driver: org.veritas.infoscale # Please provide pre-provisioned Infoscale volume name. volumeHandle: clust21432/vrtskube_dg-/vol1 fsType: vxfs Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-infoscale-pvc-static spec: accessModes: ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-infoscale-sc-prod volumeName: csi-infoscale-pv Same thing works fine on OCP. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:48:26Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Vanilla K8s setup <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage maybe similar issues should go in
Opened similar issue in external-provisioner repo also There is no reply on that as well. Can you please help understand the behavior of the feature HonorPVReclaimPolicy.
If you want Persistent Volume (PV) to be deleted when the associated Persistent Volume Claim (PVC) is deleted, you donot need to use finalizers explicitly. Kubernetes handles this behavior by default. It is ok that if you use to get the desire output using Finalizer we can do it.
As per my understanding Please follow the steps : Add finalizer in metadata of as you did above need to make change in the controller which is handling PV deletion in .go file. See whether the controller administrator is working or not. I think it will work.
Finalizer is because the feature gate HonorPVReclaimPolicy for csi-provisioner is enabled. We are not adding the finalizer explicitly. From the documentation, "The newly introduced finalizers and external- are only to dynamically provisioned volumes." Not sure why is the finalizer getting set.
Also, for deleting PV it requires the annotation "" but this annotation is set in case of dynamic PVs. Should it be set explicitly for static PV?
This feature is alpha so the feature gate is not enabled by default. Did you enable the feature gate on kube-controller-manager?
Yes, i have enabled this feature.
external- is also applied to statically provisioned volumes. FYI
Hi Which version of the external provisioner is the driver using? Does the driver meet the requirements of the CSI spec on the DeleteVolume PRC? FYI: Is this problem occasional or permanently reproducible? Can you provide the provisioner logs at that time?
/triage needs-information
cc
cc
External provisioner used is : "csi-provisioner:v3.5.0" This problem is always reproducible Provisioner logs: I1123 12:22:38. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC <nil<nilmap[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [ external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".
":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".
":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,},} I1123 12:22:38. 1 ] delete "csi-infoscale-pv": started I1123 12:22:38. 1 ] GRPC call: /csi.v1.Controller/DeleteVolume I1123 12:22:38. 1 ] GRPC request: {"volumeid":"clust40559/vrtskubedg-/vol1"} I1123 12:22:39.
1 ] GRPC response: {} I1123 12:22:39. 1 ] GRPC error: <nilI1123 12:22:39. 1 ] delete "csi-infoscale-pv": volume deleted I1123 12:22:39. 1 ] delete "csi-infoscale-pv": failed to remove finalizer for persistentvolume: Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again W1123 12:22:39. 1 ] Retrying syncing volume "csi-infoscale-pv", failure 0 E1123 12:22:39. 1 ] error syncing volume "csi-infoscale-pv": Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again I1123 12:22:39. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".
":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},} I1123 12:22:40.
1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".
":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},}
Thank you I will try to fix it. /remove-triage needs-information /assign
<!-- Please only use this template for submitting enhancement requests --What would you like to be : and functions need unit tests Why is this needed: /kind test /sig cloud-provider /area provider/azure /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What would you like to be : /sig cloud-provider /area provider/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
related to
cc
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I'd like to give this one a go, if no one is currently working on it.
sure, go ahead, thanks
I pushed some first tests for
This was the root cause of rancher/rancher. In , when is called, it is given the oldVersion of an object and the newVersion of the object. It then uses the indexers to calculate the old values (from the old version) and the new updated values. In cases where are size of 1, are of size greater than 1, and contain the sole value from , the key will be deleted for the old value will be deleted and won't be re-. This is due to the two for loops at the end of the method which are used for optimization. When the first for loop is processed, . Because of this, the key is deleted from the index for this value. In the second for loop, the , and for at least one of the values . The continue statement causes the key to not be to the index for this value (and since the value was deleted in the prior for loop, it no longer exists in the index for this value), despite the fact that the indicated that this is the desired state. Since specified that this value was still valid for this object, the key should not be deleted from the index for this value, or it should be re- when it is re-discovered in . Either way, the end state of the index should include this key in the set for the specified value. I've created a which demonstrates the issue (branch is ). To use it (assuming that you have a running kubernetes cluster to use with): the controller: . the foo crd: the controller (assumes that you have $KUBECONFIG set to the file path of your kubeconfig file). an example foo with only one label: . Observe, in the logs of the running sample-controller, that the indexer was able to locate this foo using the indexer. the foo to include a second label: . Observe, in the logs of the running sample-controller, that the indexer was not able to locate the foo using the first key any longer, despite the fact that the label was still on the object. The sample-controller fork adds an indexer for foo objects which uses the indexerFunction of . This function looks for labels that have a key with a prefix of and uses their values as index keys for the foo objects. For example, if I have a with a label of , the indexer will allow you to retrieve using .
The controller's method has been altered to just ensure that the foos can be retrieved using the indexer for all valid labels. If the controller can't retrieve foo by using the value from a valid label, then it returns an error (it tries for all valid labels). This was originally discovered in a fork of client-go that we maintain. After we reverted the changes introduced by , our issues stopped occurring. N/A - This was introduced in and appears to be in all versions of the client v0.23+. N/A - issue is with client-go No response No response No response No response
/sig api-machinery
I think that you have background in this issue/area of the code. Would you mind taking a look at this issue and the PR that I have up to fix it? Please let me know if I've misunderstood anything.
/assign Would you mind to take a look? Thank you /triage accepted
Thanks for reporting the issue. will address this issue
Reopening - since we need to cherrypick back to 1.23
Cherrypick is merged - closing.
What happened: node-local-dns caches negative queries, causing things that do polling for orchestration to fail (like operators with StatefulSets). What you expected to happen: Have a minimal TTL for negative responses (alternatively just disable caching by setting the cache size to 0) that follows what was done for kube-dns: How to reproduce it (as minimally and precisely as possible): With node-local-dns cache deployed, in my cluster it was the that failed to create clusters because the cached entry didn't expire before it had exhausted all of it's retries. Environment: Kubernetes version (use ): 1.11.6 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): any Kernel (e.g. ): any
/sig network
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work.
Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted