Change:
What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.

Discussion:
ProvisioningCleanupFailed = "ProvisioningCleanupFailed" ProvisioningSucceeded     = "ProvisioningSucceeded" WaitForFirstConsumer      = "WaitForFirstConsumer" <ins> WaitForPodScheduled       = "WaitForPodScheduled" </ins> ExternalExpanding         = "ExternalExpanding" )

podIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) for _, pod := range pods { podIndexer.Add(pod) <ins> ctrl.podIndexer.Add(pod) </ins> } ctrl.podLister = corelisters.NewPodLister(podIndexer)

storage "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/labels" </del> "k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" clientset "k8s.io/client-go/kubernetes"

classListerSynced  cache.InformerSynced podLister          corelisters.PodLister podListerSynced    cache.InformerSynced <ins> podIndexer         cache.Indexer </ins> NodeLister         corelisters.NodeLister NodeListerSynced   cache.InformerSynced

return nil } <ins> // emitEventForUnboundDelayBindingClaim generates informative event for claim // if it's in delay binding mode and not bound yet. func (ctrl *PersistentVolumeController) emitEventForUnboundDelayBindingClaim(claim *v1.PersistentVolumeClaim) error { reason := events.WaitForFirstConsumer message := "waiting for first consumer to be created before binding" podNames, err := ctrl.findNonScheduledPodsByPVC(claim) if err != nil { return err } if len(podNames) > 0 { reason = events.WaitForPodScheduled if len(podNames) > 1 { // Although only one pod is taken into account in // volume scheduling, more than one pods can reference // the PVC at the same time. We can't know which pod is // used in scheduling, all pods are included. message = fmt.Sprintf("waiting for pods %s to be scheduled", strings.Join(podNames, ",")) } else { message = fmt.Sprintf("waiting for pod %s to be scheduled", podNames[0]) } } ctrl.eventRecorder.Event(claim, v1.EventTypeNormal, reason, message) return nil } </ins> // syncUnboundClaim is the main controller method to decide what to do with an // unbound claim. func (ctrl *PersistentVolumeController) syncUnboundClaim(claim *v1.PersistentVolumeClaim) error {

// OBSERVATION: pvc is "Pending", will retry switch { case delayBinding && !pvutil.IsDelayBindingProvisioning(claim): <del> ctrl.eventRecorder.Event(claim, v1.EventTypeNormal, events.WaitForFirstConsumer, "waiting for first consumer to be created before binding") </del> <ins> if err = ctrl.emitEventForUnboundDelayBindingClaim(claim); err != nil { return err } </ins> case v1helper.GetPersistentVolumeClaimClass(claim) != "": if err = ctrl.provisionClaim(claim); err != nil { return err

return true, nil } <ins> func (ctrl *PersistentVolumeController) findPodsByPVCKey(key string) ([]*v1.Pod, error) { pods := []*v1.Pod{} objs, err := ctrl.podIndexer.ByIndex(pvcKeyIndex, key) if err != nil { return pods, err } for _, obj := range objs { pod, ok := obj.(*v1.Pod) if !ok { continue } pods = append(pods, pod) } return pods, err } </ins> // isVolumeUsed returns list of pods that use given PV. func (ctrl *PersistentVolumeController) isVolumeUsed(pv *v1.PersistentVolume) ([]string, bool, error) { if pv.Spec.ClaimRef == nil { return nil, false, nil } <del> claimName := pv.Spec.ClaimRef.Name </del> podNames := sets.NewString() <del> pods, err := ctrl.podLister.Pods(pv.Spec.ClaimRef.Namespace).List(labels.Everything()) </del> <ins> pvcKey := fmt.Sprintf("%s/%s", pv.Spec.ClaimRef.Namespace, pv.Spec.ClaimRef.Name) pods, err := ctrl.findPodsByPVCKey(pvcKey) </ins> if err != nil { <del> return nil, false, fmt.Errorf("error listing pods: %s", err) </del> <ins> return nil, false, fmt.Errorf("error finding pods by pvc %q: %s", pvcKey, err) </ins> } for _, pod := range pods { <del> if util.IsPodTerminated(pod, pod.Status) { continue } for i := range pod.Spec.Volumes { usedPV := &pod.Spec.Volumes[i] if usedPV.PersistentVolumeClaim != nil && usedPV.PersistentVolumeClaim.ClaimName == claimName { podNames.Insert(pod.Namespace + "/" + pod.Name) } } </del> <ins> podNames.Insert(pod.Namespace + "/" + pod.Name) </ins> } return podNames.List(), podNames.Len() != 0, nil } <ins> // findNonScheduledPodsByPVC returns list of non-scheduled pods that reference given PVC. func (ctrl *PersistentVolumeController) findNonScheduledPodsByPVC(pvc *v1.PersistentVolumeClaim) ([]string, error) { pvcKey := fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name) pods, err := ctrl.findPodsByPVCKey(pvcKey) if err != nil { return nil, err } podNames := []string{} for _, pod := range pods { if len(pod.Spec.NodeName) == 0 { podNames = append(podNames, pod.Name) } } return podNames, nil } </ins> // doDeleteVolume finds appropriate delete plugin and deletes given volume, returning // the volume plugin name. Also, it returns 'true', when the volume was deleted and // 'false' when the volume cannot be deleted because the deleter is external. No

"k8s.io/kubernetes/pkg/util/goroutinemap" vol "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/csimigration" <ins> "k8s.io/kubernetes/pkg/volume/util" </ins> "k8s.io/klog/v2" )

controller.classLister = p.ClassInformer.Lister() controller.classListerSynced = p.ClassInformer.Informer().HasSynced controller.podLister = p.PodInformer.Lister() <ins> controller.podIndexer = p.PodInformer.Informer().GetIndexer() </ins> controller.podListerSynced = p.PodInformer.Informer().HasSynced controller.NodeLister = p.NodeInformer.Lister() controller.NodeListerSynced = p.NodeInformer.Informer().HasSynced <ins> // This custom indexer will index pods by its PVC keys. Then we don't need // to iterate all pods every time to find pods which reference given PVC. if err := controller.podIndexer.AddIndexers(cache.Indexers{ pvcKeyIndex: indexByPVCKey, }); err != nil { return nil, fmt.Errorf("Could not initialize PersistentVolume Controller: %v", err) } </ins> csiTranslator := csitrans.New() controller.translator = csiTranslator controller.csiMigratedPluginManager = csimigration.NewPluginManager(csiTranslator)

// Stateless functions <ins> const ( pvcKeyIndex string = "pvc-key-index" ) // indexByPVCKey returns PVC keys for given pod func indexByPVCKey(obj interface{}) ([]string, error) { pod, ok := obj.(*v1.Pod) if !ok { return []string{}, nil } if util.IsPodTerminated(pod, pod.Status) { return []string{}, nil } keys := []string{} for _, podVolume := range pod.Spec.Volumes { if pvcSource := podVolume.VolumeSource.PersistentVolumeClaim; pvcSource != nil { keys = append(keys, fmt.Sprintf("%s/%s", pod.Namespace, pvcSource.ClaimName)) } } return keys, nil } </ins> func getClaimStatusForLogging(claim *v1.PersistentVolumeClaim) string { bound := metav1.HasAnnotation(claim.ObjectMeta, pvutil.AnnBindCompleted) boundByController := metav1.HasAnnotation(claim.ObjectMeta, pvutil.AnnBoundByController)