Change:
What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate). is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.

Discussion:
deps = [ "//pkg/api/v1/pod:go_default_library", "//pkg/apis/core/helper:go_default_library", <ins> "//pkg/apis/core/v1/helper:go_default_library", </ins> "//pkg/features:go_default_library", "//pkg/kubelet/apis:go_default_library", "//pkg/scheduler:go_default_library",

"k8s.io/client-go/tools/cache" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/apis/core/helper" <ins> v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" </ins> "k8s.io/kubernetes/pkg/features" kubeletapis "k8s.io/kubernetes/pkg/kubelet/apis" "k8s.io/kubernetes/pkg/scheduler"

cache.ResourceEventHandlerFuncs{ // MaxPDVolumeCountPredicate: since it relies on the counts of PV. AddFunc:    c.onPvAdd, <ins> UpdateFunc: c.onPvUpdate, </ins> DeleteFunc: c.onPvDelete, }, )

} } <ins> func (c *configFactory) onPvUpdate(old, new interface{}) { if c.enableEquivalenceClassCache { newPV, ok := new.(*v1.PersistentVolume) if !ok { glog.Errorf("cannot convert to *v1.PersistentVolume: %v", new) return } oldPV, ok := old.(*v1.PersistentVolume) if !ok { glog.Errorf("cannot convert to *v1.PersistentVolume: %v", old) return } c.invalidatePredicatesForPvUpdate(oldPV, newPV) } } func (c *configFactory) invalidatePredicatesForPvUpdate(oldPV, newPV *v1.PersistentVolume) { invalidPredicates := sets.NewString() for k, v := range newPV.Labels { // If PV update modifies the zone/region labels. if isZoneRegionLabel(k) && !reflect.DeepEqual(v, oldPV.Labels[k]) { invalidPredicates.Insert("NoVolumeZoneConflict") break } } if utilfeature.DefaultFeatureGate.Enabled(features.VolumeScheduling) { oldAffinity, err := v1helper.GetStorageNodeAffinityFromAnnotation(oldPV.Annotations) if err != nil { glog.Errorf("cannot get node affinity fo *v1.PersistentVolume: %v", oldPV) return } newAffinity, err := v1helper.GetStorageNodeAffinityFromAnnotation(newPV.Annotations) if err != nil { glog.Errorf("cannot get node affinity fo *v1.PersistentVolume: %v", newPV) return } // If node affinity of PV is changed. if !reflect.DeepEqual(oldAffinity, newAffinity) { invalidPredicates.Insert(predicates.CheckVolumeBindingPred) } } c.equivalencePodCache.InvalidateCachedPredicateItemOfAllNodes(invalidPredicates) } // isZoneRegionLabel check if given key of label is zone or region label. func isZoneRegionLabel(k string) bool { return k == kubeletapis.LabelZoneFailureDomain || k == kubeletapis.LabelZoneRegion } </ins> func (c *configFactory) onPvDelete(obj interface{}) { if c.enableEquivalenceClassCache { var pv *v1.PersistentVolume

} // If PV contains zone related label, it may impact cached NoVolumeZoneConflict <del> for k := range pv.ObjectMeta.Labels { if k == kubeletapis.LabelZoneFailureDomain || k == kubeletapis.LabelZoneRegion { </del> <ins> for k := range pv.Labels { if isZoneRegionLabel(k) { </ins> invalidPredicates.Insert("NoVolumeZoneConflict") break }

} // The bound volume type may change invalidPredicates.Insert(maxPDVolumeCountPredicateKeys...) <del> // The bound volume's label may change invalidPredicates.Insert("NoVolumeZoneConflict") </del> } c.equivalencePodCache.InvalidateCachedPredicateItemOfAllNodes(invalidPredicates)

invalidPredicates.Insert("MatchInterPodAffinity") } // NoVolumeZoneConflict will only be affected by zone related label change <del> if k == kubeletapis.LabelZoneFailureDomain || k == kubeletapis.LabelZoneRegion { </del> <ins> if isZoneRegionLabel(k) { </ins> if v != newNode.GetLabels()[k] { invalidPredicates.Insert("NoVolumeZoneConflict") }