Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Volume binding mode doesn't consider the volume attach limit per node. What you expected to happen: volume binding mode should have respected the attach limit, similar to binding mode. How to reproduce it (as minimally and precisely as possible): In CSI driver, configured the volume-attach limit to 1 per node on a 3 node cluster. In volume binding mode, only 3 pods with 3 PV/PVCs runs. In binding mode, all 5 pods runs with 5PV/PVCs on 3 node cluster. (Actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled) Anything else we need to know?: I was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 Volume limit and 500 pods running on the 5 mode cluster. Environment: Kubernetes version (use ): ~ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On-prem linux server OS (e.g. from /etc/os-release): Centos 7Kernel (e.g. ): 3.10.0 Install tools: Others:
cc
/sig storage
/assign Currently the predicate PVCs that are not bound. It should lookup the storageclass to get the provisioner/driver name. This solution will require a StorageClass object is even if you don't use dynamic provisioning.
I had a thought that it wouldn't be so simple to count unbound PVCs of other pods since binding is still in progress, but I think it's fine actually. We only count other pods that have been assigned to the same node. Those pods are assigned to the node AND still unbound because of one of these reasons: It's actually been scheduled to the node and someone manually came in and unbound the PVC: unlikely and an unsupported sequence since we implemented StorageProtection The Pod was initially created with nodeName already set, bypassing the scheduler The Pod is going to be scheduled (it's been assumed), and it's currently going through binding. This is normal with delayed binding. Either way, the fact that the Pod's nodename is set to this node means that the pod should be counting towards the node's limit.
Hmm, so after opening the PR that fixes this, I realize that - from storageclass we don't really know if the volume we are evaluating is CSI volume or not. I think a similar problem exists for existing predicate though. Like AzureDisk predicate does not know if PVC it is over counting is of type AzureVolume or not. So, I suspect that those 4 hardcoded predicates are counting all delayed binding volumes.
if the storageclass.provisioner name is not found, can we just ignore it?
I think we can ignore it. The PR currently does that.

Discussion:
"fmt" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/klog" "k8s.io/kubernetes/pkg/features"

// CSIMaxVolumeLimitChecker defines predicate needed for counting CSI volumes type CSIMaxVolumeLimitChecker struct { <del> pvInfo  PersistentVolumeInfo pvcInfo PersistentVolumeClaimInfo </del> <ins> pvInfo               PersistentVolumeInfo pvcInfo              PersistentVolumeClaimInfo scInfo               StorageClassInfo randomVolumeIDPrefix string </ins> } // NewCSIMaxVolumeLimitPredicate returns a predicate for counting CSI volumes func NewCSIMaxVolumeLimitPredicate( <del> pvInfo PersistentVolumeInfo, pvcInfo PersistentVolumeClaimInfo) FitPredicate { </del> <ins> pvInfo PersistentVolumeInfo, pvcInfo PersistentVolumeClaimInfo, scInfo StorageClassInfo) FitPredicate { </ins> c := &CSIMaxVolumeLimitChecker{ <del> pvInfo:  pvInfo, pvcInfo: pvcInfo, </del> <ins> pvInfo:               pvInfo, pvcInfo:              pvcInfo, scInfo:               scInfo, randomVolumeIDPrefix: rand.String(32), </ins> } return c.attachableLimitPredicate }

continue } <del> pvName := pvc.Spec.VolumeName // TODO - the actual handling of unbound PVCs will be fixed by late binding design. if pvName == "" { klog.V(4).Infof("Persistent volume had no name for claim %s/%s", namespace, pvcName) </del> <ins> driverName, volumeHandle := c.getCSIDriver(pvc) // if we can't find driver name or volume handle - we don't count this volume. if driverName == "" || volumeHandle == "" { </ins> continue } <del> pv, err := c.pvInfo.GetPersistentVolumeInfo(pvName) if err != nil { klog.V(4).Infof("Unable to look up PV info for PVC %s/%s and PV %s", namespace, pvcName, pvName) continue } csiSource := pv.Spec.PersistentVolumeSource.CSI if csiSource == nil { klog.V(4).Infof("Not considering non-CSI volume %s/%s", namespace, pvcName) continue } driverName := csiSource.Driver </del> volumeLimitKey := volumeutil.GetCSIAttachLimitKey(driverName) <del> result[csiSource.VolumeHandle] = volumeLimitKey </del> <ins> result[volumeHandle] = volumeLimitKey </ins> } return nil } <ins> func (c *CSIMaxVolumeLimitChecker) getCSIDriver(pvc *v1.PersistentVolumeClaim) (string, string) { pvName := pvc.Spec.VolumeName namespace := pvc.Namespace pvcName := pvc.Name placeHolderCSIDriver := "" placeHolderHandle := "" if pvName == "" { klog.V(5).Infof("Persistent volume had no name for claim %s/%s", namespace, pvcName) return c.getDriverNameFromSC(pvc) } pv, err := c.pvInfo.GetPersistentVolumeInfo(pvName) if err != nil { klog.V(4).Infof("Unable to look up PV info for PVC %s/%s and PV %s", namespace, pvcName, pvName) // If we can't fetch PV associated with PVC, may be it got deleted // or PVC was prebound to a PVC that hasn't been created yet. // fallback to using StorageClass for volume counting return c.getDriverNameFromSC(pvc) } csiSource := pv.Spec.PersistentVolumeSource.CSI if csiSource == nil { klog.V(5).Infof("Not considering non-CSI volume %s/%s", namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } return csiSource.Driver, csiSource.VolumeHandle } func (c *CSIMaxVolumeLimitChecker) getDriverNameFromSC(pvc *v1.PersistentVolumeClaim) (string, string) { namespace := pvc.Namespace pvcName := pvc.Name scName := pvc.Spec.StorageClassName placeHolderCSIDriver := "" placeHolderHandle := "" if scName == nil { // if StorageClass is not set or found, then PVC must be using immediate binding mode // and hence it must be bound before scheduling. So it is safe to not count it. klog.V(5).Infof("pvc %s/%s has no storageClass", namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } storageClass, err := c.scInfo.GetStorageClassInfo(*scName) if err != nil { klog.V(5).Infof("no storage %s found for pvc %s/%s", *scName, namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } // We use random prefix to avoid conflict with volume-ids. If PVC is bound in the middle // predicate and there is another pod(on same node) that uses same volume then we will overcount // the volume and consider both volumes as different. volumeHandle := fmt.Sprintf("%s-%s/%s", c.randomVolumeIDPrefix, namespace, pvcName) return storageClass.Provisioner, volumeHandle } </ins>

package predicates import ( <ins> "fmt" </ins> "reflect" "testing"

{ VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ <del> ClaimName: "csi-ebs", </del> <ins> ClaimName: "csi-ebs-0", </ins> }, }, },

}, } <ins> pendingVolumePod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-4", }, }, }, }, }, } // Different pod than pendingVolumePod, but using the same unbound PVC unboundPVCPod2 := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-4", }, }, }, }, }, } missingPVPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-6", }, }, }, }, }, } noSCPVCPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-5", }, }, }, }, }, } gceTwoVolPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "cs-gce-1", }, }, }, { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-gce-2", }, }, }, }, }, } </ins> tests := []struct { newPod       *v1.Pod existingPods []*v1.Pod filterName   string maxVols      int <ins> driverNames  []string </ins> fits         bool test         string }{ { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, twoVolPod}, <del> filterName:   "csi-ebs", </del> <ins> filterName:   "csi", </ins> maxVols:      4, <ins> driverNames:  []string{"ebs"}, </ins> fits:         true, test:         "fits when node capacity >= new pods CSI volume", }, { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, twoVolPod}, <del> filterName:   "csi-ebs", </del> <ins> filterName:   "csi", </ins> maxVols:      2, <ins> driverNames:  []string{"ebs"}, </ins> fits:         false, test:         "doesn't when node capacity <= pods CSI volume", }, <ins> // should count pending PVCs { newPod:       oneVolPod, existingPods: []*v1.Pod{pendingVolumePod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         false, test:         "count pending PVCs towards capacity <= pods CSI volume", }, // two same pending PVCs should be counted as 1 { newPod:       oneVolPod, existingPods: []*v1.Pod{pendingVolumePod, unboundPVCPod2, twoVolPod}, filterName:   "csi", maxVols:      3, driverNames:  []string{"ebs"}, fits:         true, test:         "count multiple pending pvcs towards capacity >= pods CSI volume", }, // should count PVCs with invalid PV name but valid SC { newPod:       oneVolPod, existingPods: []*v1.Pod{missingPVPod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         false, test:         "should count PVCs with invalid PV name but valid SC", }, // don't count a volume which has storageclass missing { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, noSCPVCPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         true, test:         "don't count pvcs with missing SC towards capacity", }, // don't count multiple volume types { newPod:       oneVolPod, existingPods: []*v1.Pod{gceTwoVolPod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs", "gce"}, fits:         true, test:         "don't count pvcs with different type towards capacity", }, { newPod:       gceTwoVolPod, existingPods: []*v1.Pod{twoVolPod, runningPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs", "gce"}, fits:         true, test:         "don't count pvcs with different type towards capacity", }, </ins> } defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.AttachVolumeLimit, true)() expectedFailureReasons := []PredicateFailureReason{ErrMaxVolumeCountExceeded} // running attachable predicate tests with feature gate and limit present on nodes for _, test := range tests { <del> node := getNodeWithPodAndVolumeLimits(test.existingPods, int64(test.maxVols), test.filterName) pred := NewCSIMaxVolumeLimitPredicate(getFakeCSIPVInfo("csi-ebs", "csi-ebs"), getFakeCSIPVCInfo("csi-ebs")) </del> <ins> node := getNodeWithPodAndVolumeLimits(test.existingPods, int64(test.maxVols), test.driverNames...) pred := NewCSIMaxVolumeLimitPredicate(getFakeCSIPVInfo(test.filterName, test.driverNames...), getFakeCSIPVCInfo(test.filterName, "csi-sc", test.driverNames...), getFakeCSIStorageClassInfo("csi-sc", test.driverNames[0])) </ins> fits, reasons, err := pred(test.newPod, GetPredicateMetadata(test.newPod, nil), node) if err != nil { t.Errorf("Using allocatable [%s]%s: unexpected error: %v", test.filterName, test.test, err)

} } <del> func getFakeCSIPVInfo(volumeName, driverName string) FakePersistentVolumeInfo { return FakePersistentVolumeInfo{ { ObjectMeta: metav1.ObjectMeta{Name: volumeName}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       driverName, VolumeHandle: volumeName, }, }, }, }, { ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-2"}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       driverName, VolumeHandle: volumeName + "-2", }, }, }, }, { ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-3"}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       driverName, VolumeHandle: volumeName + "-3", </del> <ins> func getFakeCSIPVInfo(volumeName string, driverNames ...string) FakePersistentVolumeInfo { pvInfos := FakePersistentVolumeInfo{} for _, driver := range driverNames { for j := 0; j < 4; j++ { volumeHandle := fmt.Sprintf("%s-%s-%d", volumeName, driver, j) pv := v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: volumeHandle}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       driver, VolumeHandle: volumeHandle, }, </ins> }, }, <del> }, }, </del> <ins> } pvInfos = append(pvInfos, pv) } </ins> } <ins> return pvInfos </ins> } <del> func getFakeCSIPVCInfo(volumeName string) FakePersistentVolumeClaimInfo { return FakePersistentVolumeClaimInfo{ { ObjectMeta: metav1.ObjectMeta{Name: volumeName}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: volumeName}, }, { ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-2"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: volumeName + "-2"}, }, </del> <ins> func getFakeCSIPVCInfo(volumeName, scName string, driverNames ...string) FakePersistentVolumeClaimInfo { pvcInfos := FakePersistentVolumeClaimInfo{} for _, driver := range driverNames { for j := 0; j < 4; j++ { v := fmt.Sprintf("%s-%s-%d", volumeName, driver, j) pvc := v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{Name: v}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: v}, } pvcInfos = append(pvcInfos, pvc) } } pvcInfos = append(pvcInfos, v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-4"}, Spec:       v1.PersistentVolumeClaimSpec{StorageClassName: &scName}, }) pvcInfos = append(pvcInfos, v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-5"}, Spec:       v1.PersistentVolumeClaimSpec{}, }) // a pvc with missing PV but available storageclass. pvcInfos = append(pvcInfos, v1.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-6"}, Spec:       v1.PersistentVolumeClaimSpec{StorageClassName: &scName, VolumeName: "missing-in-action"}, }) return pvcInfos } func getFakeCSIStorageClassInfo(scName, provisionerName string) FakeStorageClassInfo { return FakeStorageClassInfo{ </ins> { <del> ObjectMeta: metav1.ObjectMeta{Name: volumeName + "-3"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: volumeName + "-3"}, </del> <ins> ObjectMeta:  metav1.ObjectMeta{Name: scName}, Provisioner: provisionerName, </ins> }, } }

} } <del> func getNodeWithPodAndVolumeLimits(pods []*v1.Pod, limit int64, filter string) *schedulernodeinfo.NodeInfo { </del> <ins> func getNodeWithPodAndVolumeLimits(pods []*v1.Pod, limit int64, driverNames ...string) *schedulernodeinfo.NodeInfo { </ins> nodeInfo := schedulernodeinfo.NewNodeInfo(pods...) node := &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "node-for-max-pd-test-1"}, Status: v1.NodeStatus{ <del> Allocatable: v1.ResourceList{ getVolumeLimitKey(filter): *resource.NewQuantity(limit, resource.DecimalSI), }, </del> <ins> Allocatable: v1.ResourceList{}, </ins> }, } <ins> for _, driver := range driverNames { node.Status.Allocatable[getVolumeLimitKey(driver)] = *resource.NewQuantity(limit, resource.DecimalSI) } </ins> nodeInfo.SetNode(node) return nodeInfo }

factory.RegisterFitPredicateFactory( predicates.MaxCSIVolumeCountPred, func(args factory.PluginFactoryArgs) predicates.FitPredicate { <del> return predicates.NewCSIMaxVolumeLimitPredicate(args.PVInfo, args.PVCInfo) </del> <ins> return predicates.NewCSIMaxVolumeLimitPredicate(args.PVInfo, args.PVCInfo, args.StorageClassInfo) </ins> }, ) factory.RegisterFitPredicateFactory(