Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", </del> "//staging/src/k8s.io/cloud-provider/volume:go_default_library", "//vendor/github.com/aws/aws-sdk-go/aws:go_default_library", "//vendor/github.com/aws/aws-sdk-go/service/ec2:go_default_library",

const nodeWithImpairedVolumes = "NodeWithImpairedVolumes" const ( <del> // These constants help to identify if a node is a master or a minion labelKeyNodeRole    = "kubernetes.io/role" nodeMasterRole      = "master" nodeMinionRole      = "node" labelKeyNodeMaster  = "node-role.kubernetes.io/master" labelKeyNodeCompute = "node-role.kubernetes.io/compute" labelKeyNodeMinion  = "node-role.kubernetes.io/node" ) const ( </del> // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10 // most attach/detach operations on AWS finish within 1-4 seconds

// GetCandidateZonesForDynamicVolume retrieves  a list of all the zones in which nodes are running // It currently involves querying all instances func (c *Cloud) GetCandidateZonesForDynamicVolume() (sets.String, error) { <del> zones := sets.NewString() </del> <ins> // We don't currently cache this; it is currently used only in volume // creation which is expected to be a comparatively rare occurrence. // TODO: Caching / expose v1.Nodes to the cloud provider? // TODO: We could also query for subnets, I think // Note: It is more efficient to call the EC2 API twice with different tag // filters than to call it once with a tag filter that results in a logical // OR. For really large clusters the logical OR will result in EC2 API rate // limiting. instances := []*ec2.Instance{} </ins> <del> // TODO: list from cache? nodes, err := c.kubeClient.CoreV1().Nodes().List(metav1.ListOptions{}) </del> <ins> baseFilters := []*ec2.Filter{newEc2Filter("instance-state-name", "running")} filters := c.tagging.addFilters(baseFilters) di, err := c.describeInstances(filters) </ins> if err != nil { <del> klog.Errorf("Failed to get nodes from api server: %#v", err) </del> return nil, err } <del> for _, n := range nodes.Items { if !c.isNodeReady(&n) { klog.V(4).Infof("Ignoring not ready node %q in zone discovery", n.Name) continue } // In some cluster provisioning software, a node can be both a minion and a master. Therefore we white-list // here, and only filter out node that is not minion AND is labeled as master explicitly if c.isMinionNode(&n) || !c.isMasterNode(&n) { if zone, ok := n.Labels[v1.LabelZoneFailureDomain]; ok { zones.Insert(zone) } else { klog.Warningf("Node %s does not have zone label, ignore for zone discovery.", n.Name) } } else { klog.V(4).Infof("Ignoring master node %q in zone discovery", n.Name) </del> <ins> instances = append(instances, di...) if c.tagging.usesLegacyTags { filters = c.tagging.addLegacyFilters(baseFilters) di, err = c.describeInstances(filters) if err != nil { return nil, err </ins> } <ins> instances = append(instances, di...) </ins> } <del> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil } </del> <ins> if len(instances) == 0 { return nil, fmt.Errorf("no instances returned") } zones := sets.NewString() </ins> <del> // isNodeReady checks node condition and return true if NodeReady is marked as true func (c *Cloud) isNodeReady(node *v1.Node) bool { for _, c := range node.Status.Conditions { if c.Type == v1.NodeReady { return c.Status == v1.ConditionTrue </del> <ins> for _, instance := range instances { // We skip over master nodes, if the installation tool labels them with one of the well-known master labels // This avoids creating a volume in a zone where only the master is running - e.g. #34583 // This is a short-term workaround until the scheduler takes care of zone selection master := false for _, tag := range instance.Tags { tagKey := aws.StringValue(tag.Key) if awsTagNameMasterRoles.Has(tagKey) { master = true } </ins> } <del> } return false } // isMasterNode checks if the node is labeled as master func (c *Cloud) isMasterNode(node *v1.Node) bool { // Master node has one or more of the following labels: // // 	kubernetes.io/role: master //	node-role.kubernetes.io/master: "" //	node-role.kubernetes.io/master: "true" if val, ok := node.Labels[labelKeyNodeMaster]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMasterRole { return true } return false } </del> <ins> if master { klog.V(4).Infof("Ignoring master instance %q in zone discovery", aws.StringValue(instance.InstanceId)) continue } </ins> <del> // isMinionNode checks if the node is labeled as minion func (c *Cloud) isMinionNode(node *v1.Node) bool { // Minion node has one or more oof the following labels: // // 	kubernetes.io/role: "node" //	node-role.kubernetes.io/compute: "true" //	node-role.kubernetes.io/node: "" if val, ok := node.Labels[labelKeyNodeMinion]; ok && val != "false" { return true } else if val, ok := node.Labels[labelKeyNodeCompute]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMinionRole { return true </del> <ins> if instance.Placement != nil { zone := aws.StringValue(instance.Placement.AvailabilityZone) zones.Insert(zone) } </ins> } <del> return false </del> <ins> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil </ins> } // GetZone implements Zones.GetZone

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/client-go/kubernetes/fake" </del> cloudvolume "k8s.io/cloud-provider/volume" )

assert.False(t, isRegionValid("pl-fake-991a", fake.metadata), "expected region 'pl-fake-991' to be invalid but it was not") } <del> func TestGetCandidateZonesForDynamicVolume(t *testing.T) { tests := []struct { name          string labels        map[string]string ready         bool expectedZones sets.String }{ { name:          "master node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMasterRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label empty", labels:        map[string]string{labelKeyNodeMaster: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label true", labels:        map[string]string{labelKeyNodeMaster: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label false", labels:        map[string]string{labelKeyNodeMaster: "false", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMinionRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with minion label", labels:        map[string]string{labelKeyNodeMinion: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with compute label", labels:        map[string]string{labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "master and minion node", labels:        map[string]string{labelKeyNodeMaster: "true", labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "node not ready", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         false, expectedZones: sets.NewString(), }, { name:          "node has no zone", labels:        map[string]string{}, ready:         true, expectedZones: sets.NewString(), }, { name:          "node with no label", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, } for i, test := range tests { t.Run(test.name, func(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterID) c, _ := newAWSCloud(CloudConfig{}, awsServices) c.kubeClient = fake.NewSimpleClientset() nodeName := fmt.Sprintf("node-%d", i) _, err := c.kubeClient.CoreV1().Nodes().Create(&v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:   nodeName, Labels: test.labels, }, Status: genNodeStatus(test.ready), }) assert.Nil(t, err) zones, err := c.GetCandidateZonesForDynamicVolume() assert.Nil(t, err) assert.Equal(t, test.expectedZones, zones) }) } } func genNodeStatus(ready bool) v1.NodeStatus { status := v1.ConditionFalse if ready { status = v1.ConditionTrue } ret := v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady, Status: status, }, }, } return ret } </del> func newMockedFakeAWSServices(id string) *FakeAWSServices { s := NewFakeAWSServices(id) s.ec2 = &MockedFakeEC2{FakeEC2Impl: s.ec2.(*FakeEC2Impl)}