Change:
In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt. calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it. PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.

Discussion:
"//pkg/credentialprovider/aws:go_default_library", "//pkg/types:go_default_library", "//pkg/util/sets:go_default_library", <ins> "//pkg/util/wait:go_default_library", </ins> "//pkg/volume:go_default_library", "//vendor:github.com/aws/aws-sdk-go/aws", "//vendor:github.com/aws/aws-sdk-go/aws/awserr",

awscredentials "k8s.io/kubernetes/pkg/credentialprovider/aws" "k8s.io/kubernetes/pkg/types" "k8s.io/kubernetes/pkg/util/sets" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/volume" )

const ServiceAnnotationLoadBalancerBEProtocol = "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" const ( <del> // volumeAttachmentStatusTimeout is the maximum time to wait for a volume attach/detach to complete volumeAttachmentStatusTimeout = 30 * time.Minute </del> // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10 <del> // volumeAttachmentErrorDelay is the amount of time we wait before retrying after encountering an error, // while waiting for a volume attach/detach to complete volumeAttachmentStatusErrorDelay = 20 * time.Second // volumeAttachmentStatusPollInterval is the interval at which we poll the volume, // while waiting for a volume attach/detach to complete volumeAttachmentStatusPollInterval = 10 * time.Second </del> <ins> // volumeAttachmentStatus* is configuration of exponential backoff for // waiting for attach/detach operation to complete. Starting with 10 // seconds, multiplying by 1.2 with each step and taking 21 steps at maximum // it will time out after 31.11 minutes, which roughly corresponds to GCE // timeout (30 minutes). volumeAttachmentStatusInitialDelay = 10 * time.Second volumeAttachmentStatusFactor       = 1.2 volumeAttachmentStatusSteps        = 21 // createTag* is configuration of exponential backoff for CreateTag call. We // retry mainly because if we create an object, we cannot tag it until it is // "fully created" (eventual consistency). Starting with 1 second, doubling // it every step and taking 9 steps results in 255 second total waiting // time. createTagInitialDelay = 1 * time.Second createTagFactor       = 2.0 createTagSteps        = 9 </ins> ) // Maps from backend protocol to ELB protocol

// waitForAttachmentStatus polls until the attachment status is the expected value // On success, it returns the last attachment state. func (d *awsDisk) waitForAttachmentStatus(status string) (*ec2.VolumeAttachment, error) { <del> // We wait up to 30 minutes for the attachment to complete. // This mirrors the GCE timeout. timeoutAt := time.Now().UTC().Add(volumeAttachmentStatusTimeout).Unix() </del> <ins> backoff := wait.Backoff{ Duration: volumeAttachmentStatusInitialDelay, Factor:   volumeAttachmentStatusFactor, Steps:    volumeAttachmentStatusSteps, } </ins> // Because of rate limiting, we often see errors from describeVolume // So we tolerate a limited number of failures. // But once we see more than 10 errors in a row, we return the error describeErrorCount := 0 <ins> var attachment *ec2.VolumeAttachment </ins> <del> for { </del> <ins> err := wait.ExponentialBackoff(backoff, func() (bool, error) { </ins> info, err := d.describeVolume() if err != nil { describeErrorCount++ if describeErrorCount > volumeAttachmentStatusConsecutiveErrorLimit { <del> return nil, err </del> <ins> // report the error return false, err </ins> } else { glog.Warningf("Ignoring error from describe volume; will retry: %q", err) <del> time.Sleep(volumeAttachmentStatusErrorDelay) continue </del> <ins> return false, nil </ins> } } else { describeErrorCount = 0

// Shouldn't happen; log so we know if it is glog.Warningf("Found multiple attachments for volume %q: %v", d.awsID, info) } <del> var attachment *ec2.VolumeAttachment </del> attachmentStatus := "" for _, a := range info.Attachments { if attachmentStatus != "" {

attachmentStatus = "detached" } if attachmentStatus == status { <del> return attachment, nil } if time.Now().Unix() > timeoutAt { glog.Warningf("Timeout waiting for volume %q state: actual=%s, desired=%s", d.awsID, attachmentStatus, status) return nil, fmt.Errorf("Timeout waiting for volume %q state: actual=%s, desired=%s", d.awsID, attachmentStatus, status) </del> <ins> // Attachment is in requested state, finish waiting return true, nil </ins> } <ins> // continue waiting </ins> glog.V(2).Infof("Waiting for volume %q state: actual=%s, desired=%s", d.awsID, attachmentStatus, status) <ins> return false, nil }) </ins> <del> time.Sleep(volumeAttachmentStatusPollInterval) } </del> <ins> return attachment, err </ins> } // Deletes the EBS disk

awsTags = append(awsTags, tag) } <ins> backoff := wait.Backoff{ Duration: createTagInitialDelay, Factor:   createTagFactor, Steps:    createTagSteps, } </ins> request := &ec2.CreateTagsInput{} request.Resources = []*string{&resourceID} request.Tags = awsTags <del> // TODO: We really should do exponential backoff here attempt := 0 maxAttempts := 60 for { </del> <ins> var lastErr error err := wait.ExponentialBackoff(backoff, func() (bool, error) { </ins> _, err := c.ec2.CreateTags(request) if err == nil { <del> return nil </del> <ins> return true, nil </ins> } // We could check that the error is retryable, but the error code changes based on what we are tagging // SecurityGroup: InvalidGroup.NotFound <del> attempt++ if attempt > maxAttempts { glog.Warningf("Failed to create tags (too many attempts): %v", err) return err } </del> glog.V(2).Infof("Failed to create tags; will retry.  Error was %v", err) <del> time.Sleep(1 * time.Second) </del> <ins> lastErr = err return false, nil }) if err == wait.ErrWaitTimeout { // return real CreateTags error instead of timeout err = lastErr </ins> } <ins> return err </ins> } // Finds the value for a given tag.