Change:
awsTags = append(awsTags, tag) } <ins> backoff := wait.Backoff{ Duration: createTagInitialDelay, Factor:   createTagFactor, Steps:    createTagSteps, } </ins> request := &ec2.CreateTagsInput{} request.Resources = []*string{&resourceID} request.Tags = awsTags <del> // TODO: We really should do exponential backoff here attempt := 0 maxAttempts := 60 for { </del> <ins> var lastErr error err := wait.ExponentialBackoff(backoff, func() (bool, error) { </ins> _, err := c.ec2.CreateTags(request) if err == nil { <del> return nil </del> <ins> return true, nil </ins> } // We could check that the error is retryable, but the error code changes based on what we are tagging // SecurityGroup: InvalidGroup.NotFound <del> attempt++ if attempt > maxAttempts { glog.Warningf("Failed to create tags (too many attempts): %v", err) return err } </del> glog.V(2).Infof("Failed to create tags; will retry.  Error was %v", err) <del> time.Sleep(1 * time.Second) </del> <ins> lastErr = err return false, nil }) if err == wait.ErrWaitTimeout { // return real CreateTags error instead of timeout err = lastErr </ins> } <ins> return err </ins> } // Finds the value for a given tag.

Discussion:
In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.