Repository: cloud-custodian/cloud-custodian

Issue:
Issue with onhours
During **onhours** when instances are being started and there is an error - 

> "errorCode": "Server.InsufficientInstanceCapacity",
>     "errorMessage": "We currently do not have sufficient t3a.large capacity in the Availability Zone you requested (us-east-1d). Our system will be working on provisioning additional capacity. You can currently get t3a.large capacity by not specifying an Availability Zone in your request or choosing us-east-1a, us-east-1b, us-east-1c, us-east-1f."

Instances that are not t3a.large are also not being started. I understand this is not a code issue but a corner case. Because of this we have to turn instances manually and preventing from using offhours.


Discussion:
can you add the whole traceback and identify custodian version.

for start instances action custodian, segments by az, and instance type, and then starts each group in turn. ie. instances that aren't t3a.large in that specific az should not be effected. the last error message is raised after starting all groups to fail the policy execution and draw admin attention, with other errors logged.

Custodian version 0.8.45.3. I will try to reproduce the error, I was able to pull this info from cloud trail while debugging why instances were not started.
do you see other api calls in cloudtrail  (re start instances) for that same policy execution re attempting to start? the custodian log output for that policy execution would be very useful as well.

reviewing the code, its per what I said above (re fine grained grouping by instance type and az for start instances), the error message we raise to stop the policy execution shows all the instance-type / az pairs that couldn't be started which would be in the log.
Per cloudtrail log, there were multiple retries to start the same instance t3a.large and no traces of logs for other instance types. While looking at the resources.json file it has instances other than t3a.large being filtered in.

Below is from the run log

> 2020-01-14 12:06:39,318 - custodian.policy - DEBUG - Running policy:offhours-start resource:ec2 region:us-east-1 c7n:0.8.45.3
> 2020-01-14 12:06:39,365 - custodian.cache - DEBUG - Using cache file /root/.cache/cloud-custodian.cache
> 2020-01-14 12:06:39,365 - custodian.resources.ec2 - DEBUG - Using cached c7n.resources.ec2.EC2: 1185
> 2020-01-14 12:06:39,370 - custodian.filters - INFO - InstanceOnHour 25 of 275 instances
> 2020-01-14 12:06:39,370 - custodian.resources.ec2 - DEBUG - Filtered from 1185 to 2 ec2
> 2020-01-14 12:06:39,371 - custodian.policy - INFO - policy:offhours-start resource:ec2 region:us-east-1 count:2 time:0.05
> 2020-01-14 12:06:39,371 - custodian.actions - INFO - Start 2 of 2 instances

Also @kapilt you are correct both instances are part of same az.. but different instance type. It would be nice if we can skip and move ahead with other instance types even if they are part of the same az in this scenario.
thanks for the log info. we do skip and retry other instance types normally on persistent insufficient capacity. i think i see the issue though. it looks like for t3a and perhaps other arm/graviton types, the error code here is different `Server.InsufficientInstanceCapacity` where as we look for that string without the `Server.` prefix.

Resolving patch:
diff --git a/c7n/resources/ec2.py b/c7n/resources/ec2.py
--- a/c7n/resources/ec2.py
+++ b/c7n/resources/ec2.py
@@ -994,7 +994,7 @@ def process(self, instances):
     def process_instance_set(self, client, instances, itype, izone):
         # Setup retry with insufficient capacity as well
         retryable = ('InsufficientInstanceCapacity', 'RequestLimitExceeded',
-                     'Client.RequestLimitExceeded'),
+                     'Client.RequestLimitExceeded', 'Server.InsufficientInstanceCapacity'),
         retry = utils.get_retry(retryable, max_attempts=5)
         instance_ids = [i['InstanceId'] for i in instances]
         while instance_ids:


Test patch: