Repository: cloud-custodian/cloud-custodian

Issue:
deleting API Gateway Rest APIs: getting TooManyRequestsException when calling the DeleteRestApi operation
### Describe the bug

Hi, I'm trying to use CloudCustodian to delete API Gateway APIs that are not tagged properly. The problem is that there are 63 of them so I'm running into TooManyRequest errors. Has anyone found a way around this? TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the DeleteRestApi operation (reached max retries: 4): Too Many Requests. Is there a way around this?

### What did you expect to happen?

I expected the rest APIs to get deleted.

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

```shell
Custodian:   0.9.14
Python:      3.9.9 (main, Nov 21 2021, 03:23:42)
             [Clang 13.0.0 (clang-1300.0.29.3)]
Platform:    posix.uname_result(sysname='Darwin', nodename='MacBook-Pro', release='20.6.0', version='Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==1.12.3
attrs==21.2.0
boto3==1.19.12
botocore==1.22.12
docutils==0.17.1
importlib-metadata==4.8.1
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.2
pyyaml==5.4.1
s3transfer==0.5.0
setuptools==59.0.1
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.2
urllib3==1.26.7
zipp==3.6.0
```


### Policy

```shell
- name: api-gateway-untagged-notifier
    resource: rest-api
    description: |
      Cloud Custodian API Gateway Cleanup Notification
    comments: |
      Notify pending termination of API Gateway Rest APIs that are not tagged with retain
    filters:
      - not:
           - or:
               - tag:PAN: "true"
               - tag:retain: "true"
               - tag:LightsOut: "exclude"
               - tag:environment: "smoke"
               - tag:environment: "qc"
               - tag:gitrepo: "version-tracker-site"
               - tag:gitrepo: "tf-shared-infrastructure"
    mode:
      type: periodic
      schedule: "rate(14 days)"
      packages: [boto3, botocore, urllib3]
      role: CloudCustodian
    actions:
      - type: notify
        priority_header: '1'
        subject: '!!! Out of Compliance: API Gateway Rest APIs scheduled for termination on Friday!!!'
        violation_desc:
        action_desc: |
            The following resources do not have retain=true tags, so will be deleted on Friday if they're not tagged before then. Please review and make sure any resources you will still need are tagged with retain=true
        to:
          - badams@example.com
          - ResourceTERM@example.com
        transport:
          type: sqs
          queue: https://sqs.eu-west-2.amazonaws.com/11222334455/CloudCustodian
```


### Relevant log/traceback output

```shell
[DEBUG]	2022-02-05T02:53:45.086Z	05d299ef-5a5f-432e-b600-9a31098a701c	Disabling cache
[WARNING]	2022-02-05T02:53:45.087Z	05d299ef-5a5f-432e-b600-9a31098a701c	Custodian reserves policy lambda tags starting with custodian - policy specifies custodian-info
[DEBUG]	2022-02-05T02:53:45.087Z	05d299ef-5a5f-432e-b600-9a31098a701c	Storing output with <LogFile file:///tmp/api-gateway-delete-untagged/custodian-run.log>
[DEBUG]	2022-02-05T02:53:45.109Z	05d299ef-5a5f-432e-b600-9a31098a701c	Running policy:api-gateway-delete-untagged resource:rest-api region:eu-west-2 c7n:0.9.13
[DEBUG]	2022-02-05T02:53:45.782Z	05d299ef-5a5f-432e-b600-9a31098a701c	Filtered from 67 to 44 restapi
[INFO]	2022-02-05T02:53:45.783Z	05d299ef-5a5f-432e-b600-9a31098a701c	policy:api-gateway-delete-untagged resource:rest-api region:eu-west-2 count:44 time:0.66
[DEBUG]	2022-02-05T02:53:52.166Z	05d299ef-5a5f-432e-b600-9a31098a701c	metric:ResourceCount Count:44 policy:api-gateway-delete-untagged restype:rest-api scope:policy
[DEBUG]	2022-02-05T02:53:52.166Z	05d299ef-5a5f-432e-b600-9a31098a701c	metric:PolicyException Count:1 policy:api-gateway-delete-untagged restype:rest-api
[DEBUG]	2022-02-05T02:53:52.166Z	05d299ef-5a5f-432e-b600-9a31098a701c	metric:ApiCalls Count:6 policy:api-gateway-delete-untagged restype:rest-api
[ERROR]	2022-02-05T02:53:52.166Z	05d299ef-5a5f-432e-b600-9a31098a701c	Error while executing policy
Traceback (most recent call last):
  File "/var/task/c7n/policy.py", line 322, in run
    results = a.process(resources)
  File "/var/task/c7n/resources/apigw.py", line 244, in process
    client.delete_rest_api(restApiId=r['id'])
  File "/var/task/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the DeleteRestApi operation (reached max retries: 4): Too Many Requests
[ERROR]	2022-02-05T02:53:52.168Z	05d299ef-5a5f-432e-b600-9a31098a701c	error during policy execution
Traceback (most recent call last):
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1164, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 531, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 322, in run
    results = a.process(resources)
  File "/var/task/c7n/resources/apigw.py", line 244, in process
    client.delete_rest_api(restApiId=r['id'])
  File "/var/task/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the DeleteRestApi operation (reached max retries: 4): Too Many Requests
[ERROR] TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the DeleteRestApi operation (reached max retries: 4): Too Many Requests
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1164, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 531, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 322, in run
    results = a.process(resources)
  File "/var/task/c7n/resources/apigw.py", line 244, in process
    client.delete_rest_api(restApiId=r['id'])
  File "/var/task/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
END RequestId: 05d299ef-5a5f-432e-b600-9a31098a701c
REPORT RequestId: 05d299ef-5a5f-432e-b600-9a31098a701c	Duration: 7781.61 ms	Billed Duration: 7782 ms	Memory Size: 512 MB	Max Memory Used: 76 MB	Init Duration: 699.16 ms
```


### Extra information or context

_No response_


Discussion:

Resolving patch:
diff --git a/c7n/resources/apigw.py b/c7n/resources/apigw.py
--- a/c7n/resources/apigw.py
+++ b/c7n/resources/apigw.py
@@ -13,7 +13,7 @@
 from c7n.filters.related import RelatedResourceFilter
 from c7n.manager import resources, ResourceManager
 from c7n import query, utils
-from c7n.utils import generate_arn, type_schema
+from c7n.utils import generate_arn, type_schema, get_retry
 
 
 ANNOTATION_KEY_MATCHED_METHODS = 'c7n:matched-resource-methods'
@@ -239,9 +239,11 @@ class DeleteApi(BaseAction):
     def process(self, resources):
         client = utils.local_session(
             self.manager.session_factory).client('apigateway')
+        retry = get_retry(('TooManyRequestsException',))
+
         for r in resources:
             try:
-                client.delete_rest_api(restApiId=r['id'])
+                retry(client.delete_rest_api, restApiId=r['id'])
             except client.exceptions.NotFoundException:
                 continue
 


Test patch: