Repository: cloud-custodian/cloud-custodian

Issue:
Anomaly with AssumeRole calls at runtime
### Describe the bug

We have been seeing a ton of **AssumeRole** events/calls made by CloudCustodian on an ongoing policy execution job. They are of the order of ~ 50 times a minute and sometimes multiple times a second. I did some investigation by printing the access_key_id and expiry time of the creds [this](https://github.com/cloud-custodian/cloud-custodian/blob/main/c7n/credentials.py#L96) part of the code.

Command used for execution: `custodian run -c test.yml -s output --region us-east-1 --cache-period=25 --assume arn:aws:iam::644160558196:role/CloudCustodianRole -vvv`

Findings:

1. With a policy file that contained batch policy only. 
```
2023-03-31 00:10:48,553: custodian.commands:DEBUG Loaded file test.yml. Contains 4 policies
2023-03-31 00:10:48,909: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6***R2D6DUTL4B
2023-03-31T05:10:49+00:00
2023-03-31 00:10:49,346: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-1/custodian-run.log>
2023-03-31 00:10:49,526: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6X***2FPLI3ZNU
2023-03-31T05:10:49+00:00
2023-03-31 00:10:49,940: custodian.policy:DEBUG Running policy:test-policy-1 resource:aws.kms-key region:us-east-1 c7n:0.9.24
2023-03-31 00:10:49,942: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:10:49,943: custodian.resources.key:DEBUG Using cached c7n.resources.kms.Key: 49
2023-03-31 00:10:50,047: custodian.filters:DEBUG Querying 49 kms-keys' rotation status
2023-03-31 00:10:50,217: custodian.filters:WARNING Access denied when getting rotation status on key:arn:aws:kms:us-east-1:644160558196:key/7efe5f36-5599-4a81-97b9-4c4aa4b64f93
2023-03-31 00:10:50,766: custodian.resources.key:DEBUG Filtered from 49 to 17 key
2023-03-31 00:10:50,766: custodian.policy:INFO policy:test-policy-1 resource:aws.kms-key region:us-east-1 count:17 time:0.83
2023-03-31 00:10:50,770: custodian.output:DEBUG metric:ResourceCount Count:17 policy:test-policy-1 restype:aws.kms-key scope:policy
2023-03-31 00:10:50,770: custodian.output:DEBUG metric:ApiCalls Count:49 policy:test-policy-1 restype:aws.kms-key
2023-03-31 00:10:50,770: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-2/custodian-run.log>
2023-03-31 00:10:50,771: custodian.policy:DEBUG Running policy:test-policy-2 resource:aws.kms-key region:us-east-1 c7n:0.9.24
2023-03-31 00:10:50,772: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:10:50,773: custodian.resources.key:DEBUG Using cached c7n.resources.kms.Key: 49
2023-03-31 00:10:50,778: custodian.filters:DEBUG Querying 49 kms-keys' rotation status
2023-03-31 00:10:50,940: custodian.filters:WARNING Access denied when getting rotation status on key:arn:aws:kms:us-east-1:644160558196:key/7efe5f36-5599-4a81-97b9-4c4aa4b64f93
2023-03-31 00:10:51,505: custodian.resources.key:DEBUG Filtered from 49 to 32 key
2023-03-31 00:10:51,506: custodian.policy:INFO policy:test-policy-2 resource:aws.kms-key region:us-east-1 count:32 time:0.73
2023-03-31 00:10:51,510: custodian.output:DEBUG metric:ResourceCount Count:32 policy:test-policy-2 restype:aws.kms-key scope:policy
2023-03-31 00:10:51,510: custodian.output:DEBUG metric:ApiCalls Count:49 policy:test-policy-2 restype:aws.kms-key
2023-03-31 00:10:51,511: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-3/custodian-run.log>
2023-03-31 00:10:51,512: custodian.policy:DEBUG Running policy:test-policy-3 resource:aws.sns region:us-east-1 c7n:0.9.24
2023-03-31 00:10:51,512: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:10:51,513: custodian.resources.sns:DEBUG Using cached c7n.resources.sns.SNS: 8
2023-03-31 00:10:51,513: custodian.resources.sns:DEBUG Filtered from 8 to 7 sns
2023-03-31 00:10:51,513: custodian.policy:INFO policy:test-policy-3 resource:aws.sns region:us-east-1 count:7 time:0.00
2023-03-31 00:10:51,515: custodian.output:DEBUG metric:ResourceCount Count:7 policy:test-policy-3 restype:aws.sns scope:policy
2023-03-31 00:10:51,515: custodian.output:DEBUG metric:ApiCalls Count:0 policy:test-policy-3 restype:aws.sns
2023-03-31 00:10:51,516: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-4/custodian-run.log>
2023-03-31 00:10:51,516: custodian.policy:DEBUG Running policy:test-policy-4 resource:aws.account region:us-east-1 c7n:0.9.24
2023-03-31 00:10:51,842: custodian.resources.account:DEBUG Filtered from 1 to 1 account
2023-03-31 00:10:51,842: custodian.policy:INFO policy:test-policy-4 resource:aws.account region:us-east-1 count:1 time:0.33
2023-03-31 00:10:51,843: custodian.output:DEBUG metric:ResourceCount Count:1 policy:test-policy-4 restype:aws.account scope:policy
2023-03-31 00:10:51,843: custodian.output:DEBUG metric:ApiCalls Count:2 policy:test-policy-4 restype:aws.account
```
Looks like AssumeRole/Refresh was called twice here.

2. With a policy file that contained batch + lambda policies: 
```
2023-03-31 00:09:54,466: custodian.commands:DEBUG Loaded file test.yml. Contains 6 policies
2023-03-31 00:09:54,853: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2HDVWUBUK
2023-03-31T05:09:55+00:00
2023-03-31 00:09:55,286: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-1/custodian-run.log>
2023-03-31 00:09:55,466: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2CHZRC73J
2023-03-31T05:09:55+00:00
2023-03-31 00:09:56,031: custodian.policy:DEBUG Running policy:test-policy-1 resource:aws.kms-key region:us-east-1 c7n:0.9.24
2023-03-31 00:09:56,033: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:09:56,034: custodian.resources.key:DEBUG Using cached c7n.resources.kms.Key: 49
2023-03-31 00:09:56,186: custodian.filters:DEBUG Querying 49 kms-keys' rotation status
2023-03-31 00:09:56,735: custodian.filters:WARNING Access denied when getting rotation status on key:arn:aws:kms:us-east-1:644160558196:key/7efe5f36-5599-4a81-97b9-4c4aa4b64f93
2023-03-31 00:09:56,976: custodian.resources.key:DEBUG Filtered from 49 to 17 key
2023-03-31 00:09:56,977: custodian.policy:INFO policy:test-policy-1 resource:aws.kms-key region:us-east-1 count:17 time:0.95
2023-03-31 00:09:56,979: custodian.output:DEBUG metric:ResourceCount Count:17 policy:test-policy-1 restype:aws.kms-key scope:policy
2023-03-31 00:09:56,980: custodian.output:DEBUG metric:ApiCalls Count:49 policy:test-policy-1 restype:aws.kms-key
2023-03-31 00:09:56,980: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-2/custodian-run.log>
2023-03-31 00:09:56,981: custodian.policy:DEBUG Running policy:test-policy-2 resource:aws.kms-key region:us-east-1 c7n:0.9.24
2023-03-31 00:09:56,982: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:09:56,982: custodian.resources.key:DEBUG Using cached c7n.resources.kms.Key: 49
2023-03-31 00:09:56,987: custodian.filters:DEBUG Querying 49 kms-keys' rotation status
2023-03-31 00:09:57,399: custodian.filters:WARNING Access denied when getting rotation status on key:arn:aws:kms:us-east-1:644160558196:key/7efe5f36-5599-4a81-97b9-4c4aa4b64f93
2023-03-31 00:09:57,655: custodian.resources.key:DEBUG Filtered from 49 to 32 key
2023-03-31 00:09:57,655: custodian.policy:INFO policy:test-policy-2 resource:aws.kms-key region:us-east-1 count:32 time:0.67
2023-03-31 00:09:57,659: custodian.output:DEBUG metric:ResourceCount Count:32 policy:test-policy-2 restype:aws.kms-key scope:policy
2023-03-31 00:09:57,659: custodian.output:DEBUG metric:ApiCalls Count:49 policy:test-policy-2 restype:aws.kms-key
2023-03-31 00:09:57,659: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-3/custodian-run.log>
2023-03-31 00:09:57,660: custodian.policy:DEBUG Running policy:test-policy-3 resource:aws.sns region:us-east-1 c7n:0.9.24
2023-03-31 00:09:57,661: custodian.cache:DEBUG expiring stale cache entries
2023-03-31 00:09:57,661: custodian.resources.sns:DEBUG Using cached c7n.resources.sns.SNS: 8
2023-03-31 00:09:57,662: custodian.resources.sns:DEBUG Filtered from 8 to 7 sns
2023-03-31 00:09:57,662: custodian.policy:INFO policy:test-policy-3 resource:aws.sns region:us-east-1 count:7 time:0.00
2023-03-31 00:09:57,663: custodian.output:DEBUG metric:ResourceCount Count:7 policy:test-policy-3 restype:aws.sns scope:policy
2023-03-31 00:09:57,663: custodian.output:DEBUG metric:ApiCalls Count:0 policy:test-policy-3 restype:aws.sns
2023-03-31 00:09:57,664: custodian.output:DEBUG Storing output with <LogFile file://output/test-policy-4/custodian-run.log>
2023-03-31 00:09:57,664: custodian.policy:DEBUG Running policy:test-policy-4 resource:aws.account region:us-east-1 c7n:0.9.24
2023-03-31 00:09:57,988: custodian.resources.account:DEBUG Filtered from 1 to 1 account
2023-03-31 00:09:57,988: custodian.policy:INFO policy:test-policy-4 resource:aws.account region:us-east-1 count:1 time:0.32
2023-03-31 00:09:57,989: custodian.output:DEBUG metric:ResourceCount Count:1 policy:test-policy-4 restype:aws.account scope:policy
2023-03-31 00:09:57,990: custodian.output:DEBUG metric:ApiCalls Count:2 policy:test-policy-4 restype:aws.account
2023-03-31 00:09:57,990: custodian.output:DEBUG Storing output with <LogFile file://output/ec2-test-foo-1/custodian-run.log>
2023-03-31 00:09:57,991: custodian.policy:INFO Provisioning policy lambda: ec2-test-foo-1 region: us-east-1
2023-03-31 00:09:58,215: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2HW6EFW5X
2023-03-31T05:09:58+00:00
2023-03-31 00:09:59,220: custodian.serverless:DEBUG Created custodian serverless archive size: 3.40mb
2023-03-31 00:09:59,574: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2HPFJ6QWN
2023-03-31T05:09:59+00:00
2023-03-31 00:10:00,453: custodian.output:DEBUG metric:ApiCalls Count:4 policy:ec2-test-foo-1 restype:aws.ec2
2023-03-31 00:10:00,457: custodian.output:DEBUG Storing output with <LogFile file://output/ec2-test-foo-2/custodian-run.log>
2023-03-31 00:10:00,457: custodian.policy:INFO Provisioning policy lambda: ec2-test-foo-2 region: us-east-1
2023-03-31 00:10:00,597: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2EOJ2O66X
2023-03-31T05:10:00+00:00
2023-03-31 00:10:01,610: custodian.serverless:DEBUG Created custodian serverless archive size: 3.40mb
2023-03-31 00:10:01,978: botocore.credentials:INFO Found credentials in environment variables.
ASIAZL6XWCR2IOHK6576
2023-03-31T05:10:02+00:00
2023-03-31 00:10:03,288: custodian.output:DEBUG metric:ApiCalls Count:4 policy:ec2-test-foo-2 restype:aws.ec2
```
In addition to 2 changes in finding 1, looks like the Access_key_id/credential expiration time changed a total of 4 times during the deployment of the lambda. 


### Relevant log/traceback output

We weren't able to get AWS_CSM (SDK metrics) enabled yet due to some reason



Discussion:

Resolving patch:
diff --git a/c7n/commands.py b/c7n/commands.py
--- a/c7n/commands.py
+++ b/c7n/commands.py
@@ -290,8 +290,11 @@ def run(options, policies: List[Policy]) -> None:
     # AWS - Sanity check that we have an assumable role before executing policies
     # Todo - move this behind provider interface
     if options.assume_role and [p for p in policies if p.provider_name == 'aws']:
+        # the cli options we're being handed haven't been initialized by the
+        # provider, instead use one of the provider's policy options.
+        sample_aws = [p for p in policies if p.provider_name == 'aws'].pop()
         try:
-            local_session(clouds['aws']().get_session_factory(options))
+            local_session(clouds['aws']().get_session_factory(sample_aws.options))
         except ClientError:
             log.exception("Unable to assume role %s", options.assume_role)
             sys.exit(1)
diff --git a/c7n/manager.py b/c7n/manager.py
--- a/c7n/manager.py
+++ b/c7n/manager.py
@@ -39,6 +39,7 @@ class ResourceManager:
     executor_factory = ThreadPoolExecutor
     retry = None
     permissions = ()
+    get_client = None
 
     def __init__(self, ctx, data):
         self.ctx = ctx
diff --git a/c7n/query.py b/c7n/query.py
--- a/c7n/query.py
+++ b/c7n/query.py
@@ -264,8 +264,13 @@ def augment(self, resources):
             _augment = _batch_augment
         else:
             return resources
+        if self.manager.get_client:
+            client = self.manager.get_client()
+        else:
+            client = local_session(self.manager.session_factory).client(
+                model.service, region_name=self.manager.config.region)
         _augment = functools.partial(
-            _augment, self.manager, model, detail_spec)
+            _augment, self.manager, model, detail_spec, client)
         with self.manager.executor_factory(
                 max_workers=self.manager.max_workers) as w:
             results = list(w.map(
@@ -451,12 +456,8 @@ class QueryResourceManager(ResourceManager, metaclass=QueryMeta):
     max_workers = 3
     chunk_size = 20
 
-    permissions = ()
-
     _generate_arn = None
 
-    get_client = None
-
     retry = staticmethod(
         get_retry((
             'TooManyRequestsException',
@@ -718,10 +719,8 @@ def get_parent_manager(self):
         return self.get_resource_manager(self.resource_type.parent_spec[0])
 
 
-def _batch_augment(manager, model, detail_spec, resource_set):
+def _batch_augment(manager, model, detail_spec, client, resource_set):
     detail_op, param_name, param_key, detail_path, detail_args = detail_spec
-    client = local_session(manager.session_factory).client(
-        model.service, region_name=manager.config.region)
     op = getattr(client, detail_op)
     if manager.retry:
         args = (op,)
@@ -735,10 +734,8 @@ def _batch_augment(manager, model, detail_spec, resource_set):
     return response[detail_path]
 
 
-def _scalar_augment(manager, model, detail_spec, resource_set):
+def _scalar_augment(manager, model, detail_spec, client, resource_set):
     detail_op, param_name, param_key, detail_path = detail_spec
-    client = local_session(manager.session_factory).client(
-        model.service, region_name=manager.config.region)
     op = getattr(client, detail_op)
     if manager.retry:
         args = (op,)


Test patch: