Repository: cloud-custodian/cloud-custodian

Issue:
Policy using resource: iam-policy recieves UnrecognizedClientException during dry-run in GovCloud
### Describe the bug

I'm receiving an when running the policy outlined below in dry-run mode. It seems to go in successfully when run without dry-run, but I'm not sure if they actual policy is working or not.

This is occuring in AWS GovCloud so the issue might be specific to GovCloud regions.

### What did you expect to happen?

dry-run works successfully. Policy works.

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

```shell
Custodian:   0.9.17
Python:      3.8.0 (default, Dec  9 2021, 17:53:27)
             [GCC 8.4.0]
Platform:    posix.uname_result(sysname='Linux', nodename='<removed>', release='4.15.0-2075-aws-fips', version='#80-Ubuntu SMP Tue Jun 14 01:57:31 UTC 2022', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==2.0.0
attrs==21.4.0
boto3==1.24.10
botocore==1.27.10
docutils==0.17.1
importlib-metadata==4.11.4
importlib-resources==5.7.1
jmespath==1.0.0
jsonschema==4.6.0
pyrsistent==0.18.1
python-dateutil==2.8.2
pyyaml==6.0
s3transfer==0.6.0
six==1.16.0
tabulate==0.8.9
typing-extensions==4.2.0
urllib3==1.26.9
zipp==3.8.0
```


### Policy

```shell
- name: iam-policy-account-audit-policy
    resource: iam-policy
    description: |
      Cloud Custodian IAM Policy account:* Audit
    mode:
      type: periodic
      role: arn:aws-us-gov:iam::{account_id}:role/<removed>
      schedule: "rate(1 day)"      
    filters:
      - type: has-allow-all
      - type: used
    actions:   
      - type: notify
        template: default.html
        template_format: "html"
        priority_header: "5"
        subject: "Cloud Custodian - Security Audit: AWS IAM Policy HasAllowAll"
        to:
          - <email removed>
        transport:
          type: sqs
          queue: <sqs queue removed>
```


### Relevant log/traceback output

```shell
2022-08-04 09:30:03,594: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 526, in resources
    resources = self.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 582, in augment
    return self.source.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/resources/iam.py", line 394, in augment
    return universal_augment(self.manager, super().augment(resources))
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/tags.py", line 91, in universal_augment
    resource_tag_results = client.get_resources(
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
2022-08-04 09:30:03,599: custodian.commands:ERROR Error while executing policy iam-policy-account-audit-policy, continuing
Traceback (most recent call last):
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 1236, in __call__
    resources = PullMode(self).run()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 526, in resources
    resources = self.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/query.py", line 582, in augment
    return self.source.augment(resources)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/resources/iam.py", line 394, in augment
    return universal_augment(self.manager, super().augment(resources))
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/c7n/tags.py", line 91, in universal_augment
    resource_tag_results = client.get_resources(
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "<dir removed>/testing-cloudcustodian/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
```


### Extra information or context

_No response_


Discussion:
which region specifically? this seems like an issue on the aws side, you probably want to inquire to aws support or your TAM. specifically we've already done the iam call, but then we're getting an error using the resource group tagging api with the same credentials.
Us-gov-west-1. Is there a way to trace the calls made to AWS by cloud
custodian? It seems I don't even see a failed call in AWS/etc. in the
target account when this occurs.

On Thu, Aug 4, 2022, 10:29 AM Kapil Thangavelu ***@***.***>
wrote:

> which region specifically? this seems like an issue on the aws side, you
> probably want to inquire to aws support or your TAM. specifically we've
> already done the iam call, but then we're getting an error using the
> resource group tagging api with the same credentials.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/cloud-custodian/cloud-custodian/issues/7610#issuecomment-1205335359>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAOSFGVTXMENKLEO4KXHNADVXPHTJANCNFSM55SSQ32A>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

regarding seeing api calls, the easiest is probably via the builtin aws sdk instrumentation capabilities, ie.
https://boto3.amazonaws.com/v1/documentation/api/1.14.0/guide/sdk-metrics.html

 an alternative, you can optionally enable X-ray tracing via the cli (--trace xray) but given this is a failure scenario its not clear if that's viable here.
i typically use https://vector.dev with csm as a sink/log of calls here's the config file i use for vector, and then i just set the AWS_CSM* env vars up.

```toml
[sources.aws_csm]
type = "socket"
address = "127.0.0.1:31000"
max_length = 102_400
mode = "udp"

[transforms.parse]
type = "remap"
inputs = [ "aws_csm" ]
source = """
. = parse_json!(.message)
del(.SessionToken)
"""

[transforms.filter]
type = "filter"
inputs = ["parse"]
condition = '.Type != "ApiCallAttempt" && .Service != "SSO"'


[sinks.console]
inputs = ["filter"]
type = "console"
encoding = "json"

```
I also want to add my experience, as I see this issue, but only with govcloud regions.

Running custodian version `0.9.35`

Having a policy as simple as what's below, returns the same `botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid`:

```
---
policies:
  - name: iam-all-policies
    resource: aws.iam-policy
    description: |
      Identify All IAM Policies.
```


`custodian run ../UnrecognizedClientException/iam-policy-all.yml -s . --verbose --debug > debug_output.txt 2>&1  `

```
2024-04-08 16:29:37,659: custodian.commands:DEBUG Loaded file ../UnrecognizedClientException/iam-policy-all.yml. Contains 1 policies
2024-04-08 16:29:37,663: custodian.aws:DEBUG using default region:us-gov-west-1 from boto
2024-04-08 16:29:38,994: custodian.output:DEBUG Storing output with <LogFile file://./iam-all-policies/custodian-run.log>
2024-04-08 16:29:38,998: custodian.policy:DEBUG Running policy:iam-all-policies resource:aws.iam-policy region:us-gov-west-1 c7n:0.9.35
2024-04-08 16:29:39,001: custodian.cache:DEBUG expired 1 stale cache entries
2024-04-08 16:29:40,253: custodian.output:DEBUG metric:PolicyException Count:1 policy:iam-all-policies restype:aws.iam-policy
2024-04-08 16:29:40,253: custodian.output:DEBUG metric:ApiCalls Count:2 policy:iam-all-policies restype:aws.iam-policy
2024-04-08 16:29:40,253: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/lib/python3.12/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 543, in resources
    resources = self.augment(resources)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 599, in augment
    return self.source.augment(resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/resources/iam.py", line 396, in augment
    return universal_augment(self.manager, super().augment(resources))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/tags.py", line 92, in universal_augment
    resource_tag_results = client.get_resources(
                           ^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
Traceback (most recent call last):
  File "/lib/python3.12/site-packages/c7n/cli.py", line 363, in main
    command(config)
  File "/lib/python3.12/site-packages/c7n/commands.py", line 128, in _load_policies
    return f(options, list(policies))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/commands.py", line 307, in run
    policy()
  File "/lib/python3.12/site-packages/c7n/policy.py", line 1357, in __call__
    resources = mode.run()
                ^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/policy.py", line 330, in run
    resources = self.policy.resource_manager.resources()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 543, in resources
    resources = self.augment(resources)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/query.py", line 599, in augment
    return self.source.augment(resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/resources/iam.py", line 396, in augment
    return universal_augment(self.manager, super().augment(resources))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/c7n/tags.py", line 92, in universal_augment
    resource_tag_results = client.get_resources(
                           ^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetResources operation: The security token included in the request is invalid
> /lib/python3.12/site-packages/botocore/client.py(1009)_make_api_call()
-> raise error_class(parsed_response, operation_name)
(Pdb) 
```

The same user can run `awscli` commands fine.  

Resolving patch:
diff --git a/c7n/tags.py b/c7n/tags.py
--- a/c7n/tags.py
+++ b/c7n/tags.py
@@ -72,10 +72,8 @@ def universal_augment(self, resources):
     if not resources:
         return resources
 
-    # For global resources, tags don't populate in the get_resources call
-    # unless the call is being made to us-east-1
-    region = getattr(self.resource_type, 'global_resource', None) and 'us-east-1' or self.region
-
+    region = utils.get_resource_tagging_region(self.resource_type, self.region)
+    self.log.debug("Using region %s for resource tagging" % region)
     client = utils.local_session(
         self.session_factory).client('resourcegroupstaggingapi', region_name=region)
 
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -871,6 +871,18 @@ def get_support_region(manager):
     return support_region
 
 
+def get_resource_tagging_region(resource_type, region):
+    # For global resources, tags don't populate in the get_resources call
+    # unless the call is being made to us-east-1. For govcloud this is us-gov-west-1.
+
+    partition = get_partition(region)
+    if partition == "aws":
+        return getattr(resource_type, 'global_resource', None) and 'us-east-1' or region
+    elif partition == "aws-us-gov":
+        return getattr(resource_type, 'global_resource', None) and 'us-gov-west-1' or region
+    return region
+
+
 def get_eni_resource_type(eni):
     if eni.get('Attachment'):
         instance_id = eni['Attachment'].get('InstanceId')


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -10,6 +10,7 @@
 from botocore.exceptions import ClientError
 from dateutil.parser import parse as parse_date
 
+from c7n import query
 from c7n import utils
 from c7n.config import Config
 from .common import BaseTest
@@ -574,6 +575,40 @@ def test_get_support_region(self):
         res = utils.get_support_region(mock_manager)
         self.assertEqual("cn-north-1", res)
 
+    def test_get_resource_tagging_region(self):
+
+        resource_type = query.TypeInfo()
+
+        # Regional endpoint checks
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-east-2'), 'us-east-2')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'ap-southeast-1'), 'ap-southeast-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'eu-west-1'), 'eu-west-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-gov-east-1'), 'us-gov-east-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'cn-north-1'), 'cn-north-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-iso-east-1'), 'us-iso-east-1')
+
+        # Global resource checks
+        resource_type.global_resource = True
+
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-east-2'), 'us-east-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-gov-east-1'), 'us-gov-west-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'cn-north-1'), 'cn-north-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'us-iso-east-1'), 'us-iso-east-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                           'ap-southeast-1'), 'us-east-1')
+        self.assertEqual(utils.get_resource_tagging_region(resource_type,
+                                                              'eu-west-1'), 'us-east-1')
+
     def test_get_eni_resource_type(self):
         self.assertEqual(
             utils.get_eni_resource_type(