Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org crashing when it hits an account not authorized
### Describe the bug

Running thru a large organizations yaml.  Some accounts aren't not authorized to assume (yet).  Normally c7n-org warns and moves on but with c7n-org 0.6.14 on (amazon linux 2) ARM64
Python 3.7.10 (default, Jun  3 2021, 00:01:07)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-13)] on linux
c7n-org crashes???

```2022-04-16 12:23:10,740: c7n_org:ERROR unable to obtain credentials for account:EBL role:arn:aws:iam::264xxxxxx3791:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role error:An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::XXXXXXX:assumed-role/cloudcustodian-task-role/i-06be7XXXXXX9060 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::264xxxxxx791:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role
Traceback (most recent call last):
  File "/usr/local/bin/c7n-org", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 681, in run
    for r in resolve_regions(region or a.get('regions', ()), a):
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 201, in resolve_regions
    session = get_session(account, 'c7n-org', "us-east-1")
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 236, in get_session
    session=s)
  File "/usr/local/lib/python3.7/site-packages/c7n/credentials.py", line 104, in assumed_session
    metadata=refresh(),
  File "/usr/local/lib/python3.7/site-packages/c7n/credentials.py", line 95, in refresh
    session, region).assume_role, **parameters)['Credentials']
  File "/usr/local/lib/python3.7/site-packages/c7n/utils.py", line 446, in _retry
    return func(*args, **kw)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 745, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::773xxxxxxx49:assumed-role/cloudcustodian-task-role/i-06bZZZZZb809060 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::264XXXXXXX1:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role```



### What did you expect to happen?

I expected c7n-org to continue on into all the accounts and simply warn as it has on other versions?

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

```shell
Custodian:   0.9.15
Python:      3.7.10 (default, Jun  3 2021, 00:01:07)
             [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
Platform:    posix.uname_result(sysname='Linux', nodename='1bc5fbf4d642', release='4.14.231-173.361.amzn2.aarch64', version='#1 SMP Mon Apr 26 20:57:10 UTC 2021', machine='aarch64')
Using venv:  False
Docker: True
Installed:

argcomplete==2.0.0
attrs==21.4.0
boto3==1.21.40
botocore==1.24.40
docutils==0.17.1
importlib-metadata==4.11.3
importlib-resources==5.7.0
jmespath==0.10.0
jsonschema==4.4.0
pyrsistent==0.18.1
python-dateutil==2.8.2
pyyaml==6.0
s3transfer==0.5.2
six==1.16.0
tabulate==0.8.9
typing-extensions==4.1.1
urllib3==1.26.9
zipp==3.8.0
```
```


### Policy

_No response_

### Relevant log/traceback output

```shell
2022-04-16 12:23:10,740: c7n_org:ERROR unable to obtain credentials for account:EBL role:arn:aws:iam::264079423791:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role error:An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::7733xxxxxx549:assumed-role/cloudcustodian-task-role/i-06bexxxxxx060 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::2640xxxxxx91:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role
Traceback (most recent call last):
  File "/usr/local/bin/c7n-org", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 681, in run
    for r in resolve_regions(region or a.get('regions', ()), a):
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 201, in resolve_regions
    session = get_session(account, 'c7n-org', "us-east-1")
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 236, in get_session
    session=s)
  File "/usr/local/lib/python3.7/site-packages/c7n/credentials.py", line 104, in assumed_session
    metadata=refresh(),
  File "/usr/local/lib/python3.7/site-packages/c7n/credentials.py", line 95, in refresh
    session, region).assume_role, **parameters)['Credentials']
  File "/usr/local/lib/python3.7/site-packages/c7n/utils.py", line 446, in _retry
    return func(*args, **kw)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 745, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::77XXXXXX49:assumed-role/cloudcustodian-task-role/i-06be060 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::2640XXXXX1:role/cl/app/crossaccount/cloudcustodian/cloudops-custodian-role
```


### Extra information or context

_No response_


Discussion:
yeah. this is a regression i think from when we switched region resolution to be account specific to handle account specific region enablement.
Is there a workaround other than back rev'n c7n-org?  Just ran into this again - keep forgetting to set version on c7n-org install.

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -198,9 +198,16 @@ def init(config, use, debug, verbose, accounts, tags, policies, resource=None, p
 
 def resolve_regions(regions, account):
     if 'all' in regions:
-        session = get_session(account, 'c7n-org', "us-east-1")
-        client = session.client('ec2')
-        return [region['RegionName'] for region in client.describe_regions()['Regions']]
+        try:
+            session = get_session(account, 'c7n-org', 'us-east-1')
+            client = session.client('ec2')
+            return [region['RegionName'] for region in client.describe_regions()['Regions']]
+        except ClientError as e:
+            if e.response['Error']['Code'] == 'AccessDenied':
+                log.warning('access denied listing available regions for account:%s',
+                    account['name'])
+                return []
+            raise
     if not regions:
         return ('us-east-1', 'us-west-2')
 


Test patch: