Repository: cloud-custodian/cloud-custodian

Issue:
'custodian -r all', when running in Python 3, throws error if region disabled
AWS recently introduced a new region, HKG (ap-east-1) which is disabled by default.

I installed the most current version of c7n as of today (0.8.44.2) in two separate environments, one running Amazon Linux 1 and Python 2.7, and the other running Amazon Linux 2 and Python 3.

When the 'custodian' command is run with '-r all' parameter in Python 3 environment, and HKG is disabled in the AWS account, it will throw errors.

The same 'custodian' command run with '-r all' parameter in a Python 2.7 environment, with HKG also disabled in the AWS account, does NOT throw any errors.

Shell outputs below:

**Python 3:**
```
$ python --version
Python 3.7.3

$ pip --version
pip 19.1.1 from /home/ec2-user/custodian/lib/python3.7/site-packages/pip (python 3.7)

$ pip list|grep c7n
c7n                0.8.44.2 
c7n-org            0.5.3    

$ custodian run -s ~/output -r all list-ec2-instances.yaml --cache-period 0

2019-06-29 02:53:26,186: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/policy.py", line 229, in run
    resources = self.policy.resource_manager.resources()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/resources/ec2.py", line 92, in resources
    return super(EC2, self).resources(query=query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 453, in resources
    resources = self.source.resources(query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 230, in resources
    return self.query.filter(self.manager, **query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 90, in filter
    getattr(resource_manager, 'retry', None)) or []
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 69, in _invoke_client_enum
    data = results.build_full_result()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/paginate.py", line 449, in build_full_result
    for response in self:
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 679, in _make_request
    return self.retry(self._method, **current_kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/utils.py", line 383, in _retry
    return func(*args, **kw)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
2019-06-29 02:53:26,187: custodian.commands:ERROR Error while executing policy list-ec2-instances, continuing
Traceback (most recent call last):
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/commands.py", line 269, in run
    policy()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/policy.py", line 945, in __call__
    resources = mode.run()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/policy.py", line 229, in run
    resources = self.policy.resource_manager.resources()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/resources/ec2.py", line 92, in resources
    return super(EC2, self).resources(query=query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 453, in resources
    resources = self.source.resources(query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 230, in resources
    return self.query.filter(self.manager, **query)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 90, in filter
    getattr(resource_manager, 'retry', None)) or []
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 69, in _invoke_client_enum
    data = results.build_full_result()
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/paginate.py", line 449, in build_full_result
    for response in self:
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/query.py", line 679, in _make_request
    return self.retry(self._method, **current_kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/c7n/utils.py", line 383, in _retry
    return func(*args, **kw)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/custodian/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
2019-06-29 02:53:26,834: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-northeast-1 count:0 time:0.63
2019-06-29 02:53:27,456: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-northeast-2 count:0 time:0.60
2019-06-29 02:53:28,463: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-south-1 count:0 time:0.99
2019-06-29 02:53:29,238: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-southeast-1 count:0 time:0.76
2019-06-29 02:53:30,093: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-southeast-2 count:0 time:0.84
2019-06-29 02:53:30,505: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ca-central-1 count:0 time:0.39
2019-06-29 02:53:31,297: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-central-1 count:0 time:0.77
2019-06-29 02:53:32,292: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-north-1 count:0 time:0.97
2019-06-29 02:53:32,955: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-1 count:0 time:0.64
2019-06-29 02:53:33,677: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-2 count:0 time:0.70
2019-06-29 02:53:34,443: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-3 count:0 time:0.75
2019-06-29 02:53:35,446: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:sa-east-1 count:0 time:0.98
2019-06-29 02:53:35,984: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-east-1 count:0 time:0.52
2019-06-29 02:53:36,393: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-east-2 count:0 time:0.39
2019-06-29 02:53:36,611: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-west-1 count:0 time:0.20
2019-06-29 02:53:36,756: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-west-2 count:2 time:0.13
```

**Python 2.7:**
```
$ python --version
Python 2.7.16

$ pip --version
pip 19.1.1 from /home/ec2-user/custodian/local/lib/python2.7/site-packages/pip (python 2.7)

$ pip list|grep c7n
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
c7n                0.8.44.2   
c7n-org            0.5.3      

$ custodian run -s ~/output -r all list-ec2-instances.yaml --cache-period 0
2019-06-29 03:04:53,630: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-northeast-1 count:0 time:0.65
2019-06-29 03:04:54,487: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-northeast-2 count:0 time:0.84
2019-06-29 03:04:55,717: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-south-1 count:0 time:1.21
2019-06-29 03:04:56,757: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-southeast-1 count:0 time:1.02
2019-06-29 03:04:57,777: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ap-southeast-2 count:0 time:1.00
2019-06-29 03:04:58,284: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:ca-central-1 count:0 time:0.49
2019-06-29 03:04:59,398: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-central-1 count:0 time:1.09
2019-06-29 03:05:00,518: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-north-1 count:0 time:1.10
2019-06-29 03:05:01,657: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-1 count:0 time:1.12
2019-06-29 03:05:02,484: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-2 count:0 time:0.81
2019-06-29 03:05:03,798: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:eu-west-3 count:0 time:1.30
2019-06-29 03:05:04,825: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:sa-east-1 count:0 time:1.01
2019-06-29 03:05:05,873: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-east-1 count:0 time:1.03
2019-06-29 03:05:06,415: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-east-2 count:0 time:0.53
2019-06-29 03:05:06,780: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-west-1 count:0 time:0.35
2019-06-29 03:05:07,127: custodian.policy:INFO policy:list-ec2-instances resource:ec2 region:us-west-2 count:3 time:0.33
```



Discussion:
Note that if HKG region is enabled in the account, error goes away in Python 3.
Hi @tzd-sh, 

this was fixed for `c7n-org` in #4006, but I guess it hasn't been fixed yet in `c7n`. Pretty sure the guys here would appreciate a pull request fixing it! 

Cheers,
Dario
I'll appreciate a fix too, thx!
@tzd-sh contributions welcome.
I only write shell and perl scripts.
@kapilt @dvianello I'm willing to contribute a Python PR but it's not apparent to me where in the code to make a change similar to PR #4006. Seems to be many layers of abstraction and classes deep.

Resolving patch:
diff --git a/c7n/resources/aws.py b/c7n/resources/aws.py
--- a/c7n/resources/aws.py
+++ b/c7n/resources/aws.py
@@ -537,7 +537,13 @@ def initialize_policies(self, policy_collection, options):
         policies = []
         service_region_map, resource_service_map = get_service_region_map(
             options.regions, policy_collection.resource_types)
-
+        if 'all' in options.regions:
+            enabled_regions = set([
+                r['RegionName'] for r in
+                get_profile_session(options).client('ec2').describe_regions(
+                    Filters=[{'Name': 'opt-in-status',
+                              'Values': ['opt-in-not-required', 'opted-in']}]
+                ).get('Regions')])
         for p in policy_collection:
             if 'aws.' in p.resource_type:
                 _, resource_type = p.resource_type.split('.', 1)
@@ -553,7 +559,7 @@ def initialize_policies(self, policy_collection, options):
                 candidate = candidates and candidates[0] or 'us-east-1'
                 svc_regions = [candidate]
             elif 'all' in options.regions:
-                svc_regions = available_regions
+                svc_regions = list(set(available_regions).intersection(enabled_regions))
             else:
                 svc_regions = options.regions
 


Test patch:
diff --git a/tests/data/placebo/test_aws_policy_global_expand/ec2.DescribeRegions_1.json b/tests/data/placebo/test_aws_policy_global_expand/ec2.DescribeRegions_1.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_aws_policy_global_expand/ec2.DescribeRegions_1.json
@@ -0,0 +1,44 @@
+{
+    "status_code": 200,
+    "data": {
+        "Regions": [
+            {
+                "Endpoint": "ec2.eu-west-2.amazonaws.com",
+                "RegionName": "eu-west-2",
+                "OptInStatus": "opt-in-not-required"
+            },
+            {
+                "Endpoint": "ec2.eu-west-1.amazonaws.com",
+                "RegionName": "eu-west-1",
+                "OptInStatus": "opt-in-not-required"
+            },
+            {
+                "Endpoint": "ec2.us-east-1.amazonaws.com",
+                "RegionName": "us-east-1",
+                "OptInStatus": "opt-in-not-required"
+            },
+            {
+                "Endpoint": "ec2.us-east-2.amazonaws.com",
+                "RegionName": "us-east-2",
+                "OptInStatus": "opt-in-not-required"
+            },
+            {
+                "Endpoint": "ec2.us-west-2.amazonaws.com",
+                "RegionName": "us-west-2",
+                "OptInStatus": "opt-in-not-required"
+            }
+        ],
+        "ResponseMetadata": {
+            "RequestId": "888d6a44-3acb-418c-bf0a-fb0f0f65f95d",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "content-type": "text/xml;charset=UTF-8",
+                "content-length": "3655",
+                "vary": "accept-encoding",
+                "date": "Mon, 30 Sep 2019 13:22:31 GMT",
+                "server": "AmazonEC2"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
diff --git a/tests/data/placebo/test_aws_policy_region_expand/ec2.DescribeRegions_1.json b/tests/data/placebo/test_aws_policy_region_expand/ec2.DescribeRegions_1.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_aws_policy_region_expand/ec2.DescribeRegions_1.json
@@ -0,0 +1,29 @@
+{
+    "status_code": 200,
+    "data": {
+        "Regions": [
+            {
+                "Endpoint": "ec2.us-east-1.amazonaws.com",
+                "RegionName": "us-east-1",
+                "OptInStatus": "opt-in-not-required"
+            },
+            {
+                "Endpoint": "ec2.us-west-1.amazonaws.com",
+                "RegionName": "us-west-1",
+                "OptInStatus": "opt-in-not-required"
+            }
+        ],
+        "ResponseMetadata": {
+            "RequestId": "d6b5f1ee-dc2d-4180-af85-7f4f73d2d562",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "content-type": "text/xml;charset=UTF-8",
+                "content-length": "3655",
+                "vary": "accept-encoding",
+                "date": "Mon, 30 Sep 2019 13:21:32 GMT",
+                "server": "AmazonEC2"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
diff --git a/tests/test_policy.py b/tests/test_policy.py
--- a/tests/test_policy.py
+++ b/tests/test_policy.py
@@ -23,6 +23,7 @@
 from c7n import policy, manager
 from c7n.provider import clouds
 from c7n.exceptions import ResourceLimitExceeded, PolicyValidationError
+from c7n.resources import aws
 from c7n.resources.aws import AWS
 from c7n.resources.ec2 import EC2
 from c7n.utils import dumps
@@ -362,6 +363,7 @@ def test_expand_partitions(self):
         original = policy.PolicyCollection.from_data(
             {"policies": [{"name": "foo", "resource": "ec2"}]}, cfg
         )
+
         collection = AWS().initialize_policies(original, cfg)
         self.assertEqual(
             sorted([p.options.region for p in collection]),
@@ -369,6 +371,9 @@ def test_expand_partitions(self):
         )
 
     def test_policy_account_expand(self):
+        factory = self.replay_flight_data('test_aws_policy_region_expand')
+        self.patch(aws, '_profile_session', factory())
+
         original = policy.PolicyCollection.from_data(
             {"policies": [{"name": "foo", "resource": "account"}]},
             Config.empty(regions=["us-east-1", "us-west-2"]),
@@ -400,6 +405,9 @@ def test_policy_expand_group_region(self):
              ('foo', 'us-west-2')])
 
     def test_policy_region_expand_global(self):
+        factory = self.replay_flight_data('test_aws_policy_global_expand')
+        self.patch(aws, '_profile_session', factory())
+
         original = policy.PolicyCollection.from_data(
             {
                 "policies": [