Repository: cloud-custodian/cloud-custodian

Issue:
ConfigRule: InsufficientPermissionsException 
two policies:

```
policies:
  - name: cross-account-rds-snapshot
    resource: rds-snapshot
    mode:
      type: config-rule
      role: "arn:aws:iam::xxxxxxxxx:role/Custodian"
    tags:
      - security:external
    filters:
    - or:
        - type: cross-account
          whitelist: *accounts
        - "tag:rds-snapshot-exempt": absent
        - "tag:security-exempt": absent
    actions:
      - *notify

  - name: cross-account-lambda
    resource: lambda
    mode:
      type: config-rule
      role: "arn:aws:iam::xxxxxxxxx:role/Custodian"
    tags:
      - security:external
    filters:
      - or:
        - type: cross-account
          whitelist: *accounts
        - "tag:lambda-exempt": absent
        - "tag:security-exempt": absent
    actions:
      - *notify
```

running command with Admin priv access keys

```
custodian run -r us-east-2 -s /tmp/test dev-resources-access.yml --profile default
```

Output:
```
2019-08-08 16:03:56,496: custodian.policy:INFO Provisioning policy lambda cross-account-rds-snapshot
2019-08-08 16:03:58,918: custodian.policy:INFO Provisioning policy lambda cross-account-lambda
2019-08-08 16:04:02,008: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 497, in provision
    role=self.policy.options.assume_role)
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 309, in publish
    if e.add(func):
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 1543, in add
    return self.client.put_config_rule(ConfigRule=params)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-2:xxxxxxxxx:function:custodian-cross-account-lambda cannot be invoked. Check the specified function ARN, and check the function's permissions.
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/c7n/cli.py", line 368, in main
    command(config)
  File "/Library/Python/2.7/site-packages/c7n/commands.py", line 136, in _load_policies
    return f(options, list(policies))
  File "/Library/Python/2.7/site-packages/c7n/commands.py", line 268, in run
    policy()
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 945, in __call__
    resources = mode.provision()
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 497, in provision
    role=self.policy.options.assume_role)
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 309, in publish
    if e.add(func):
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 1543, in add
    return self.client.put_config_rule(ConfigRule=params)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-2:xxxxxxxxxxx:function:custodian-cross-account-lambda cannot be invoked. Check the specified function ARN, and check the function's permissions.
> /Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py(661)_make_api_call()
-> raise error_class(parsed_response, operation_name)
(Pdb) Traceback (most recent call last):
  File "/usr/local/bin/custodian", line 9, in <module>
    load_entry_point('c7n==0.8.43.1', 'console_scripts', 'custodian')()
  File "/Library/Python/2.7/site-packages/c7n/cli.py", line 373, in main
    pdb.post_mortem(sys.exc_info()[-1])
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 1267, in post_mortem
    p.interaction(None, t)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py", line 210, in interaction
    self.cmdloop()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cmd.py", line 130, in cmdloop
    line = raw_input(self.prompt)
KeyboardInterrupt
```


Discussion:
If I look at config I see only one:

<img width="1165" alt="Screen Shot 2019-08-08 at 9 53 05 PM" src="https://user-images.githubusercontent.com/51141202/62754982-fac13380-ba26-11e9-9bcb-83fcd1cdc3be.png">

even tried adding, role has admin access:

```
      execution-options:
        metrics_enabled: aws://?region=us-east-2
        assume_role: arn:aws:iam::xxxxxxxx:role/Custodian
```
was this originally deployed as a config rule, ie. was this a lambda policy that was changed? we generally add the permission the lambda post put of the config rule, that generally works fine, and i checked terraform's generation here where they do the same. i'll continue to investigate but it would be useful to understand the differential here.
clean all old lambda and rules, running these policies:

```
config-rule: &config-mode
  type: config-rule
  environment:
    Variables:
      C7N_CATCH_ERR: 'yes'
  role: "arn:aws:iam::xxxxxx:role/CustodianGuard"
  tags:
    App: Custodian
  execution-options:
    metrics_enabled: aws://?region=us-east-2

policies:
- name: cross-account-lambda
  mode:
    type: config-rule
    role: "arn:aws:iam::xxxxxxxx:role/CustodianGuard"
  tags:
    - security:external
  resource: lambda
  filters:
    - type: cross-account
      whitelist: *accounts
    - "tag:lambda-exempt": absent
    - "tag:security-exempt": absent
  actions:
    - *notify


```


1st run:
```
.tomar$ custodian run -r us-east-2 -s /tmp/resources-access-config-rule.yml --profile default
2019-08-09 10:30:48,340: custodian.policy:INFO Provisioning policy lambda cross-account-lambda
2019-08-09 10:30:51,722: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 497, in provision
    role=self.policy.options.assume_role)
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 309, in publish
    if e.add(func):
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 1543, in add
    return self.client.put_config_rule(ConfigRule=params)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-2:xxxxxxxxxx:function:custodian-cross-account-lambda cannot be invoked. Check the specified function ARN, and check the function's permissions.

tomar$ 
```

only 1 policiy ran: rds
2nd attempt with adding 3 more policies:

```
  - name: audit-group-access
    description: "Checks for overly privileged groups"
    resource: iam-group
    region: us-east-2
    tags:
      - security:iam
      - security:privilege
    mode: *config-mode

  - name: audit-user-access
    description: "Check for overly privileged users with access keys"
    resource: iam-user
    region: us-east-2
    tags:
      - security:iam
      - security:privilege
    mode: *config-mode

  - name: audit-user-unused
    description: "Check for user accounts that haven't been used in 365 days"
    resource: iam-user
    region: us-east-2
    tags:
      - security:iam
      - security:unused
    mode: *config-mode
```
2 worked from here:

```
2019-08-09 11:23:06,987: custodian.policy:INFO Provisioning policy lambda audit-group-access
2019-08-09 11:23:09,655: custodian.policy:INFO Provisioning policy lambda audit-user-access
2019-08-09 11:23:12,174: custodian.policy:INFO Provisioning policy lambda audit-user-unused
2019-08-09 11:23:15,468: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 497, in provision
    role=self.policy.options.assume_role)
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 309, in publish
    if e.add(func):
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 1543, in add
    return self.client.put_config_rule(ConfigRule=params)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-2:xxxxxxxxxx:function:custodian-audit-user-unused cannot be invoked. Check the specified function ARN, and check the function's permissions.
2019-08-09 11:23:15,470: custodian.commands:ERROR Error while executing policy audit-user-unused, continuing
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/c7n/commands.py", line 268, in run
    policy()
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 945, in __call__
    resources = mode.provision()
  File "/Library/Python/2.7/site-packages/c7n/policy.py", line 497, in provision
    role=self.policy.options.assume_role)
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 309, in publish
    if e.add(func):
  File "/Library/Python/2.7/site-packages/c7n/mu.py", line 1543, in add
    return self.client.put_config_rule(ConfigRule=params)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/varun.tomar/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-2:xxxxxxxxxx:function:custodian-audit-user-unused cannot be invoked. Check the specified function ARN, and check the function's permissions.
```
ic.. its eventually consistent issue with iam.
I've tried a few times to reproduce this but even targeting distant regions, i'm not seeing any errors.


```
(custodian) kapilt@realms-scythe ~/p/custodian> export AWS_DEFAULT_REGION=us-west-2
(custodian) kapilt@realms-scythe ~/p/custodian> custodian run cr.yml -s out -v

2019-09-25 10:16:39,021: custodian.commands:DEBUG Loaded file cr.yml. Contains 2 policies                                                                                   
2019-09-25 10:16:39,033: custodian.aws:DEBUG using default region:us-west-2 from boto
2019-09-25 10:16:39,197: custodian.output:DEBUG Storing output with <LogFile file://out/cross-account-rds-snapshot/custodian-run.log>                                       
2019-09-25 10:16:39,213: custodian.policy:INFO Provisioning policy lambda cross-account-rds-snapshot                                                                        
2019-09-25 10:16:39,466: custodian.serverless:DEBUG Created custodian serverless archive size: 0.55mb                                                                       
2019-09-25 10:16:39,885: custodian.serverless:INFO Publishing custodian policy lambda function custodian-cross-account-rds-snapshot                                         
2019-09-25 10:16:41,959: custodian.serverless:DEBUG Adding config rule for custodian-cross-account-rds-snapshot                                                             
2019-09-25 10:16:42,422: custodian.serverless:DEBUG Added event source: <ConfigRule> to function: arn:aws:lambda:us-west-2:xxx:function:custodian-cross-account-rds-snapshot
2019-09-25 10:16:42,426: custodian.output:DEBUG metric:ApiCalls Count:5 policy:cross-account-rds-snapshot restype:rds-snapshot                                              
2019-09-25 10:16:42,429: custodian.output:DEBUG Storing output with <LogFile file://out/cross-account-lambda/custodian-run.log>                                             
2019-09-25 10:16:42,430: custodian.policy:INFO Provisioning policy lambda cross-account-lambda                                                                              
2019-09-25 10:16:42,550: custodian.serverless:DEBUG Created custodian serverless archive size: 0.55mb                                                                       
2019-09-25 10:16:42,939: custodian.serverless:INFO Publishing custodian policy lambda function custodian-cross-account-lambda                                               
2019-09-25 10:16:45,103: custodian.serverless:DEBUG Adding config rule for custodian-cross-account-lambda                                                                   
2019-09-25 10:16:45,567: custodian.serverless:DEBUG Added event source: <ConfigRule> to function: arn:aws:lambda:us-west-2:xxx:function:custodian-cross-account-lambda
2019-09-25 10:16:45,571: custodian.output:DEBUG metric:ApiCalls Count:5 policy:cross-account-lambda restype:lambda            
                                              
(custodian) kapilt@realms-scythe ~/p/custodian> export AWS_DEFAULT_REGION=eu-west-1
(custodian) kapilt@realms-scythe ~/p/custodian> custodian run cr.yml -s out -v
2019-09-25 10:17:12,840: custodian.commands:DEBUG Loaded file cr.yml. Contains 2 policies
2019-09-25 10:17:12,853: custodian.aws:DEBUG using default region:eu-west-1 from boto
2019-09-25 10:17:12,976: custodian.output:DEBUG Storing output with <LogFile file://out/cross-account-rds-snapshot/custodian-run.log>
2019-09-25 10:17:12,990: custodian.policy:INFO Provisioning policy lambda cross-account-rds-snapshot
2019-09-25 10:17:13,136: custodian.serverless:DEBUG Created custodian serverless archive size: 0.55mb
2019-09-25 10:17:13,597: custodian.serverless:INFO Publishing custodian policy lambda function custodian-cross-account-rds-snapshot
2019-09-25 10:17:15,885: custodian.serverless:DEBUG Adding config rule for custodian-cross-account-rds-snapshot
2019-09-25 10:17:16,423: custodian.serverless:DEBUG Added event source: <ConfigRule> to function: arn:aws:lambda:eu-west-1:xxx:function:custodian-cross-account-rds-snapshot
2019-09-25 10:17:16,427: custodian.output:DEBUG metric:ApiCalls Count:5 policy:cross-account-rds-snapshot restype:rds-snapshot
2019-09-25 10:17:16,430: custodian.output:DEBUG Storing output with <LogFile file://out/cross-account-lambda/custodian-run.log>
2019-09-25 10:17:16,431: custodian.policy:INFO Provisioning policy lambda cross-account-lambda
2019-09-25 10:17:16,547: custodian.serverless:DEBUG Created custodian serverless archive size: 0.55mb
2019-09-25 10:17:16,967: custodian.serverless:INFO Publishing custodian policy lambda function custodian-cross-account-lambda
2019-09-25 10:17:19,187: custodian.serverless:DEBUG Adding config rule for custodian-cross-account-lambda
2019-09-25 10:17:19,659: custodian.serverless:DEBUG Added event source: <ConfigRule> to function: arn:aws:lambda:eu-west-1:xxx:function:custodian-cross-account-lambda
2019-09-25 10:17:19,663: custodian.output:DEBUG metric:ApiCalls Count:5 policy:cross-account-lambda restype:lambda
```
looking over terraform's provisioning on config rule, they have an explicit catch retry on permission issues as they note iam is eventually consistent, so its going with the same theory that its an iam race condition, we should probably just catch and retry explicitly on permission denied.


```golang
                _, err := conn.PutConfigRule(&input)
                if err != nil {
                        if awsErr, ok := err.(awserr.Error); ok {
                                if awsErr.Code() == "InsufficientPermissionsException" {
                                        // IAM is eventually consistent
                                        return resource.RetryableError(err)
                                } 
                        } 

                        return resource.NonRetryableError(fmt.Errorf("Failed to create AWSConfig rule: %s", err))
                }
```
also when reporting errors its also very helpful to use -v for verbose output.

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -37,10 +37,12 @@
 from c7n.exceptions import ClientError
 from c7n.cwe import CloudWatchEvents
 from c7n.logs_support import _timestamp_from_string
-from c7n.utils import parse_s3, local_session
+from c7n.utils import parse_s3, local_session, get_retry
 
 log = logging.getLogger('custodian.serverless')
 
+LambdaRetry = get_retry(('InsufficientPermissionsException',), max_attempts=2)
+
 
 class PythonPackageArchive(object):
     """Creates a zip file for python lambda functions.
@@ -1620,7 +1622,7 @@ def add(self, func):
         if rule and self.delta(rule, params):
             log.debug("Updating config rule for %s" % self)
             rule.update(params)
-            return self.client.put_config_rule(ConfigRule=rule)
+            return LambdaRetry(self.client.put_config_rule, ConfigRule=rule)
         elif rule:
             log.debug("Config rule up to date")
             return
@@ -1636,7 +1638,7 @@ def add(self, func):
             pass
 
         log.debug("Adding config rule for %s" % func.name)
-        return self.client.put_config_rule(ConfigRule=params)
+        return LambdaRetry(self.client.put_config_rule, ConfigRule=params)
 
     def remove(self, func):
         rule = self.get(func.name)


Test patch: