Repository: cloud-custodian/cloud-custodian

Issue:
CloudWatch Event Rule fails to create/cleanup
During creation of a CloudWatch Event rule, a ResourceNotFoundException was thrown during the when the CloudWatchEventSource.run method ran the ListTargetsByRule API call. This appears to be due to a race condition, where the Event Rule was successfully created but an attempt to list targets of that new rule immediately after creation failed. The rule does indeed eventually get created, it just seems the call to list against that new rule is made too quickly.

https://github.com/cloud-custodian/cloud-custodian/blob/a337c977e933bc9bf7cf13f0c71ff63312a492fa/c7n/mu.py#L1095
        if rule and self.delta(rule, params):
            log.debug("Updating cwe rule for %s" % func.name)
            response = self.client.put_rule(**params)
        elif not rule:
            log.debug("Creating cwe rule for %s" % (self))
            response = self.client.put_rule(**params)
        else:
            response = {'RuleArn': rule['Arn']}

        client = self.session.client('lambda')
        try:
            client.add_permission(
                FunctionName=func.name,
                StatementId=func.name,
                SourceArn=response['RuleArn'],
                Action='lambda:InvokeFunction',
                Principal='events.amazonaws.com')
            log.debug('Added lambda invoke cwe rule permission')
        except client.exceptions.ResourceConflictException:
            pass

        # Add Targets
        found = False
        response = self.client.list_targets_by_rule(Rule=func.name)

Perhaps a try/except on the list_targets_by_rule call to catch and retry in this case or reworking the code so that the list_targets_by_rule call isn't event attempted when this is a new cloudwatch event.

To further complicate matters, cleaning up the failed resource also fails for a similar reason.  The remove function performs a list_targets_by_rule and then immediately trys to remove those targets. However because the rule never had any targets added, the call to remove_targets fails because the Value for Ids is 0 and a ParamValidationError is thrown (valid range: 1-inf)

The remove code assumes that at least one target exists, 

https://github.com/cloud-custodian/cloud-custodian/blob/a337c977e933bc9bf7cf13f0c71ff63312a492fa/c7n/mu.py#L1154

                targets = self.client.list_targets_by_rule(
                    Rule=func.name)['Targets']
                self.client.remove_targets(
                    Rule=func.name,
                    Ids=[t['Id'] for t in targets])

perhaps this instead:

                targets = self.client.list_targets_by_rule(
                    Rule=func.name)['Targets']
                if targets:
                    self.client.remove_targets(
                        Rule=func.name,
                        Ids=[t['Id'] for t in targets])



Discussion:
thanks for the bug report, which region was this eventual consistency evident in? cleanup is done via tools/ops/mugc.py at the moment. but noted re retry on list and empty target handling on delete.
Issue occured in us-east-1. I was actually using mugc.py for the cleanup, using the region_gc function, when I hit the removal issue.

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -47,6 +47,7 @@
 log = logging.getLogger('custodian.serverless')
 
 LambdaRetry = get_retry(('InsufficientPermissionsException',), max_attempts=2)
+RuleRetry = get_retry(('ResourceNotFoundException',), max_attempts=2)
 
 
 class PythonPackageArchive:
@@ -1113,7 +1114,7 @@ def add(self, func):
 
         # Add Targets
         found = False
-        response = self.client.list_targets_by_rule(Rule=func.name)
+        response = RuleRetry(self.client.list_targets_by_rule, Rule=func.name)
         # CloudWatchE seems to be quite picky about function arns (no aliases/versions)
         func_arn = func.arn
 
@@ -1155,9 +1156,10 @@ def remove(self, func):
             try:
                 targets = self.client.list_targets_by_rule(
                     Rule=func.name)['Targets']
-                self.client.remove_targets(
-                    Rule=func.name,
-                    Ids=[t['Id'] for t in targets])
+                if targets:
+                    self.client.remove_targets(
+                        Rule=func.name,
+                        Ids=[t['Id'] for t in targets])
             except ClientError as e:
                 log.warning(
                     "Could not remove targets for rule %s error: %s",


Test patch: