Repository: cloud-custodian/cloud-custodian

Issue:
Action Delay
So I have made this rule.

```
policies:
  - name: untagged-offhour-ec2-rule
    resource: ec2
    mode:
     type: cloudtrail
     role: arn:aws:iam::************:role/cloudcustodian_lambda_role
     events:
       - source: ec2.amazonaws.com
         event: StartInstances
         ids: "responseElements.instancesSet.items[].instanceId"
       - source: ec2.amazonaws.com
         event: RunInstances
         ids: "responseElements.instancesSet.items[].instanceId"
       - source: ec2.amazonaws.com
         event: RebootInstances
         ids: "responseElements.instancesSet.items[].instanceId"
    filters:
     - "tag:OffHour": empty
    actions:
      - terminate
```

This works fine. Besides when creating other resources that create EC2's, since it takes too long for the console to tag the EC2 it gets deleted too fast.

Is there a way to add a delay before the action? Eg: find a newly created EC2 without the correct tagging, then wait a minute before deleting. (rather than deleting it the second it is created) 

Ps I'm aware of 'CRON' but I need a 1 minute delay if possible.

Discussion:
tjstansell: Linking this to the ticket ... #3165 

Patch:
diff --git a/c7n/policy.py b/c7n/policy.py
index 63b9b9d2b..bae8e04cb 100644
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -600,6 +600,8 @@ class CloudTrailMode(LambdaMode):
 
     schema = utils.type_schema(
         'cloudtrail',
+        delay={'type': 'integer',
+               'description': 'sleep for delay seconds before processing an event'},
         events={'type': 'array', 'items': {
             'oneOf': [
                 {'type': 'string'},
@@ -629,6 +631,13 @@ class CloudTrailMode(LambdaMode):
                     "resource:%s does not support cloudtrail mode policies" % (
                         self.policy.resource_type))
 
+    def resolve_resources(self, event):
+        # override to enable delay before fetching resources
+        delay = self.policy.data.get('mode', {}).get('delay')
+        if delay:
+            time.sleep(delay)
+        return super().resolve_resources(event)
+
 
 @execution.register('ec2-instance-state')
 class EC2InstanceState(LambdaMode):


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
index 44a2c36d9..f4d713178 100644
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -229,6 +229,35 @@ class PolicyLambdaProvision(BaseTest):
              'source': ['aws.health']}
         )
 
+    def test_cloudtrail_delay(self):
+        p = self.load_policy({
+            'name': 'aws-account',
+            'resource': 'aws.account',
+            'mode': {
+                'type': 'cloudtrail',
+                'delay': 32,
+                'role': 'CustodianRole',
+                'events': ['RunInstances']}})
+        from c7n import policy
+
+        class time:
+
+            invokes = []
+
+            @classmethod
+            def sleep(cls, duration):
+                cls.invokes.append(duration)
+
+        self.patch(policy, 'time', time)
+        trail_mode = p.get_execution_mode()
+        results = trail_mode.run({
+            'detail': {
+                'eventSource': 'ec2.amazonaws.com',
+                'eventName': 'RunInstances'}},
+            None)
+        self.assertEqual(len(results), 0)
+        self.assertEqual(time.invokes, [32])
+
     def test_user_pattern_merge(self):
         p = self.load_policy({
             'name': 'ec2-retire',