Repository: cloud-custodian/cloud-custodian

Issue:
Action Delay
So I have made this rule.

```
policies:
  - name: untagged-offhour-ec2-rule
    resource: ec2
    mode:
     type: cloudtrail
     role: arn:aws:iam::************:role/cloudcustodian_lambda_role
     events:
       - source: ec2.amazonaws.com
         event: StartInstances
         ids: "responseElements.instancesSet.items[].instanceId"
       - source: ec2.amazonaws.com
         event: RunInstances
         ids: "responseElements.instancesSet.items[].instanceId"
       - source: ec2.amazonaws.com
         event: RebootInstances
         ids: "responseElements.instancesSet.items[].instanceId"
    filters:
     - "tag:OffHour": empty
    actions:
      - terminate
```

This works fine. Besides when creating other resources that create EC2's, since it takes too long for the console to tag the EC2 it gets deleted too fast.

Is there a way to add a delay before the action? Eg: find a newly created EC2 without the correct tagging, then wait a minute before deleting. (rather than deleting it the second it is created) 

Ps I'm aware of 'CRON' but I need a 1 minute delay if possible.


Discussion:
doing tag compliance in a cloud trail policy for ec2 is inherently racy since most provisioning tools aren't yet using the new run instances api parameter for tags, but instead tagging post create. adding random delays isn't really something we've wanted to support as it will typically just work some of the time, its better to structure a policy as not being racy. see my comments in #3162 for alternatives.

you can `mark-for-op` to tag something for consideration by a future policy execution (using the filter `marked-for-op` on the later policy).



I've actually been thinking about this problem set lately with regard to S3 buckets.  I'd really like to be able to watch for bucket creations, then apply a whole series of polices... but don't want to necessarily overwrite settings that might be getting applied by cloudformation or terraform for the bucket.  Unfortunately, since S3 bucket configurations are each a separate API call, this takes several seconds.  In one recent bucket, these were the timings:
```
04/24/2020 16:05:21.000 -0700	CreateBucket
04/24/2020 16:05:22.000 -0700	PutBucketEncryption
04/24/2020 16:05:22.000 -0700	PutBucketTagging
04/24/2020 16:05:23.000 -0700	PutBucketLifecycle
04/24/2020 16:05:24.000 -0700	PutBucketLogging
04/24/2020 16:05:25.000 -0700	PutBucketVersioning
04/24/2020 16:05:25.000 -0700	PutBucketPublicAccessBlock
04/24/2020 16:05:47.000 -0700	PutBucketPolicy
```
Sure, we could tag new resources and then at a later time scan for resources that need to be reviewed.  Unfortunately, that also tends to create at least a 1 hour gap where the resource exists before we can check against our policies.  If we could add an intrinsic delay between receiving the event and querying the resources, we could be much more confident that all settings have been applied and our policies could be evaluated (nearly) immediately.  And I'll point out that we already have this race condition today.  I've seen custodian policies fire only 3 seconds after the cloudtrail event and as you can see in the above example, it was still setting the versioning and public access block configs after 4 seconds.

An obvious option would be to provide a `delay` option that would internally just sleep for some amount of time after processing the `conditions` block but before processing any resources.  For S3 actions where i'm not looking at the bucket policy or even for EC2 creations, I could see waiting just 10-15 seconds.  That should be plenty of time to give these tools to apply most settings.  Sure, it's not perfect, but in my opinion, it's better than waiting an hour or more or having to wire up our own queue/delay mechanism via custom lambda actions or something.

In brainstorming this with a coworker, it seemed you could support this with minimal cost by wiring up the event rule to a step function with a single `wait` step, then forward the event off to the lambda.  This would cause the wait to happen before you even receive the event, but should be minimal cost and wouldn't actually need to change how policy executions are run.  It would simply be the provisioning side of the `mode: cloudtrail` policies.  It should also be able to be done by creating a single step function and just modifying the event rule target to include the amount of time to sleep and the final destination to forward the message.  Then there's minimal overhead and minimal infrastructure to have to "clean up" (mugc.py).

Thoughts?
I agree that while the mark for op/tagging approach is effective, it is a bit painful to have to wait until the next execution or even to continually be running a sweeper policy to begin with.

Another, potentially simpler option would be to use time.sleep and just pass a delay value to mu when creating the lambda. Though then you are really time bound by the 15 minute execution window. If the community thinks it is a good idea, I have a coworker who has implemented it in a few POCs and could ask them to submit a PR. 
Yea, to add to @john's point, we did a POC with a new execution mode named as: `delayed-cloudtrail` that would only get used for cases like the one mentioned above. Policy sample:
```
    base_policy = {
                      'name': 'delay',
                      'resource': 'sqs',
                      'mode': {
                          'type': 'delayed-cloudtrail',
                          'delay': 30,
                          'role': 'arn:aws:iam::{account_id}:role/FooBar',
                          'events': [{
                              'source': 'sqs.amazonaws.com',
                              'event': 'CreateQueue',
                              'ids': 'requestParameters.queueName'
                          }],
                      }
                  }
```


Yeah. I agree a simple `sleep` between the `conditions` evaluation and beginning the actual policy execution would be sufficient.  We were just brainstorming ideas of how to support an arbitrary timeframe with minimal cost, because if you sleep inside a lambda, you're paying for that execution time.  It really depends on how many executions might get triggered, etc.  But as a data point, we did a cost estimate of 1,000,000 lambda executions with 512MB size and a 60 second execution time and that would cost roughly $500.  Something similar with a step function and a 60-second wait step would cost more like $50, if i remember right.  And the step function option gives you an arbitrary delay that can go beyond the 15 minute lambda limit.  Ultimately, with the realistic number of invocations we're talking about here it's probably not worth the added complexity.  One interesting thing to note, however, is if a state machine was supported, you could get super fancy and have the state machine sit in a loop waiting until a different event was received OR some upper time limit happened...  but again, more things to get wired up properly.

At the very least, I'm thinking a `delay` option that just sleeps is sufficient.  I was more curious if this would be something allowed to be merged, given @kapilt 's previous comment here... though that was a while ago :) 
yeah, this has come up a few times before and I've been a bit resistant because, once you see failure modes this becomes quite unpredictable to reason about on remediation. that said, I've come around that its a usability exercise and the real question is about to enable it safely. the concern is a simple delay is just a time bomb during an outage or transient that enables a cascade, ie this is inherently dealing with races by rolling the dice, instead of actually dealing with the races.

the alternative is using of the other alternative custodian modes, say config, or per previous tag workflow/poll periodic to handle this post provisioning quiescence. 
I've been thinking about what kinds of failures you might be referring to that could cause a time bomb ... i suppose by imposing an intrinsic delay inside the lambda, that would increase the concurrency rate of the lambda.  It's pretty easy to write custodian policies that storm yourself ... I certainly did. Then, instead of policies running in 1-2 seconds now they might take 30-60 seconds... so you have potentially a huge number of simultaneous invocations, which in itself could cause problems.

This would be one benefit to using a step function to wait instead ... as far as I can see, you're just delaying execution slightly just as if cloudwatch events were running a little slow.  Using an express workflow gives you over 100,000 invocations per second ... but a max of 5 minute execution time.  I would argue this is plenty for this particular situation.  If you want something longer than that, you're probably just going to use hourly polling.

The problem with using many of the alternatives, like config, is that you end up going back to a polling model and thereby greatly delay your action.  The problem with the race condition here is that there's no way to determine when the race is over.  There's no signal to let us know that configuration is now complete.  So the only thing you can do is wait for some period where your confidence that things _should be_ complete is high enough to satisfy your needs.

In the situation here you've already got about a 3-6 second delay (in normal situations) just from the event bus infrastructure.  For most S3 bucket creations or delayed tagging on EC2 instances, you're really only looking for 5-10 seconds of delay to ensure those settings are applied.  Bucket policies seem to be another story because (at least with cloudformation) it creates the bucket first, then creates the bucket policy, then applies it to the bucket. My example above showed an extra 20 second delay.

And I'll point out that we're not only talking about tag compliance here.  We're talking about things like encryption on S3 buckets that ideally should be set immediately to prevent objects being written in that first hour without encryption.

So, thoughts here on how to actually solve this in a way that doesn't cause time bombs?  You can see I'm kind of a fan of the step function option since it's relatively simple.  Other folks here seem to have already just added a basic sleep function and since they haven't chimed in saying "don't do that" I assume it's been working okay.  That might be greatly dependent on how well policies are designed/written, however.
Interestingly, I looked at AWS Config and how it seems to record changes.  Looks like at least for S3 buckets, it records changes around 2 minutes after an initial API call is made:
```
04/13/2020 14:55:38.000 -0700 CreateBucket
04/13/2020 14:55:39.000 -0700 PutBucketEncryption
04/13/2020 14:55:39.000 -0700 PutBucketTagging
04/13/2020 14:55:40.000 -0700 PutBucketLifecycle
04/13/2020 14:55:40.000 -0700 PutBucketLogging
04/13/2020 14:55:41.000 -0700 PutBucketVersioning
04/13/2020 14:55:41.000 -0700 PutBucketPublicAccessBlock
04/13/2020 14:56:04.000 -0700 PutBucketPolicy
           14:57:20.000 -0700 ConfigRecorded
04/13/2020 16:10:02.000 -0700 PutBucketPolicy
           16:11:40.000 -0700 ConfigRecorded
04/13/2020 16:14:20.000 -0700 PutBucketPolicy
04/13/2020 16:15:48.000 -0700 PutBucketPolicy
           16:16:33.000 -0700 ConfigRecorded
```
So even if you use AWS Config, you're really just letting them make the decision on how long to wait before declaring the race condition over, which seems to be 90-120 seconds in these examples.

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -600,6 +600,8 @@ class CloudTrailMode(LambdaMode):
 
     schema = utils.type_schema(
         'cloudtrail',
+        delay={'type': 'integer',
+               'description': 'sleep for delay seconds before processing an event'},
         events={'type': 'array', 'items': {
             'oneOf': [
                 {'type': 'string'},
@@ -629,6 +631,13 @@ def validate(self):
                     "resource:%s does not support cloudtrail mode policies" % (
                         self.policy.resource_type))
 
+    def resolve_resources(self, event):
+        # override to enable delay before fetching resources
+        delay = self.policy.data.get('mode', {}).get('delay')
+        if delay:
+            time.sleep(delay)
+        return super().resolve_resources(event)
+
 
 @execution.register('ec2-instance-state')
 class EC2InstanceState(LambdaMode):


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -229,6 +229,35 @@ def test_phd_mode(self):
              'source': ['aws.health']}
         )
 
+    def test_cloudtrail_delay(self):
+        p = self.load_policy({
+            'name': 'aws-account',
+            'resource': 'aws.account',
+            'mode': {
+                'type': 'cloudtrail',
+                'delay': 32,
+                'role': 'CustodianRole',
+                'events': ['RunInstances']}})
+        from c7n import policy
+
+        class time:
+
+            invokes = []
+
+            @classmethod
+            def sleep(cls, duration):
+                cls.invokes.append(duration)
+
+        self.patch(policy, 'time', time)
+        trail_mode = p.get_execution_mode()
+        results = trail_mode.run({
+            'detail': {
+                'eventSource': 'ec2.amazonaws.com',
+                'eventName': 'RunInstances'}},
+            None)
+        self.assertEqual(len(results), 0)
+        self.assertEqual(time.invokes, [32])
+
     def test_user_pattern_merge(self):
         p = self.load_policy({
             'name': 'ec2-retire',