Repository: cloud-custodian/cloud-custodian

Issue:
Set VPC Flow log error - "state" attribute
I am trying to design a policy to enable VPC flow logs for all VPC that does not have it enabled. The policy is:
```yaml
policies:
  - name: enable-vpc-flow-logs
    resource: vpc
    comments: |
      Enables VPC flow logs
    mode:
      execution-options:
        log_group: /aws/enable-vpc-flow-logs/resources
      role: arn:aws:iam::{account_id}:role/CloudCustodianExecutionRole
      type: periodic
      schedule: "rate(2 minutes)"
    filters:
      - type: flow-logs
        enabled: false
    actions:
      - type: set-flow-log
        DeliverLogsPermissionArn: arn:aws:iam::${ACCOUNT}:role/role
        LogGroupName: /custodian/vpc/flowlogs/
        TrafficType: ALL
        state: true
```

However i am getting the following error, not sure why even when i am passing the state information:
```
START RequestId: bd7b329c-d677-11e8-8925-01840da49e11 Version: $LATEST
[INFO]	2018-10-23T03:57:20.426Z	bd7b329c-d677-11e8-8925-01840da49e11	Processing event
{
    "account": "[sanitized]",
    "region": "us-east-1",
    "detail": {},
    "detail-type": "Scheduled Event",
    "source": "aws.events",
    "version": "0",
    "time": "2018-10-23T03:56:45Z",
    "debug": true,
    "id": "a83eb09d-de5e-2155-62ef-24490c335d43",
    "resources": [
        "arn:aws:events:us-east-1:8[sanitized]:rule/custodian-enable-vpc-flow-logs"
    ]
}

[DEBUG]	2018-10-23T03:57:20.427Z	bd7b329c-d677-11e8-8925-01840da49e11	Storing output with <CloudWatchLogOutput to group:/aws/enable-vpc-flow-logs/resources stream:enable-vpc-flow-logs>
[DEBUG]	2018-10-23T03:57:20.899Z	bd7b329c-d677-11e8-8925-01840da49e11	Running policy enable-vpc-flow-logs resource: vpc region:us-east-1 c7n:0.8.31.2
[DEBUG]	2018-10-23T03:57:21.312Z	bd7b329c-d677-11e8-8925-01840da49e11	Filtered from 1 to 1 vpc
[INFO]	2018-10-23T03:57:21.312Z	bd7b329c-d677-11e8-8925-01840da49e11	policy: enable-vpc-flow-logs resource:vpc region:us-east-1 count:1 time:0.25
[DEBUG]	2018-10-23T03:57:21.321Z	bd7b329c-d677-11e8-8925-01840da49e11	metric:ResourceCount Count:1 policy:enable-vpc-flow-logs restype:vpc scope:policy
[DEBUG]	2018-10-23T03:57:21.321Z	bd7b329c-d677-11e8-8925-01840da49e11	metric:PolicyException Count:1 policy:enable-vpc-flow-logs restype:vpc
[DEBUG]	2018-10-23T03:57:21.321Z	bd7b329c-d677-11e8-8925-01840da49e11	metric:ApiCalls Count:0 policy:enable-vpc-flow-logs restype:vpc
[ERROR]	2018-10-23T03:57:21.522Z	bd7b329c-d677-11e8-8925-01840da49e11	Error while executing policy
Traceback (most recent call last):
File "/var/task/c7n/policy.py", line 273, in run
results = a.process(resources)
File "/var/task/c7n/resources/vpc.py", line 1810, in process
if not self.state:
AttributeError: 'CreateFlowLogs' object has no attribute 'state'
'CreateFlowLogs' object has no attribute 'state': AttributeError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 91, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 834, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 516, in run
return PullMode.run(self)
File "/var/task/c7n/policy.py", line 273, in run
results = a.process(resources)
File "/var/task/c7n/resources/vpc.py", line 1810, in process
if not self.state:
AttributeError: 'CreateFlowLogs' object has no attribute 'state'

END RequestId: bd7b329c-d677-11e8-8925-01840da49e11
```


Discussion:
Thanks for the bug report, the state attr is being set during the validation of the action, which is not expected/desired behavior

instantiating attributes in validate presupposes that validate will get run on every execution which is not a valid assumption
in set-flow-log action:

```python
    def validate(self):
        self.state = self.data.get('state', True)
        if self.state:
            if not self.data.get('DeliverLogsPermissionArn'):
                raise PolicyValidationError(
                    'DeliverLogsPermissionArn required when '
                    'creating flow-logs on %s' % (self.manager.data,))
            if not self.data.get('LogGroupName'):
                raise ValueError(
                    'LogGroupName required when creating flow-logs on %s' % (
                        self.manager.data))
        return self
```
Thanks for looking into it. Is there any timeframe when you think the bug would be fixed and released. Actually we were planning to use these feature for some remediation across board and thus are in a time crunch. OR, is there a workaround?

@mukavik for now, you can run this without running in periodic mode for it to work
Yes, one time run of the policy does work. I am assuming it would also work on CloudTrail triggers. Appreciate the help. Will keep an eye out for the bug fix. 
Just a quick question w.r.t . I am getting another error now, the traceback follows:
2018-10-29 14:35:46,629: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/site-packages/c7n/policy.py", line 273, in run
 results = a.process(resources)
 File "/usr/local/lib/python2.7/site-packages/c7n/resources/vpc.py", line 1818, in process
 results = client.create_flow_logs(**params)
 File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
 return self._make_api_call(operation_name, kwargs)
 File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call
 raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidVpcId.NotFound) when calling the CreateFlowLogs operation: Unknown resource vpc-28640553.

The VPC is there with the given ID. Not sure what is causing this?
@thisisshi figured the above error out. I have to run the policy one at a time against each region. If i am running it against all region some how it is searching for the filtered VPC ID while CreateFlow log function and thus cannot find the VPC ID in other region and throwing an error. Is this also a bug or am i doing something wrong in the policy ?
which version of c7n are you using here?
0.8.31.2

There was a session caching fix that resolved this.
@kapilt @mukavik - Im still experiencing the bug when running vpc using periodic. Has that bug been fixed?
@rheak what version of custodian?
@kapilt - 0.8.42.1
Hi @kapilt  Im still having issues executing this policy in Lambda. Here are the error logs from Lambda.

**[ERROR]	Error while executing policy**
`Traceback (most recent call last):
File "/var/task/c7n/handler.py", line 165, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 903, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 520, in run
return PullMode.run(self)
File "/var/task/c7n/policy.py", line 264, in run
results = a.process(resources)
File "/var/task/c7n/resources/vpc.py", line 1888, in process
if not self.state:
AttributeError: 'CreateFlowLogs' object has no attribute 'state'`

**[ERROR] AttributeError: 'CreateFlowLogs' object has no attribute 'state'**
`Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 903, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 520, in run
    return PullMode.run(self)
  File "/var/task/c7n/policy.py", line 264, in run
    results = a.process(resources)
  File "/var/task/c7n/resources/vpc.py", line 1888, in process
    if not self.state:`
I think I see the issue, pr coming shortly

Resolving patch:
diff --git a/c7n/handler.py b/c7n/handler.py
--- a/c7n/handler.py
+++ b/c7n/handler.py
@@ -162,6 +162,9 @@ def dispatch_event(event, context):
     if policies:
         for p in policies:
             try:
+                # validation provides for an initialization point for
+                # some filters/actions.
+                p.validate()
                 p.push(event, context)
             except Exception:
                 log.exception("error during policy execution")


Test patch:
diff --git a/tests/test_handler.py b/tests/test_handler.py
--- a/tests/test_handler.py
+++ b/tests/test_handler.py
@@ -131,11 +131,16 @@ def cleanup():
         self.change_environment(C7N_OUTPUT_DIR=self.run_dir)
 
         policy_execution = []
+        validation_called = []
+
+        def validate(self):
+            validation_called.append(True)
 
         def push(self, event, context):
             policy_execution.append((event, context))
 
         self.patch(Policy, "push", push)
+        self.patch(Policy, "validate", validate)
 
         from c7n import handler
 
@@ -161,6 +166,7 @@ def push(self, event, context):
         )
         self.assertEqual(handler.dispatch_event({"detail": {}}, None), True)
         self.assertEqual(policy_execution, [({"detail": {}, "debug": True}, None)])
+        self.assertEqual(validation_called, [True])
 
         config = handler.Config.empty()
         self.assertEqual(config.assume_role, None)