Repository: cloud-custodian/cloud-custodian

Issue:
Skipping failed operation - Allow invoke action for an event having errorCode
Our use case:  
Send a notification whenever a client application uses a Customer Master Key (KMS) that has been marked as **"pending deletion"**.

Whenever a client (for e.g. aws cli kms encrypt....) uses the key that is marked for deletion, a cloudTrail/CloudWatch event is generated with "errorCode". But, as per the c7n/handler.py (line 54), the lambda function returns if it finds the errorCode in the event and therefore no action is invoked.  

Is there a easy way to override this behavior? Thanks.
periodic mode, option to disable provider automatic retries on error
Hi,

on a busy account, with lots of instances, I am running a snaphost policy every days. 

Once in a while, I am getting this error : 

```
An error occurred (InternalError) when calling the CreateSnapshot operation (reached max retries: 4): An internal error has occurred: ClientError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 105, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 832, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 510, in run
return PullMode.run(self)
File "/var/task/c7n/policy.py", line 264, in run
results = a.process(resources)
File "/var/task/c7n/resources/ec2.py", line 1229, in process
raise f.exception()
ClientError: An error occurred (InternalError) when calling the CreateSnapshot operation (reached max retries: 4): An internal error has occurred
```

Looking at my snapshots, I am having 4 times more snapshot than I should have (4k instead of 1k). 



c7n/handler.py  - metrics_enabled check?
` 
    if 'metrics_enabled' in options_overrides and isinstance(
            options_overrides['metrics_enabled'], bool):
        options_overrides['metrics_enabled'] = 'aws'`


The above check  above for metrics_enablement in 0.8.33.0 - if metrics_enabled is set in the config.json, is the if statement always supposed to evaulate to true, even if metrics_enabled is set to False?


Discussion:
The error catch and skip there because there is no control plane resource change on an error. Data plane usage (like s3 put/get, or kms encrypt/decrypt) is typically a vastly differently cardinality then control plane activity which custodian is commonly used for. There isn't anything inherently at issue with that, ie. we can provide for explicit configuration to allow for error event policies, but the question would be what you want to do with such an event? For notifications, without messaging aggregation downstream, you'll potentially be sending a lot of messages to users.
Per se, it is not an issue, but preventing us to implement the mandatory compliance requirement as stated above. Our use cases cover both control & data plane cases. we are able to achieve the other data plane use cases when there is no error code (like for e.g. action/notification of cloudfront usage without https).

Though we can implement this requirement using custom lambda or cloudWatch rules, it would deviate from the other policies we have written using c7n. This may add up additional maintenance, testing & deployment overheads for those custom implementation. 
I think there will not be lot of messages sent to users, like other filter & match pattern, notifications would be sent only for the matching filters (in this case, action/notification is triggered only for filters matching specific errorMessage /errorCode ). Therefore I think there will not be lot of messages to users. I guess this the same principle for other data plane use case too.
As I am new to c7n, you may please assess the merit of this requirement and advise.
contributions welcome
I'm not sure what the question is. InternalError denotes a ec2 control plane error. Custodian will retry on throttle exceptions which happen before the snapshot api machinery gets invoked (ie throttles happen on frontend of aws api machinery), unknown errors stop as in this case. Wrt to instance snapshots, its a snapshot per attached volume.
Hi, 

thank you for your reply. 

Today my report is showing that I have 4k snapshots instead of the usual 1K. 

Looking at the lambda logs, it started over 4 times with the same error. 

do you have any idea on what can cause the error and why I have 4K snapshots ?

wrt to the why on additional snapshots, doing policies in lambdas, needs to be cognizant of builtin lambda retry behavior https://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html .. wrt to the why on the error, it sounds like the account is not healthy, if its throwing regular internal exception errors, i'd check cloudtrail to see how common this is (along with rate errors) and contact your support channels about internal errors on api or app teams to reduce the bad api call behavior.

i'm wondering if we should do a top level error catch on periodic mode policies to handle this to avoid the builtin lambda retries, but given a primary mechanism for monitoring lambda policies is checking against the lambda error metrics, it has some downside. we do publish a separate error metric since 0.8.33.x  so we could direct folks to that.
for gcp the execution mode configuration lets one directly specify the retry behavior, not sure if this configurable directly in azure or what the retry behavior is there.
good catch, technically metrics_enabled is not a boolean anymore, this was compatibility code for folks that were explicitly setting an enabled boolean. Its not really clear there's a use case for setting explicitly to false, given that would be the default if its not set at all. Nonetheless your correct it should be checking the value before turning it on.
:) fwiw - when i upgraded from 0.8.31.0 to 0.8.33.0, metrics_enabled was added to the config.json as false. I only picked it up as the role that the lambda executes on, doesn't have the IAM permission to PutMetric!

thanks for the quick confirmation.

Resolving patch:
diff --git a/c7n/handler.py b/c7n/handler.py
--- a/c7n/handler.py
+++ b/c7n/handler.py
@@ -36,72 +36,136 @@
 logging.getLogger('urllib3').setLevel(logging.WARNING)
 log = logging.getLogger('custodian.lambda')
 
-account_id = None
+#
+# Env serverless specific configuration options, these are part of "public" interface
+#
+# We default to skipping events which denote they have errors
+C7N_SKIP_EVTERR = os.environ.get('C7N_SKIP_ERR_EVENT', 'yes') == 'yes' and True or False
 
-# On cold start load all resources, requires a pythonpath directory scan
-if 'AWS_EXECUTION_ENV' in os.environ:
-    load_resources()
+# We default to logging the full event that triggered lambda execution
+C7N_DEBUG_EVENT = os.environ.get('C7N_DEBUG_EVENT', 'yes') == 'yes' and True or False
 
+# We default to not catching policy errors in lambda, which will lead to retry behavior
+C7N_CATCH_ERR = os.environ.get('C7N_CATCH_ERR', 'no').strip().lower() == 'yes' and True or False
 
-def dispatch_event(event, context):
+#
+# Internal global variables
+#
+# Default global cache of execution account id for initial configuration setup.
+account_id = None
 
-    global account_id
+# config.json policy data dict
+policy_config = None
 
-    error = event.get('detail', {}).get('errorCode')
-    if error:
-        log.debug("Skipping failed operation: %s" % error)
-        return
-
-    event['debug'] = True
-    if event['debug']:
-        log.info("Processing event\n %s", format_event(event))
+# On cold start load all resources, requires a pythonpath directory scan
+if 'AWS_EXECUTION_ENV' in os.environ:
+    load_resources()
 
-    # Policies file should always be valid in lambda so do loading naively
-    with open('config.json') as f:
-        policy_config = json.load(f)
 
-    if not policy_config or not policy_config.get('policies'):
-        return False
+def get_local_output_dir():
+    """Create a local output directory per execution.
 
-    # Initialize output directory, we've seen occassional perm issues with
-    # lambda on temp directory and changing unix execution users, so
-    # use a per execution temp space.
-    output_dir = os.environ.get(
-        'C7N_OUTPUT_DIR',
-        '/tmp/' + str(uuid.uuid4()))
+    We've seen occassional (1/100000) perm issues with lambda on temp
+    directory and changing unix execution users (2015-2018), so use a
+    per execution temp space. With firecracker lambdas this may be outdated.
+    """
+    output_dir = os.environ.get('C7N_OUTPUT_DIR', '/tmp/' + str(uuid.uuid4()))
     if not os.path.exists(output_dir):
         try:
             os.mkdir(output_dir)
         except OSError as error:
             log.warning("Unable to make output directory: {}".format(error))
+    return output_dir
+
 
-    # TODO. This enshrines an assumption of a single policy per lambda.
-    options_overrides = policy_config[
-        'policies'][0].get('mode', {}).get('execution-options', {})
+def init_config(policy_config):
+    """Get policy lambda execution configuration.
+
+    cli parameters are serialized into the policy lambda config,
+    we merge those with any policy specific execution options.
+
+    --assume role and -s output directory get special handling, as
+    to disambiguate any cli context.
+
+    account id is sourced from the config options or from api call
+    and cached as a global
+    """
+    global account_id
+
+    exec_options = policy_config.get('execution-options', {})
+
+    # Remove some configuration options that don't make sense to translate from
+    # cli to lambda automatically.
+    #  - assume role on cli doesn't translate, it is the default lambda role and
+    #    used to provision the lambda.
+    #  - profile doesnt translate to lambda its `home` dir setup dependent
+    #  - dryrun doesn't translate (and shouldn't be present)
+    #  - region doesn't translate from cli (the lambda is bound to a region), and
+    #    on the cli represents the region the lambda is provisioned in.
+    for k in ('assume_role', 'profile', 'region', 'dryrun', 'cache'):
+        exec_options.pop(k, None)
+
+    # a cli local directory doesn't translate to lambda
+    if not exec_options.get('output_dir', '').startswith('s3'):
+        exec_options['output_dir'] = get_local_output_dir()
+
+    # we can source account id from the cli parameters to avoid the sts call
+    if exec_options.get('account_id'):
+        account_id = exec_options['account_id']
+
+    # merge with policy specific configuration
+    exec_options.update(
+        policy_config['policies'][0].get('mode', {}).get('execution-options', {}))
 
     # if using assume role in lambda ensure that the correct
     # execution account is captured in options.
-    if 'assume_role' in options_overrides:
-        account_id = options_overrides['assume_role'].split(':')[4]
+    if 'assume_role' in exec_options:
+        account_id = exec_options['assume_role'].split(':')[4]
     elif account_id is None:
         session = boto3.Session()
         account_id = get_account_id_from_sts(session)
+    exec_options['account_id'] = account_id
 
     # Historical compatibility with manually set execution options
     # previously this was a boolean, its now a string value with the
     # boolean flag triggering a string value of 'aws'
-    if 'metrics_enabled' in options_overrides and isinstance(
-            options_overrides['metrics_enabled'], bool):
-        options_overrides['metrics_enabled'] = 'aws'
+    if 'metrics_enabled' in exec_options \
+       and isinstance(exec_options['metrics_enabled'], bool) \
+       and exec_options['metrics_enabled']:
+        exec_options['metrics_enabled'] = 'aws'
+
+    return Config.empty(**exec_options)
+
 
-    options_overrides['account_id'] = account_id
+def dispatch_event(event, context):
+    error = event.get('detail', {}).get('errorCode')
+    if error and C7N_SKIP_EVTERR:
+        log.debug("Skipping failed operation: %s" % error)
+        return
+
+    if C7N_DEBUG_EVENT:
+        event['debug'] = True
+        log.info("Processing event\n %s", format_event(event))
+
+    # Policies file should always be valid in lambda so do loading naively
+    global policy_config
+    if policy_config is None:
+        with open('config.json') as f:
+            policy_config = json.load(f)
+
+    if not policy_config or not policy_config.get('policies'):
+        return False
 
-    if 'output_dir' not in options_overrides:
-        options_overrides['output_dir'] = output_dir
-    options = Config.empty(**options_overrides)
+    options = init_config(policy_config)
 
     policies = PolicyCollection.from_data(policy_config, options)
     if policies:
         for p in policies:
-            p.push(event, context)
+            try:
+                p.push(event, context)
+            except Exception:
+                log.exception("error during policy execution")
+                if C7N_CATCH_ERR:
+                    continue
+                raise
     return True
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -866,7 +866,8 @@ def get_events(self, session_factory):
     def get_archive(self):
         self.archive.add_contents(
             'config.json', json.dumps(
-                {'policies': [self.policy.data]}, indent=2))
+                {'execution-options': dict(self.policy.options),
+                 'policies': [self.policy.data]}, indent=2))
         self.archive.add_contents('custodian_policy.py', PolicyHandlerTemplate)
         self.archive.close()
         return self.archive


Test patch:
diff --git a/c7n/testing.py b/c7n/testing.py
--- a/c7n/testing.py
+++ b/c7n/testing.py
@@ -162,7 +162,8 @@ def cleanup_env():
             os.environ.update(original_environ)
 
         os.environ.clear()
-        for key, value in kwargs.items():
+
+        for key, value in list(kwargs.items()):
             if value is None:
                 del (kwargs[key])
         os.environ.update(kwargs)
diff --git a/tests/test_handler.py b/tests/test_handler.py
--- a/tests/test_handler.py
+++ b/tests/test_handler.py
@@ -15,14 +15,108 @@
 
 import json
 import logging
+import mock
 import os
 
 from .common import BaseTest
+from c7n.exceptions import PolicyExecutionError
 from c7n.policy import Policy
+from c7n import handler
 
 
 class HandleTest(BaseTest):
 
+    def test_get_local_output_dir(self):
+        temp_dir = self.get_temp_dir()
+        os.rmdir(temp_dir)
+        self.change_environment(C7N_OUTPUT_DIR=temp_dir)
+        self.assertEqual(
+            handler.get_local_output_dir(), temp_dir)
+
+    def test_init_config_exec_option_merge(self):
+        policy_config = {
+            'execution-options': {
+                'region': 'us-east-1',
+                'assume_role': 'arn:::',
+                'profile': 'dev',
+                'tracer': 'xray',
+                'account_id': '004',
+                'dryrun': True,
+                'cache': '/foobar.cache'},
+            'policies': [
+                {'mode': {
+                    'type': 'period',
+                    'schedule': "rate(1 minute)",
+                    'execution-options': {
+                        'metrics_enabled': True,
+                        'assume_role': 'arn::::007:foo',
+                        'output_dir': 's3://mybucket/output'}},
+                 'resource': 'aws.ec2',
+                 'name': 'check-dev'}
+            ]}
+        self.assertEqual(
+            dict(handler.init_config(policy_config)),
+            {'assume_role': 'arn::::007:foo',
+             'metrics_enabled': 'aws',
+             'tracer': 'xray',
+             'account_id': '007',
+             'region': 'us-east-1',
+             'output_dir': 's3://mybucket/output',
+
+             # defaults
+             'external_id': None,
+             'dryrun': False,
+             'profile': None,
+             'authorization_file': None,
+             'cache': '',
+             'regions': (),
+             'cache_period': 0,
+             'log_group': None})
+
+    def test_dispatch_log_event(self):
+        self.patch(handler, 'policy_config', {'policies': []})
+        output = self.capture_logging('custodian.lambda', level=logging.INFO)
+        self.change_environment(C7N_DEBUG_EVENT=None)
+        handler.dispatch_event({'detail': {'resource': 'xyz'}}, {})
+        self.assertTrue('xyz' in output.getvalue())
+
+        self.patch(handler, 'C7N_DEBUG_EVENT', False)
+        handler.dispatch_event({'detail': {'resource': 'abc'}}, {})
+        self.assertFalse('abc' in output.getvalue())
+
+    @mock.patch('c7n.handler.PolicyCollection')
+    def test_dispatch_err_event(self, mock_collection):
+        self.patch(handler, 'policy_config', {
+            'execution-options': {'output_dir': 's3://xyz', 'account_id': '004'},
+            'policies': [{'resource': 'ec2', 'name': 'xyz'}]})
+        mock_collection.from_data.return_value = []
+        output = self.capture_logging('custodian.lambda', level=logging.DEBUG)
+        handler.dispatch_event({'detail': {'errorCode': 'unauthorized'}}, None)
+        self.assertTrue('Skipping failed operation: unauthorized' in output.getvalue())
+        self.patch(handler, 'C7N_SKIP_EVTERR', False)
+        handler.dispatch_event({'detail': {'errorCode': 'foi'}}, None)
+        self.assertFalse('Skipping failed operation: foi' in output.getvalue())
+        mock_collection.from_data.assert_called_once()
+
+    @mock.patch('c7n.handler.PolicyCollection')
+    def test_dispatch_err_handle(self, mock_collection):
+        self.patch(handler, 'policy_config', {
+            'execution-options': {'output_dir': 's3://xyz', 'account_id': '004'},
+            'policies': [{'resource': 'ec2', 'name': 'xyz'}]})
+        output = self.capture_logging('custodian.lambda', level=logging.WARNING)
+        pmock = mock.MagicMock()
+        pmock.push.side_effect = PolicyExecutionError("foo")
+        mock_collection.from_data.return_value = [pmock]
+
+        self.assertRaises(
+            PolicyExecutionError,
+            handler.dispatch_event,
+            {'detail': {'xyz': 'oui'}}, None)
+
+        self.patch(handler, 'C7N_CATCH_ERR', True)
+        handler.dispatch_event({'detail': {'xyz': 'oui'}}, None)
+        self.assertEqual(output.getvalue().count('error during'), 2)
+
     def test_handler(self):
         level = logging.root.level
         botocore_level = logging.getLogger("botocore").level