Repository: cloud-custodian/cloud-custodian

Issue:
aws event mode should support more event selectivity options
The hub-finding mode should support additional filtering within the generated CWE event by enabling filtering on ~~edited~~ see comments from me on 9/7/2019

The current behavior is that every custodian policy based on hub-finding mode gets invoked for every finding event securityhub generates, which is not an efficient use of resources when often the policy's purpose is targeted to a specific resource type or finding type.  I notice this the most when the Security Hub CIS checks report findings with compliance_status = PASSED for lots of rules every 12 hours.


Discussion:
generalizing, this isn't really particular to hub finding mode, and its not jmespath key filtering. its cwe event pattern language.

of course the event and resources can be filtered at the policy, at the additional cost of a lambda execution, which is generally considered negligible. if you have math that shows otherwise i'd be interested.

the issue is that security hub doesn't really support that many resources natively, perhaps for those  few resources it does suppot we could auto insert additional pattern language here, but there is no defined vocabulary around other, and custodian will want to be used for findings other than the ones it generates. wrt to user defined patterns, i think its tbd, and also independent of hub-mode.

also fyi, i'm planning on refactoring the existing hub implementation into a dedicated module, currently concerns feel pretty littered throughout the codebase.
i was wrong to say jmes key filtering, instead what i mean is, to support
additional cwe event pattern filtering by allowing the policy writer to
specify the pattern for c7n to insert into the cwe pattern it generates.
Make sense now?
Example for "detail-type": "Security Hub Findings - Custom Action"

add to the cwe pattern a filter on "detail.actionName" equal to the name of the custom action.  Currently all hub actions deployed by cloud custodian get invoked any time their is a custom action action invoked which had been setup by Cloud Custodian.  Adding just this detail.actionName to the filter will eliminate the other invocations.  I noticed then when a custom action on an ebs-snapshot resulted in another custom action configured for ec2 resources failed when it tryied to do a describe-instances call on the snapshot-id which had been passed in.
When   "detail-type": "Security Hub Findings - Imported"
my ask is to add the CWE pattern of details.findings[0].ProductFields.resource with value equal to the c7n policies's resource type (ec2, ebs-snapshot, iam-user, etc)
people will use custodian to process findings generated by other products, we can't insert in something that will only let it respond to its own product fields for mode: hub-finding

for mode: hub-action we already insert into the event pattern the specific custom action resource, so its only responding to events targeted to the action.

@kapilt Does some combination of https://aws.amazon.com/about-aws/whats-new/2020/02/introducing-content-filtering-amazon-eventbridge/ plus making the my ask opt-in enable this to be done?
thanks for highlighting this, it looks pretty interesting indeed

https://docs.aws.amazon.com/eventbridge/latest/userguide/content-filtering-with-event-patterns.html
https://aws.amazon.com/blogs/compute/reducing-custom-code-by-using-advanced-rules-in-amazon-eventbridge/

we'll need to look at if its supported in cwe, transitioning cwe to EventBridge is likely going to involve some iam pain, and we'll need to likely support both for the foreseeable future on a transition to ensure compatibility.
Is it also worth investigating (depending on how capable the rules can be written) to look at delaying the full initialization of the resources when running in a lambda event context until after any event filters have been processed?  You'd still end up firing the lambdas, but you would only do the processing after you've determined it's an event you should pay attention to.

This would help with my particular situation where I want to trigger things upon tagging, but ignore tagging being done by cloud custodian itself.

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -46,7 +46,7 @@
 from c7n.exceptions import ClientError
 from c7n.cwe import CloudWatchEvents
 from c7n.logs_support import _timestamp_from_string
-from c7n.utils import parse_s3, local_session, get_retry
+from c7n.utils import parse_s3, local_session, get_retry, merge_dict
 
 log = logging.getLogger('custodian.serverless')
 
@@ -1115,6 +1115,8 @@ def render_event_pattern(self):
                 "Unknown lambda event source type: %s" % event_type)
         if not payload:
             return None
+        if self.data.get('pattern'):
+            payload = merge_dict(payload, self.data['pattern'])
         return json.dumps(payload)
 
     def add(self, func):
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -412,6 +412,7 @@ class LambdaMode(ServerlessExecutionMode):
             'runtime': {'enum': ['python2.7', 'python3.6',
                                  'python3.7', 'python3.8']},
             'role': {'type': 'string'},
+            'pattern': {'type': 'object', 'minProperties': 1},
             'timeout': {'type': 'number'},
             'memory': {'type': 'number'},
             'environment': {'type': 'object'},
@@ -769,7 +770,6 @@ class ConfigRuleMode(LambdaMode):
     """a lambda policy that executes as a config service rule.
         http://docs.aws.amazon.com/config/latest/APIReference/API_PutConfigRule.html
     """
-
     cfg_event = None
     schema = utils.type_schema('config-rule', rinherit=LambdaMode.schema)
 
@@ -779,6 +779,10 @@ def validate(self):
             raise PolicyValidationError(
                 "policy:%s AWS Config does not support resource-type:%s" % (
                     self.policy.name, self.policy.resource_type))
+        if self.policy.data['mode'].get('pattern'):
+            raise PolicyValidationError(
+                "policy:%s AWS Config does not support event pattern filtering" % (
+                    self.policy.name))
 
     def resolve_resources(self, event):
         source = self.policy.resource_manager.get_source('config')
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -644,3 +644,21 @@ def parse(cls, data):
 
 def get_annotation_prefix(s):
     return 'c7n:{}'.format(s)
+
+
+def merge_dict(a, b):
+    """Perform a merge of dictionaries a and b
+
+    Any subdictionaries will be recursively merged.
+    Any leaf elements in the form of a list or scalar will use the value from a
+    """
+    d = {}
+    for k, v in a.items():
+        if k not in b:
+            d[k] = v
+        elif isinstance(v, dict) and isinstance(b[k], dict):
+            d[k] = merge_dict(v, b[k])
+    for k, v in b.items():
+        if k not in d:
+            d[k] = v
+    return d


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -208,6 +208,29 @@ def test_phd_mode(self):
              'source': ['aws.health']}
         )
 
+    def test_user_pattern_merge(self):
+        p = self.load_policy({
+            'name': 'ec2-retire',
+            'resource': 'ec2',
+            'mode': {
+                'type': 'cloudtrail',
+                'pattern': {
+                    'detail': {
+                        'userIdentity': {
+                            'userName': [{'anything-but': 'deputy'}]}}},
+                'events': [{
+                    'ids': 'responseElements.subnet.subnetId',
+                    'source': 'ec2.amazonaws.com',
+                    'event': 'CreateSubnet'}]}})
+        p_lambda = PolicyLambda(p)
+        events = p_lambda.get_events(None)
+        self.assertEqual(
+            json.loads(events[0].render_event_pattern()),
+            {'detail': {'eventName': ['CreateSubnet'],
+                        'eventSource': ['ec2.amazonaws.com'],
+                        'userIdentity': {'userName': [{'anything-but': 'deputy'}]}},
+             'detail-type': ['AWS API Call via CloudTrail']})
+
     def test_cwl_subscriber(self):
         self.patch(CloudWatchLogSubscription, "iam_delay", 0.01)
         session_factory = self.replay_flight_data("test_cwl_subscriber")
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -132,6 +132,22 @@ def test_all_proxy_with_full_url(self):
 
 class UtilTest(BaseTest):
 
+    def test_merge_dict(self):
+        a = {'detail': {'eventName': ['CreateSubnet'],
+                    'eventSource': ['ec2.amazonaws.com']},
+             'detail-type': ['AWS API Call via CloudTrail']}
+        b = {'detail': {'userIdentity': {
+            'userName': [{'anything-but': 'deputy'}]}}}
+        self.assertEqual(
+            utils.merge_dict(a, b),
+            {'detail-type': ['AWS API Call via CloudTrail'],
+             'detail': {
+                 'eventName': ['CreateSubnet'],
+                 'eventSource': ['ec2.amazonaws.com'],
+                 'userIdentity': {
+                     'userName': [
+                         {'anything-but': 'deputy'}]}}})
+
     def test_local_session_region(self):
         policies = [
             self.load_policy(