Repository: StackStorm/st2

Issue:
Allow multiple evaluations of the same field in a rule
When writing rules, gui or CLI, if I try to evaluate {{ trigger.body }} with a contains and an additional ncontains, only one of the criteria is saved.  It would be VERY handy to be able to do 2 or more criteria evals on the same field.  



Discussion:
@lmarty You can achieve the same by writing another rule. This is intentionally designed that way so rules are simple to grok. Nevertheless, good feedback. 

@lmarty do you mean something like `OR` operator (by default, if you have multiple conditions, it behaves as `AND`).

If you indeed mean `OR`, you can achieve that, as @lakshmi-kannan has pointed out, by using multiple rules.

If that behavior is not documented yet, or documented poorly, we should make it more explicit in the documentation.

So I need to create a workflow if I want to evaluate message bodies for multiple criteria?  That seems off to me, with the way the GUI is laid out.  We can click to add AND criteria, no mention of limits there or in the docs.  
  @Kami, I am not looking for an OR, definitely an AND. For example, an Amazon SQS message triggers an event.  I first want to evaluate trigger.body of that event and make sure it is the type of event I am looking for, then, I want to check to make sure it did not originate from an expected source.  ie: Message contains text X AND ncontains text Y.  Both of those data points being in trigger.body.  If this is truly by design, it should definitely be called out in the docs/tooltips/help/somewhere.  

@lmarty Looks like I misunderstood you then.

If you are looking for AND, this should work out of the box just fine - you simple need to add multiple criteria expressions.

In your case, it would look something like that:

``` yaml
...
criteria:
    trigger.body:
        type: "contains"
        pattern : "X"
    trigger.body:
        type: "ncontains"
        pattern : "Y"
...
```

You can also add more criteria expressions if needed.

Thanks Kami!  Unfortunately, when I do as described above, via the GUI or the CLI, it only takes and displays the last criteria element.  So in the above case, the "contains" criteria would be dropped and the "ncontains" would be the only active element.  I'm on st2 1.4, and I've rebuilt the box fresh to validate the behavior.   

@lmarty You are welcome.

This actually sounds like a WebUI bug. It should work just fine if you create rule using the CLI (st2 rule create path/to/rule/file.yaml and then st2 rule get <rule id> to verify the criteria).

/cc @enykeev 

@lmarty Actually, sorry, I was wrong (I just noticed it now).

Now I see why the example I posted above won't work - `criteria` is a dictionary which means you can't reference the same key / field (in this case `trigger.body`) twice.

That's an unfortunate limitation and one of the workarounds you can probably use is a regex operator and regular expression which fits your criteria (e.g. something along the lines of `^(?=.*contains)(?!.*notcontains).*`).

/cc @lakshmi-kannan @manasdk et. al. this seems a pretty unfortunate limitation, we should discuss how to fix it.

One option would be to allow body of a single criteria expression to be a list of dictionaries, but for backward comparability we would also allow plain dictionaries, or something along those lines.

> Now I see why the example I posted above won't work - criteria is a dictionary which means you can't reference the same key / field (in this case trigger.body) twice.

This is indeed a limitation. 

I knew I wasn't crazy.   Bummer guys! This would be a _very_ handy feature to have for us.  Keep me in the loop as you move forward.  I'll try and find a way around this with other sources of unique in my event stream in the meantime.   

My guess is that this would need some sort of schema change to do this the right way. 

Took a closer look at this today and the only reasonable approach would require a schema change of the criteria field in rules; move it from `dictionary` to `list`. This of course has right up to what we save in the DB so would require migration scripts etc. 

Any update on this? I would like to do something like the following:
```
"trigger.body.data.latitude": {
  "type": "greaterthan",
  "pattern": "30"
},
"trigger.body.data.latitude": {
  "type": "lessthan",
  "pattern": "50"
},
"trigger.body.data.longitude": {
  "type": "lessthan",
  "pattern": "-90"
},
"trigger.body.data.longitude": {
  "type": "greaterthan",
  "pattern": "-100"
},
```
Sort of a geo-fence. I get the same problem described above, only the latest key is kept.

Resolving patch:
diff --git a/st2common/st2common/operators.py b/st2common/st2common/operators.py
--- a/st2common/st2common/operators.py
+++ b/st2common/st2common/operators.py
@@ -58,8 +58,10 @@ def search(value, criteria_pattern, criteria_condition, check_function):
 
     value: the payload list to search
     condition: one of:
-      * any - return true if any items of the list match and false if none of them match
-      * all - return true if all items of the list match and false if any of them do not match
+      * any - return true if any payload items of the list match all criteria items
+      * all - return true if all payload items of the list match all criteria items
+      * all2any - return true if all payload items of the list match any criteria items
+      * any2any - return true if any payload items match any criteria items
     pattern: a dictionary of criteria to apply to each item of the list
 
     This operator has O(n) algorithmic complexity in terms of number of child patterns.
@@ -86,18 +88,20 @@ def search(value, criteria_pattern, criteria_condition, check_function):
         ]
     }
 
-    And an example usage in criteria:
+    Example #1
 
     ---
     criteria:
       trigger.fields:
         type: search
         # Controls whether this criteria has to match any or all items of the list
-        condition: any  # or all
+        condition: any  # or all or all2any or any2any
         pattern:
           # Here our context is each item of the list
           # All of these patterns have to match the item for the item to match
           # These are simply other operators applied to each item in the list
+          # "#" and text after are ignored.
+          # This allows dictionary keys to be unique but refer to the same field
           item.field_name:
             type: "equals"
             pattern: "Status"
@@ -105,59 +109,53 @@ def search(value, criteria_pattern, criteria_condition, check_function):
           item.to_value:
             type: "equals"
             pattern: "Approved"
+
+          item.field_name#1:
+            type: "greaterthan"
+            pattern: 40
+
+          item.field_name#2:
+            type: "lessthan"
+            pattern: 50
     """
+    if isinstance(value, dict):
+        value = [value]
+    payloadItemMatch = all
+    patternMatch = all
     if criteria_condition == "any":
-        # Any item of the list can match all patterns
-        rtn = any(
-            [
-                # Any payload item can match
-                all(
-                    [
-                        # Match all patterns
-                        check_function(
-                            child_criterion_k,
-                            child_criterion_v,
-                            PayloadLookup(
-                                child_payload, prefix=TRIGGER_ITEM_PAYLOAD_PREFIX
-                            ),
-                        )
-                        for child_criterion_k, child_criterion_v in six.iteritems(
-                            criteria_pattern
-                        )
-                    ]
-                )
-                for child_payload in value
-            ]
-        )
-    elif criteria_condition == "all":
-        # Every item of the list must match all patterns
-        rtn = all(
-            [
-                # All payload items must match
-                all(
-                    [
-                        # Match all patterns
-                        check_function(
-                            child_criterion_k,
-                            child_criterion_v,
-                            PayloadLookup(
-                                child_payload, prefix=TRIGGER_ITEM_PAYLOAD_PREFIX
-                            ),
-                        )
-                        for child_criterion_k, child_criterion_v in six.iteritems(
-                            criteria_pattern
-                        )
-                    ]
-                )
-                for child_payload in value
-            ]
-        )
-    else:
+        payloadItemMatch = any
+    elif criteria_condition == "all2any":
+        patternMatch = any
+    elif criteria_condition == "any2any":
+        payloadItemMatch = any
+        patternMatch = any
+    elif criteria_condition != "all":
         raise UnrecognizedConditionError(
-            "The '%s' search condition is not recognized, only 'any' "
-            "and 'all' are allowed" % criteria_condition
+            "The '%s' condition is not recognized for type search, 'any', 'all', 'any2any'"
+            " and 'all2any' are allowed" % criteria_condition
         )
 
+    rtn = payloadItemMatch(
+        [
+            # any/all payload item can match
+            patternMatch(
+                [
+                    # Match any/all patterns
+                    check_function(
+                        child_criterion_k,
+                        child_criterion_v,
+                        PayloadLookup(
+                            child_payload, prefix=TRIGGER_ITEM_PAYLOAD_PREFIX
+                        ),
+                    )
+                    for child_criterion_k, child_criterion_v in six.iteritems(
+                        criteria_pattern
+                    )
+                ]
+            )
+            for child_payload in value
+        ]
+    )
     return rtn
 
 
diff --git a/st2reactor/st2reactor/rules/filter.py b/st2reactor/st2reactor/rules/filter.py
--- a/st2reactor/st2reactor/rules/filter.py
+++ b/st2reactor/st2reactor/rules/filter.py
@@ -154,8 +154,10 @@ def _check_criterion(self, criterion_k, criterion_v, payload_lookup):
 
             return (False, None, None)
 
+        # Avoids the dict unique keys limitation. Allows multiple evaluations of the same payload item by a rule.
+        criterion_k_hash_strip = criterion_k.split("#", 1)[0]
         try:
-            matches = payload_lookup.get_value(criterion_k)
+            matches = payload_lookup.get_value(criterion_k_hash_strip)
             # pick value if only 1 matches else will end up being an array match.
             if matches:
                 payload_value = matches[0] if len(matches) > 0 else matches


Test patch:
diff --git a/st2common/tests/unit/test_operators.py b/st2common/tests/unit/test_operators.py
--- a/st2common/tests/unit/test_operators.py
+++ b/st2common/tests/unit/test_operators.py
@@ -564,6 +564,203 @@ def record_function_args(criterion_k, criterion_v, payload_lookup):
             ],
         )
 
+    def _test_function(self, criterion_k, criterion_v, payload_lookup):
+        op = operators.get_operator(criterion_v["type"])
+        return op(payload_lookup.get_value("item.to_value")[0], criterion_v["pattern"])
+
+    def test_search_any2any(self):
+        # true if any payload items match any criteria
+        op = operators.get_operator("search")
+
+        payload = [
+            {
+                "field_name": "waterLevel",
+                "to_value": 30,
+            },
+            {
+                "field_name": "waterLevel",
+                "to_value": 45,
+            },
+        ]
+
+        criteria_pattern = {
+            "item.waterLevel#1": {
+                "type": "lessthan",
+                "pattern": 40,
+            },
+            "item.waterLevel#2": {
+                "type": "greaterthan",
+                "pattern": 50,
+            },
+        }
+
+        result = op(payload, criteria_pattern, "any2any", self._test_function)
+        self.assertTrue(result)
+
+        payload[0]["to_value"] = 44
+
+        result = op(payload, criteria_pattern, "any2any", self._test_function)
+        self.assertFalse(result)
+
+    def test_search_any(self):
+        # true if any payload items match all criteria
+        op = operators.get_operator("search")
+        payload = [
+            {
+                "field_name": "waterLevel",
+                "to_value": 45,
+            },
+            {
+                "field_name": "waterLevel",
+                "to_value": 20,
+            },
+        ]
+
+        criteria_pattern = {
+            "item.waterLevel#1": {
+                "type": "greaterthan",
+                "pattern": 40,
+            },
+            "item.waterLevel#2": {
+                "type": "lessthan",
+                "pattern": 50,
+            },
+            "item.waterLevel#3": {
+                "type": "equals",
+                "pattern": 46,
+            },
+        }
+
+        result = op(payload, criteria_pattern, "any", self._test_function)
+        self.assertFalse(result)
+
+        payload[0]["to_value"] = 46
+
+        result = op(payload, criteria_pattern, "any", self._test_function)
+        self.assertTrue(result)
+
+        payload[0]["to_value"] = 45
+        del criteria_pattern["item.waterLevel#3"]
+
+        result = op(payload, criteria_pattern, "any", self._test_function)
+        self.assertTrue(result)
+
+    def test_search_all2any(self):
+        # true if all payload items match any criteria
+        op = operators.get_operator("search")
+        payload = [
+            {
+                "field_name": "waterLevel",
+                "to_value": 45,
+            },
+            {
+                "field_name": "waterLevel",
+                "to_value": 20,
+            },
+        ]
+
+        criteria_pattern = {
+            "item.waterLevel#1": {
+                "type": "greaterthan",
+                "pattern": 40,
+            },
+            "item.waterLevel#2": {
+                "type": "lessthan",
+                "pattern": 50,
+            },
+            "item.waterLevel#3": {
+                "type": "equals",
+                "pattern": 46,
+            },
+        }
+
+        result = op(payload, criteria_pattern, "all2any", self._test_function)
+        self.assertTrue(result)
+
+        criteria_pattern["item.waterLevel#2"]["type"] = "greaterthan"
+
+        result = op(payload, criteria_pattern, "all2any", self._test_function)
+        self.assertFalse(result)
+
+    def test_search_all(self):
+        # true if all payload items match all criteria items
+        op = operators.get_operator("search")
+        payload = [
+            {
+                "field_name": "waterLevel",
+                "to_value": 45,
+            },
+            {
+                "field_name": "waterLevel",
+                "to_value": 46,
+            },
+        ]
+
+        criteria_pattern = {
+            "item.waterLevel#1": {
+                "type": "greaterthan",
+                "pattern": 40,
+            },
+            "item.waterLevel#2": {
+                "type": "lessthan",
+                "pattern": 50,
+            },
+        }
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertTrue(result)
+
+        payload[0]["to_value"] = 30
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertFalse(result)
+
+        payload[0]["to_value"] = 45
+
+        criteria_pattern["item.waterLevel#3"] = {
+            "type": "equals",
+            "pattern": 46,
+        }
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertFalse(result)
+
+    def test_search_payload_dict(self):
+        op = operators.get_operator("search")
+        payload = {
+            "field_name": "waterLevel",
+            "to_value": 45,
+        }
+
+        criteria_pattern = {
+            "item.waterLevel#1": {
+                "type": "greaterthan",
+                "pattern": 40,
+            },
+            "item.waterLevel#2": {
+                "type": "lessthan",
+                "pattern": 50,
+            },
+        }
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertTrue(result)
+
+        payload["to_value"] = 30
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertFalse(result)
+
+        payload["to_value"] = 45
+
+        criteria_pattern["item.waterLevel#3"] = {
+            "type": "equals",
+            "pattern": 46,
+        }
+
+        result = op(payload, criteria_pattern, "all", self._test_function)
+        self.assertFalse(result)
+
 
 class OperatorTest(unittest2.TestCase):
     def test_matchwildcard(self):
diff --git a/st2reactor/tests/unit/test_filter.py b/st2reactor/tests/unit/test_filter.py
--- a/st2reactor/tests/unit/test_filter.py
+++ b/st2reactor/tests/unit/test_filter.py
@@ -414,3 +414,25 @@ class MockSystemLookup(object):
         }
         f = RuleFilter(MOCK_TRIGGER_INSTANCE, MOCK_TRIGGER, rule)
         self.assertTrue(f.filter())
+
+    def test_hash_strip_int_value(self):
+        rule = MOCK_RULE_1
+        rule.criteria = {
+            "trigger.int": {"type": "gt", "pattern": 0},
+            "trigger.int#2": {"type": "lt", "pattern": 2},
+        }
+        f = RuleFilter(MOCK_TRIGGER_INSTANCE, MOCK_TRIGGER, rule)
+        self.assertTrue(f.filter(), "equals check should have passed.")
+
+        rule = MOCK_RULE_1
+        rule.criteria = {
+            "trigger.int": {"type": "gt", "pattern": 2},
+            "trigger.int#2": {"type": "lt", "pattern": 3},
+        }
+        f = RuleFilter(MOCK_TRIGGER_INSTANCE, MOCK_TRIGGER, rule)
+        self.assertFalse(f.filter(), "trigger value is gt than 0 but didn't match.")
+
+        rule = MOCK_RULE_1
+        rule.criteria = {"trigger.int#1": {"type": "lt", "pattern": 2}}
+        f = RuleFilter(MOCK_TRIGGER_INSTANCE, MOCK_TRIGGER, rule)
+        self.assertTrue(f.filter(), "trigger value is gt than 0 but didn't match.")