Repository: rasahq/rasa
Issue: Rule starting with action-intent pair wrongly contradicts stories

**Rasa version**: 2.3.4 (`main`)

**Python version**: 3.8.5

**Operating system** (windows, osx, ...): OSX

**Issue**:
I want to write a rule that basically conditions on an intent `i3` IF that intent follows after a particular action `utter_1` (see the training rules below). Unfortunately, such rule (very non-intuitively) also predicts `action_listen` after `utter_1`. Hence, if I have a training story that contains a different action after `utter_1`, it creates a contradiction (see the training stories below). Note that having two stories in which `utter_1` is followed once by an intent and the other time by an action works fine -- because the conversation history is different for these two cases. But as soon as I create the mentioned rule, things break.

**Suggested solution:**
See [this Slack thread](https://rasa-hq.slack.com/archives/C8LEHMTAM/p1615466129437400) for more context and a discussion that led to this suggestion. Basically, if there is a rule that starts with an action followed by an intent and another action, we'll assume that it is only the last action that the user really meant to predict with the rule. Therefore, IF the predicted `action_listen` (which the user most likely doesn't know about) contradicts a story, we remove this particular rule (only the one that predicts the `action_listen`) from `RulePolicy`'s lookup and proceed (possibly also raising a warning).

**Error (including full traceback)**:
```
InvalidRule:
Contradicting rules or stories found 🚨

- the prediction of the action 'utter_2' in story 'action after action' is contradicting with rule(s) 'conditioned on action' which predicted action 'action_listen'.
Please update your stories and rules so that they don't contradict each other.
You can find more information about the usage of rules at https://rasa.com/docs/rasa/rules.
```

**Command or request that led to error**:
```
rasa train core --augmentation 0
```

**Content of configuration file (config.yml)** (if relevant):
```yml
language: en
policies:
  - name: RulePolicy
  - name: TEDPolicy
    epochs: 1
```

**Content of domain file (domain.yml)** (if relevant):
```yml
intents:
- i1
- i2
- i3
- i4
actions:
- utter_1
- utter_2
- utter_3
```

**Training data**:
```yml
stories:
- story: action after action
  steps:
  - intent: i1
  - action: utter_1
  - action: utter_2 # action goes after utter_1
- story: intent after action
  steps:
  - intent: i2
  - action: utter_1
  - intent: i3
  - action: utter_2 # intent goes after utter_1 because of different history
  - intent: i4
  - action: utter_3
rules:
- rule: conditioned on action
  steps:
  - action: utter_1
  - intent: i3 # this intent is preceded by an implicit action_listen, which break things
  - action: utter_2 

Changed files:
rasa/core/policies/rule_policy.py, tests/core/policies/test_rule_policy.py

diff --git a/rasa/core/policies/rule_policy.py b/rasa/core/policies/rule_policy.py
index 13a7f8089f1..b3d3a77c699 100644
--- a/rasa/core/policies/rule_policy.py
+++ b/rasa/core/policies/rule_policy.py
@@ -519,6 +519,39 @@ class RulePolicy(MemoizationPolicy):
             )
         return loop_sources
 
+    def _should_delete(
+        self,
+        prediction_source: Text,
+        tracker: TrackerWithCachedStates,
+        predicted_action_name: Text,
+    ) -> bool:
+        """Checks whether this contradiction is due to action, intent pair.
+
+        Args:
+            prediction_source: the states that result in the prediction
+            tracker: the tracker that raises the contradiction
+
+        Returns:
+            true if the contradiction is a result of an action, intent pair in the rule.
+        """
+        if (
+            # only apply to contradicting story, not rule
+            tracker.is_rule_tracker
+            # only apply for prediction after unpredictable action
+            or prediction_source.count(PREVIOUS_ACTION) > 1
+            # only apply for prediction of action_listen
+            or predicted_action_name != ACTION_LISTEN_NAME
+        ):
+            return False
+        for source in self.lookup[RULES]:
+            # remove rule only if another action is predicted after action_listen
+            if (
+                source.startswith(prediction_source[:-2])
+                and not prediction_source == source
+            ):
+                return True
+        return False
+
     def _check_prediction(
         self,
         tracker: TrackerWithCachedStates,
@@ -529,6 +562,10 @@ class RulePolicy(MemoizationPolicy):
         if not predicted_action_name or predicted_action_name == gold_action_name:
             return []
 
+        if self._should_delete(prediction_source, tracker, predicted_action_name):
+            self.lookup[RULES].pop(prediction_source)
+            return []
+
         tracker_type = "rule" if tracker.is_rule_tracker else "story"
         contradicting_rules = {
             rule_name
@@ -916,8 +953,6 @@ class RulePolicy(MemoizationPolicy):
 
         return None, None
 
-        return None
-
     def _find_action_from_rules(
         self,
         tracker: DialogueStateTracker,
diff --git a/tests/core/policies/test_rule_policy.py b/tests/core/policies/test_rule_policy.py
index 665a0f5c5a1..b775c29709d 100644
--- a/tests/core/policies/test_rule_policy.py
+++ b/tests/core/policies/test_rule_policy.py
@@ -40,7 +40,7 @@ from rasa.shared.core.events import (
 )
 from rasa.shared.nlu.interpreter import RegexInterpreter
 from rasa.core.nlg import TemplatedNaturalLanguageGenerator
-from rasa.core.policies.rule_policy import RulePolicy, InvalidRule
+from rasa.core.policies.rule_policy import RulePolicy, InvalidRule, RULES
 from rasa.shared.core.trackers import DialogueStateTracker
 from rasa.shared.core.generator import TrackerWithCachedStates
 
@@ -2624,3 +2624,232 @@ def test_hide_rule_turn_with_loops_as_followup_action():
             ACTIVE_LOOP: {LOOP_NAME: form_name},
         },
     ]
+
+
+def test_remove_action_listen_prediction_if_contradicts_with_story():
+    intent_1 = "intent_1"
+    utter_1 = "utter_1"
+    utter_2 = "utter_2"
+    domain = Domain.from_yaml(
+        f"""
+        version: "2.0"
+        intents:
+        - {intent_1}
+        actions:
+        - {utter_1}
+        - {utter_2}
+        """
+    )
+    rule = TrackerWithCachedStates.from_events(
+        "conditioned on action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_1),
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_2),
+        ],
+        is_rule_tracker=True,
+    )
+    story = TrackerWithCachedStates.from_events(
+        "intent after action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_1),
+            ActionExecuted(utter_2),
+        ],
+    )
+    policy = RulePolicy()
+    policy.train(
+        [rule, story], domain, RegexInterpreter(),
+    )
+    prediction_source = [{PREVIOUS_ACTION: {ACTION_NAME: utter_1}}]
+    key = policy._create_feature_key(prediction_source)
+    assert key not in policy.lookup[RULES]
+    assert len(policy.lookup[RULES]) == 1
+
+
+def test_keep_action_listen_prediction_after_predictable_action():
+    intent_1 = "intent_1"
+    utter_1 = "utter_1"
+    utter_2 = "utter_2"
+    utter_3 = "utter_3"
+    domain = Domain.from_yaml(
+        f"""
+        version: "2.0"
+        intents:
+        - {intent_1}
+        actions:
+        - {utter_1}
+        - {utter_2}
+        - {utter_3}
+        """
+    )
+    rule = TrackerWithCachedStates.from_events(
+        "action_listen after predictable action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_2),
+            ActionExecuted(utter_1),
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_2),
+        ],
+        is_rule_tracker=True,
+    )
+    story = TrackerWithCachedStates.from_events(
+        "intent after action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_2),
+            ActionExecuted(utter_1),
+            ActionExecuted(utter_3),
+        ],
+    )
+    policy = RulePolicy()
+    # prediction of action_listen should only be removed if it occurs after the first
+    # action (unpredictable)
+    with pytest.raises(InvalidRule):
+        policy.train(
+            [rule, story], domain, RegexInterpreter(),
+        )
+
+
+def test_keep_action_listen_prediction_if_last_prediction():
+    intent_1 = "intent_1"
+    utter_1 = "utter_1"
+    utter_2 = "utter_2"
+    domain = Domain.from_yaml(
+        f"""
+        version: "2.0"
+        intents:
+        - {intent_1}
+        actions:
+        - {utter_1}
+        - {utter_2}
+        """
+    )
+    rule = TrackerWithCachedStates.from_events(
+        "last prediction is action_listen",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_1),
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered(intent={"name": intent_1}),
+        ],
+        is_rule_tracker=True,
+    )
+    story = TrackerWithCachedStates.from_events(
+        "intent after action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_1),
+            ActionExecuted(utter_2),
+        ],
+    )
+    policy = RulePolicy()
+    # prediction of action_listen should only be removed if it's not the last prediction
+    with pytest.raises(InvalidRule):
+        policy.train(
+            [rule, story], domain, RegexInterpreter(),
+        )
+
+
+def test_keep_action_listen_prediction_if_contradicts_with_rule():
+    intent_1 = "intent_1"
+    utter_1 = "utter_1"
+    utter_2 = "utter_2"
+    domain = Domain.from_yaml(
+        f"""
+        version: "2.0"
+        intents:
+        - {intent_1}
+        actions:
+        - {utter_1}
+        - {utter_2}
+        """
+    )
+    rule = TrackerWithCachedStates.from_events(
+        "conditioned on action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_1),
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_2),
+        ],
+        is_rule_tracker=True,
+    )
+    other_rule = TrackerWithCachedStates.from_events(
+        "intent after action",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_1),
+            ActionExecuted(utter_2),
+        ],
+        is_rule_tracker=True,
+    )
+    policy = RulePolicy()
+    with pytest.raises(InvalidRule):
+        policy.train(
+            [rule, other_rule], domain, RegexInterpreter(),
+        )
+
+
+def test_raise_contradiction_if_rule_contradicts_with_story():
+    intent_1 = "intent_1"
+    utter_1 = "utter_1"
+    utter_2 = "utter_2"
+    domain = Domain.from_yaml(
+        f"""
+        version: "2.0"
+        intents:
+        - {intent_1}
+        actions:
+        - {utter_1}
+        - {utter_2}
+        """
+    )
+    rule = TrackerWithCachedStates.from_events(
+        "rule without action_listen",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(RULE_SNIPPET_ACTION_NAME),
+            ActionExecuted(utter_1),
+            ActionExecuted(utter_2),
+        ],
+        is_rule_tracker=True,
+    )
+    story = TrackerWithCachedStates.from_events(
+        "contradicts with rule",
+        domain=domain,
+        slots=domain.slots,
+        evts=[
+            ActionExecuted(utter_1),
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered(intent={"name": intent_1}),
+            ActionExecuted(utter_2),
+        ],
+    )
+    policy = RulePolicy()
+    with pytest.raises(InvalidRule):
+        policy.train(
+            [rule, story], domain, RegexInterpreter(),
+        )