Repository: RasaHQ/rasa
Pull request: predict core fallback if no rule found
**Proposed changes**:
- fix https://github.com/RasaHQ/rasa/issues/6208

```yaml
policies:
  - name: RulePolicy
    # Confidence of the prediction if no rule matched d de-facto threshold for a
    # core fallback.
    core_fallback_threshold: 0.3
    # Name of the action which should be predicted if no rule matched.
    core_fallback_action_name: "action_default_fallback"
    # If `True` `core_fallback_action_name` is predicted in case no rule matched.
    enable_fallback_prediction: True

```
**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [ ] updated the documentation (will happen in https://github.com/RasaHQ/rasa/pull/6195/files)
- [ ] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
RulePolicy should predict Core fallback action if no rule was found
With the RulePolicy you can now write rules to handle low NLU confidence.
However, as we deprecate the FallbackPolicies we now have no policy which predicts a fallback action in case the Core confidence is low.

**Proposed Steps**
- [ ] The `RulePolicy` should predict a fallback action `x` with a confidence `y` in case no matching rule was applied
- [ ] the name `x` of the fallback action can be configured in the policy config (`config.yml`)
- [ ] the confidence threshold `y` for the fallback can be configured in the policy config
- [ ] add a config value to enable / disable this behavior


```yaml
polices:
- name: RulePolicy
  core_threshold: 0.5
  core_fallback_action_name: action_default_fallback
  enable_fallback_prediction: True/False
```

Hints from issue discussion:

Commit message: support predicting a core fallback in case no matching rule was found

Patch to review:
diff --git a/examples/rules/config.yml b/examples/rules/config.yml
index d27831df4fb0..c19c8000e9c1 100644
--- a/examples/rules/config.yml
+++ b/examples/rules/config.yml
@@ -8,7 +8,7 @@ pipeline:
   - name: DucklingHTTPExtractor
     url: http://localhost:8000
     dimensions:
-      - number
+    - number
   - name: DIETClassifier
     epochs: 100
   - name: EntitySynonymMapper
@@ -20,3 +20,10 @@ pipeline:
 
 policies:
   - name: RulePolicy
+    # Confidence of the prediction if no rule matched d de-facto threshold for a
+    # core fallback.
+    core_fallback_threshold: 0.3
+    # Name of the action which should be predicted if no rule matched.
+    core_fallback_action_name: "action_default_fallback"
+    # If `True` `core_fallback_action_name` is predicted in case no rule matched.
+    enable_fallback_prediction: True
diff --git a/rasa/core/constants.py b/rasa/core/constants.py
index c9a4062db143..babd8bfb09d8 100644
--- a/rasa/core/constants.py
+++ b/rasa/core/constants.py
@@ -8,8 +8,6 @@
 
 DEFAULT_CORE_FALLBACK_THRESHOLD = 0.0
 
-DEFAULT_FALLBACK_ACTION = "action_default_fallback"
-
 DEFAULT_REQUEST_TIMEOUT = 60 * 5  # 5 minutes
 
 DEFAULT_RESPONSE_TIMEOUT = 60 * 60  # 1 hour
diff --git a/rasa/core/policies/ensemble.py b/rasa/core/policies/ensemble.py
index a16a12482af9..ad669dc1fae5 100644
--- a/rasa/core/policies/ensemble.py
+++ b/rasa/core/policies/ensemble.py
@@ -90,7 +90,11 @@ def check_domain_ensemble_compatibility(
         from rasa.core.policies.mapping_policy import MappingPolicy
         from rasa.core.policies.two_stage_fallback import TwoStageFallbackPolicy
 
-        policies_needing_validation = [MappingPolicy, TwoStageFallbackPolicy]
+        policies_needing_validation = [
+            MappingPolicy,
+            TwoStageFallbackPolicy,
+            RulePolicy,
+        ]
         for policy in policies_needing_validation:
             policy.validate_against_domain(ensemble, domain)
 
diff --git a/rasa/core/policies/fallback.py b/rasa/core/policies/fallback.py
index d95ec163dbc5..cb4f1cf855c2 100644
--- a/rasa/core/policies/fallback.py
+++ b/rasa/core/policies/fallback.py
@@ -4,7 +4,7 @@
 from typing import Any, List, Text, Optional, Dict, Tuple
 
 from rasa.constants import DOCS_URL_MIGRATION_GUIDE
-from rasa.core.actions.action import ACTION_LISTEN_NAME
+from rasa.core.actions.action import ACTION_LISTEN_NAME, ACTION_DEFAULT_FALLBACK_NAME
 
 import rasa.utils.io
 from rasa.utils import common as common_utils
@@ -35,7 +35,7 @@ def __init__(
         nlu_threshold: float = 0.3,
         ambiguity_threshold: float = 0.1,
         core_threshold: float = 0.3,
-        fallback_action_name: Text = "action_default_fallback",
+        fallback_action_name: Text = ACTION_DEFAULT_FALLBACK_NAME,
     ) -> None:
         """Create a new Fallback policy.
 
diff --git a/rasa/core/policies/rule_policy.py b/rasa/core/policies/rule_policy.py
index 45bea583964c..1a0f94654270 100644
--- a/rasa/core/policies/rule_policy.py
+++ b/rasa/core/policies/rule_policy.py
@@ -1,11 +1,11 @@
 import logging
-from typing import List, Dict, Text, Optional, Any, Set
+from typing import List, Dict, Text, Optional, Any, Set, TYPE_CHECKING
 
 import re
 from collections import defaultdict
 
 from rasa.core.events import FormValidation
-from rasa.core.domain import PREV_PREFIX, ACTIVE_FORM_PREFIX, Domain
+from rasa.core.domain import PREV_PREFIX, ACTIVE_FORM_PREFIX, Domain, InvalidDomain
 from rasa.core.featurizers import TrackerFeaturizer
 from rasa.core.interpreter import NaturalLanguageInterpreter, RegexInterpreter
 from rasa.core.policies.memoization import MemoizationPolicy
@@ -23,8 +23,12 @@
     ACTION_BACK_NAME,
     ACTION_SESSION_START_NAME,
     RULE_SNIPPET_ACTION_NAME,
+    ACTION_DEFAULT_FALLBACK_NAME,
 )
 
+if TYPE_CHECKING:
+    from rasa.core.policies.ensemble import PolicyEnsemble
+
 logger = logging.getLogger(__name__)
 
 # These are Rasa Open Source default actions and overrule everything at any time.
@@ -59,14 +63,59 @@ def __init__(
         featurizer: Optional[TrackerFeaturizer] = None,
         priority: int = FORM_POLICY_PRIORITY,
         lookup: Optional[Dict] = None,
+        core_fallback_threshold: float = 0.3,
+        core_fallback_action_name: Text = ACTION_DEFAULT_FALLBACK_NAME,
+        enable_fallback_prediction: bool = True,
     ) -> None:
+        """Create a `RulePolicy` object.
+
+        Args:
+            featurizer: `Featurizer` which is used to convert conversation states to
+                features.
+            priority: Priority of the policy which is used if multiple policies predict
+                actions with the same confidence.
+            lookup: Lookup table which is used to pick matching rules for a conversation
+                state.
+            core_fallback_threshold: Confidence of the prediction if no rule matched
+                and de-facto threshold for a core fallback.
+            core_fallback_action_name: Name of the action which should be predicted
+                if no rule matched.
+            enable_fallback_prediction: If `True` `core_fallback_action_name` is
+                predicted in case no rule matched.
+        """
         if not featurizer:
             # max history is set to `None` in order to capture lengths of rule stories
             featurizer = self._standard_featurizer()
             featurizer.max_history = None
 
+        self._core_fallback_threshold = core_fallback_threshold
+        self._fallback_action_name = core_fallback_action_name
+        self._enable_fallback_prediction = enable_fallback_prediction
+
         super().__init__(featurizer=featurizer, priority=priority, lookup=lookup)
 
+    @classmethod
+    def validate_against_domain(
+        cls, ensemble: Optional["PolicyEnsemble"], domain: Optional[Domain]
+    ) -> None:
+        if ensemble is None:
+            return
+
+        rule_policy = next(
+            (p for p in ensemble.policies if isinstance(p, RulePolicy)), None
+        )
+        if not rule_policy or not rule_policy._enable_fallback_prediction:
+            return
+
+        if (
+            domain is None
+            or rule_policy._fallback_action_name not in domain.action_names
+        ):
+            raise InvalidDomain(
+                "The configured fallback action {p._fallback_action_name} must be "
+                "present in the domain."
+            )
+
     def _create_feature_key(self, states: List[Dict]) -> Text:
 
         feature_str = ""
@@ -458,3 +507,12 @@ def predict_action_probabilities(
             return self._prediction_result(rules_action_name, tracker, domain)
 
         return result
+
+    def _default_predictions(self, domain: Domain) -> List[float]:
+        result = super()._default_predictions(domain)
+
+        if self._enable_fallback_prediction:
+            result[
+                domain.index_for_action(self._fallback_action_name)
+            ] = self._core_fallback_threshold
+        return result
diff --git a/tests/core/policies/test_rule_policy.py b/tests/core/policies/test_rule_policy.py
index 2f7ef35ec062..acdddb692a0b 100644
--- a/tests/core/policies/test_rule_policy.py
+++ b/tests/core/policies/test_rule_policy.py
@@ -1,4 +1,4 @@
-from typing import List, Text
+from typing import List, Text, Optional
 
 import pytest
 from rasa.constants import DEFAULT_NLU_FALLBACK_INTENT_NAME
@@ -124,9 +124,12 @@ def test_faq_rule():
 
 
 def assert_predicted_action(
-    action_probabilities: List[float], domain: Domain, expected_action_name: Text
+    action_probabilities: List[float],
+    domain: Domain,
+    expected_action_name: Text,
+    confidence: float = 1.0,
 ) -> None:
-    assert max(action_probabilities) == 1
+    assert max(action_probabilities) == confidence
     index_of_predicted_action = action_probabilities.index(max(action_probabilities))
     prediction_action_name = domain.action_names[index_of_predicted_action]
     assert prediction_action_name == expected_action_name
@@ -639,7 +642,7 @@ async def test_form_unhappy_path_no_validation_from_rule():
     tracker = DialogueStateTracker.from_events(
         "casd", evts=conversation_events, slots=domain.slots
     )
-    action_probabilities = policy.predict_action_probabilities(tracker, domain,)
+    action_probabilities = policy.predict_action_probabilities(tracker, domain)
     assert_predicted_action(action_probabilities, domain, form_name)
     # check that RulePolicy added FormValidation False event based on the training rule
     assert tracker.events[-1] == FormValidation(False)
@@ -705,7 +708,7 @@ async def test_form_unhappy_path_no_validation_from_story():
     tracker = DialogueStateTracker.from_events(
         "casd", evts=conversation_events, slots=domain.slots
     )
-    action_probabilities = policy.predict_action_probabilities(tracker, domain,)
+    action_probabilities = policy.predict_action_probabilities(tracker, domain)
     # there is no rule for next action
     assert max(action_probabilities) == 0
     # check that RulePolicy added FormValidation False event based on the training story
@@ -1097,3 +1100,75 @@ def test_default_actions(intent_name: Text, expected_action_name: Text):
     action_probabilities = policy.predict_action_probabilities(new_conversation, domain)
 
     assert_predicted_action(action_probabilities, domain, expected_action_name)
+
+
+@pytest.mark.parametrize(
+    "rule_policy, expected_confidence, expected_prediction",
+    [
+        (RulePolicy(), 0.3, ACTION_DEFAULT_FALLBACK_NAME),
+        (
+            RulePolicy(
+                core_fallback_threshold=0.1,
+                core_fallback_action_name="my_core_fallback",
+            ),
+            0.1,
+            "my_core_fallback",
+        ),
+    ],
+)
+def test_predict_core_fallback(
+    rule_policy: RulePolicy, expected_confidence: float, expected_prediction: Text
+):
+    other_intent = "other"
+    domain = Domain.from_yaml(
+        f"""
+    intents:
+    - {GREET_INTENT_NAME}
+    - {other_intent}
+    actions:
+    - {UTTER_GREET_ACTION}
+    - my_core_fallback
+        """
+    )
+    rule_policy.train([GREET_RULE], domain, RegexInterpreter())
+
+    new_conversation = DialogueStateTracker.from_events(
+        "bla2",
+        evts=[
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered("haha", {"name": other_intent}),
+        ],
+    )
+
+    action_probabilities = rule_policy.predict_action_probabilities(
+        new_conversation, domain
+    )
+
+    assert_predicted_action(
+        action_probabilities, domain, expected_prediction, expected_confidence
+    )
+
+
+def test_predict_nothing_if_fallback_disabled():
+    other_intent = "other"
+    domain = Domain.from_yaml(
+        f"""
+    intents:
+    - {GREET_INTENT_NAME}
+    - {other_intent}
+    actions:
+    - {UTTER_GREET_ACTION}
+        """
+    )
+    policy = RulePolicy(enable_fallback_prediction=False)
+    policy.train([GREET_RULE], domain, RegexInterpreter())
+    new_conversation = DialogueStateTracker.from_events(
+        "bla2",
+        evts=[
+            ActionExecuted(ACTION_LISTEN_NAME),
+            UserUttered("haha", {"name": other_intent}),
+        ],
+    )
+    action_probabilities = policy.predict_action_probabilities(new_conversation, domain)
+
+    assert max(action_probabilities) == 0
diff --git a/tests/core/test_agent.py b/tests/core/test_agent.py
index f76cc9018f6c..fd12b3f02b0e 100644
--- a/tests/core/test_agent.py
+++ b/tests/core/test_agent.py
@@ -292,6 +292,63 @@ def test_two_stage_fallback_without_deny_suggestion(
     assert "The intent 'out_of_scope' must be present" in str(execinfo.value)
 
 
+@pytest.mark.parametrize(
+    "domain, policy_config",
+    [
+        (
+            {"actions": ["other-action"]},
+            {
+                "policies": [
+                    {"name": "RulePolicy", "core_fallback_action_name": "my_fallback"}
+                ]
+            },
+        )
+    ],
+)
+def test_rule_policy_without_fallback_action_present(
+    domain: Dict[Text, Any], policy_config: Dict[Text, Any]
+):
+    with pytest.raises(InvalidDomain):
+        Agent(
+            domain=Domain.from_dict(domain),
+            policies=PolicyEnsemble.from_dict(policy_config),
+        )
+
+
+@pytest.mark.parametrize(
+    "domain, policy_config",
+    [
+        (
+            {"actions": ["other-action"]},
+            {
+                "policies": [
+                    {
+                        "name": "RulePolicy",
+                        "core_fallback_action_name": "my_fallback",
+                        "enable_fallback_prediction": False,
+                    }
+                ]
+            },
+        ),
+        (
+            {"actions": ["my-action"]},
+            {
+                "policies": [
+                    {"name": "RulePolicy", "core_fallback_action_name": "my-action"}
+                ]
+            },
+        ),
+        ({}, {"policies": [{"name": "MemoizationPolicy"}]}),
+    ],
+)
+def test_rule_policy_valid(domain: Dict[Text, Any], policy_config: Dict[Text, Any]):
+    # no exception should be thrown
+    Agent(
+        domain=Domain.from_dict(domain),
+        policies=PolicyEnsemble.from_dict(policy_config),
+    )
+
+
 async def test_agent_update_model_none_domain(trained_rasa_model: Text):
     agent = await load_agent(model_path=trained_rasa_model)
     agent.update_model(


Code review comments:
diff_hunk: @@ -20,3 +20,10 @@ pipeline:
 
 policies:
   - name: RulePolicy
+    # Confidence of the prediction if no rule matched d de-facto threshold for a
path: examples/rules/config.yml
text: @user1:
```suggestion
    # Confidence of the prediction if no rule matched and de-facto threshold for a
```

diff_hunk: @@ -59,14 +63,59 @@ def __init__(
         featurizer: Optional[TrackerFeaturizer] = None,
         priority: int = FORM_POLICY_PRIORITY,
         lookup: Optional[Dict] = None,
+        core_fallback_threshold: float = 0.3,
+        core_fallback_action_name: Text = ACTION_DEFAULT_FALLBACK_NAME,
+        enable_fallback_prediction: bool = True,
     ) -> None:
+        """Create a `RulePolicy` object.
+
+        Args:
+            featurizer: `Featurizer` which is used to convert conversation states to
+                features.
+            priority: Priority of the policy which is used if multiple policies predict
+                actions with the same confidence.
+            lookup: Lookup table which is used to pick matching rules for a conversation
+                state.
+            core_fallback_threshold: Confidence of the prediction if no rule matched
+                and de-facto threshold for a core fallback.
+            core_fallback_action_name: Name of the action which should be predicted
+                if no rule matched.
+            enable_fallback_prediction: If `True` `core_fallback_action_name` is
+                predicted in case no rule matched.
+        """
         if not featurizer:
             # max history is set to `None` in order to capture lengths of rule stories
             featurizer = self._standard_featurizer()
             featurizer.max_history = None
 
+        self._core_fallback_threshold = core_fallback_threshold
+        self._fallback_action_name = core_fallback_action_name
+        self._enable_fallback_prediction = enable_fallback_prediction
+
         super().__init__(featurizer=featurizer, priority=priority, lookup=lookup)
 
+    @classmethod
+    def validate_against_domain(
+        cls, ensemble: Optional["PolicyEnsemble"], domain: Optional[Domain]
+    ) -> None:
+        if ensemble is None:
+            return
+
+        rule_policy = next(
+            (p for p in ensemble.policies if isinstance(p, RulePolicy)), None
+        )
+        if not rule_policy or not rule_policy._enable_fallback_prediction:
+            return
+
+        if (
+            domain is None
+            or rule_policy._fallback_action_name not in domain.action_names
+        ):
+            raise InvalidDomain(
+                "The configured fallback action {p._fallback_action_name} must be "
+                "present in the domain."
path: rasa/core/policies/rule_policy.py
text: @user1:
```suggestion
                f"The configured fallback action '{rule_policy._fallback_action_name}' "
                f"must be present in the domain."
```

diff_hunk: @@ -1,4 +1,4 @@
-from typing import List, Text
+from typing import List, Text, Optional
path: tests/core/policies/test_rule_policy.py
text: @user1:
```suggestion
from typing import List, Text
```

diff_hunk: @@ -23,8 +23,12 @@
     ACTION_BACK_NAME,
     ACTION_SESSION_START_NAME,
     RULE_SNIPPET_ACTION_NAME,
+    ACTION_DEFAULT_FALLBACK_NAME,
 )
 
+if TYPE_CHECKING:
+    from rasa.core.policies.ensemble import PolicyEnsemble
path: rasa/core/policies/rule_policy.py
text: @user1:
I think this should solve the "Check Types" CI issue. (The file with `PolicyEnsemble` imports `RulePolicy`, so we have an import cycle. But as the one here is only for type checking, it should be fine to simple disable.)
```suggestion
    from rasa.core.policies.ensemble import PolicyEnsemble # pytype: disable=pyi-error
```