Repository: RasaHQ/rasa
Pull request: emulate form rejection in evaluation
**Proposed changes**:
-  FormAction is not run, so we add `ActionExecutionRejected` event if form action is predicted, but not expected and if "unhappy" intent is remembered by FormPolicy
- fixes #2792 

**Status (please check what you already did)**:
- [x] made PR ready for code review
- [ ] added some tests for the functionality
- [ ] updated the documentation
- [ ] updated the changelog

Problem statement:
How to use e2e evaluation for FormAction
<!-- THIS INFORMATION IS MANDATORY - YOUR ISSUE WILL BE CLOSED IF IT IS MISSING. If you don't know your Rasa Core version, use `pip list | grep rasa_core`. Removing the below information is allowed for FEATURE REQUESTS. --> 

**Rasa Core version**: 0.13.0a1

**Python version**: 3.6.6

**Operating system** (windows, osx, ...): osx

**Issue**:
Right now I'm using FormAction for my stories. I am having troubles writing `e2e_stories.md` for e2e evaluation.
Is there an appropriate way to write `e2e_stories.md` when using FormAction?


**Content of domain file** (if used & relevant):
```yaml

```

Hints from issue discussion:
Hey, so the thing is that if you explicitly write all the steps within a form, it has to be written with a form prefix (see https://rasa.com/docs/core/interactive_learning/#the-form-prefix) so that rasa core doesn't get confused. This also means that those lines will get ignored from training though. So I don't think e2e evaluation with forms is working right now.
@Ghostvv do you have any additional thoughts on this?
I was trying to use core evaluation instead. But it turned out that core evaluation didn't work neither. 
I used the "formbot" in examples for testing, simply using all training stories as testing stories. Out of 10 stories, only 1 passed (the happy path). A typical failed story is:
```
## chitchat stop but continue path
* request_restaurant
    - restaurant_form
    - form{"name": "restaurant_form"}
* form: chitchat
    - form: utter_chitchat   <!-- predicted: restaurant_form -->
    - form: restaurant_form
* form: stop
    - form: utter_ask_continue   <!-- predicted: restaurant_form -->
    - form: action_listen   <!-- predicted: restaurant_form -->
* form: affirm
    - form: restaurant_form
    - form{"name": null}
    - utter_slots_values
* thankyou
    - utter_noworries
```
ok so that story looks slightly wrong. the `chitchat` intent shouldn't have a `form:` prefix. Everything that isn't the happy path shouldn't have a form prefix. I think it might be easiest to generate these stories via interactive learning because it can be a bit confusing.
Let me know if that works
The story I pasted above is generated from the core evaluation (in the file `failed_stories.md`), not the one I used for test. The testing stories are exactly the same with training stories, in this case:
```
## chitchat stop but continue path
* request_restaurant
    - restaurant_form
    - form{"name": "restaurant_form"}
* chitchat
    - utter_chitchat
    - restaurant_form
* stop
    - utter_ask_continue
* affirm
    - restaurant_form
    - form{"name": null}
    - utter_slots_values
* thankyou
    - utter_noworries
```
The form prefix is added during core evaluation.
I track to `StoryStep.as_story_string` in `rasa_core.training.structures` (line: 169).
```python
            if isinstance(s, UserUttered):
                if self.story_string_helper.active_form is None:
                    result += self._user_string(s, e2e)
                else:
                    # form is active
                    # it is not known whether the form will be
                    # successfully executed, so store this
                    # story string for later
                    self._store_user_strings(s, e2e, FORM_PREFIX)
```
When `tracker` export stories, as long as a form is active, the user utterance will be appended a FORM_PREFIX. However, sometimes the user utterance is not part of the form. In this example, "chitchat" is not part of `restaurant_form`, therefore should not be appended a FORM_PREFIX.
I tried interactive learning. It generated correct stories. I haven't figured out why in evaluation the stories are presented differently.
Maybe because in interactive learning the endpoints are connected so the bot can understand what intents should be ignored and what should not (which are specified in `actions.py`).
The guy I tagged in the comment above (@Ghostvv) probably has some ideas, but he's currently away so may not get back to you until next week
@yobekiko thank you for spotting it, yes during interactive training the `FormAction` fails to validate, so `core` knows when to add `form: ` prefix and when not. Since during e2e evaluation `FormAction` doesn't run, `core` doesn't know when it fails
The solution is to ignore `FormPolicy` predictions during evaluation
@Ghostvv can you please elaborate on how to do that? I am also having the same problem. I am unable to use e2e for our bot since it almost consists of forms only (with unhappy paths steering back the conversations to reask the form questions).

Commit message: Merge branch 'master' into fix_form_evaluation

Patch to review:
diff --git a/examples/concertbot/Makefile b/examples/concertbot/Makefile
index 81bb58949732..b16b61e84158 100644
--- a/examples/concertbot/Makefile
+++ b/examples/concertbot/Makefile
@@ -20,7 +20,7 @@ run-core:
 	python -m rasa.core.run --core models/dialogue --endpoints endpoints.yml
 
 run-actions:
-	python -m rasa.core_sdk.endpoint --actions actions
+	python -m rasa_core_sdk.endpoint --actions actions
 
 train-core:
 	python train.py
diff --git a/examples/formbot/Makefile b/examples/formbot/Makefile
index b2763e2f6a77..465d52d54e5c 100644
--- a/examples/formbot/Makefile
+++ b/examples/formbot/Makefile
@@ -15,10 +15,10 @@ train-core:
 	python -m rasa.core.train -s data/stories.md -d domain.yml -o models/dialogue --debug
 
 run-core:
-	python -m rasa.core.run --core models/dialogue --nlu models/nlu/current --verbose --endpoints endpoints.yml
+	python -m rasa.core.run --core models/dialogue --verbose --endpoints endpoints.yml
 
 run-actions:
-	python -m rasa.core_sdk.endpoint --actions actions
+	python -m rasa_core_sdk.endpoint --actions actions
 
 run:
 	make run-actions&
@@ -32,3 +32,6 @@ visualize:
 
 train-nlu:
 	python -m rasa.nlu.train -c config.yml --fixed_model_name current --data data/nlu.md -o models --project nlu --verbose
+
+evaluate-core:
+	python -m rasa.core.test --core models/dialogue -s data/stories.md --fail_on_prediction_errors
diff --git a/rasa/core/policies/form_policy.py b/rasa/core/policies/form_policy.py
index 482412c4d4f9..a12b1bb83620 100644
--- a/rasa/core/policies/form_policy.py
+++ b/rasa/core/policies/form_policy.py
@@ -79,6 +79,28 @@ def recall(self,
         # modify the states
         return self._recall_states(self._modified_states(states))
 
+    def state_is_unhappy(self, tracker, domain):
+        # since it is assumed that training stories contain
+        # only unhappy paths, notify the form that
+        # it should not be validated if predicted by other policy
+        tracker_as_states = self.featurizer.prediction_states(
+            [tracker], domain)
+        states = tracker_as_states[0]
+
+        memorized_form = self.recall(states, tracker, domain)
+
+        state_is_unhappy = (
+            memorized_form is not None and
+            memorized_form == tracker.active_form.get('name')
+        )
+
+        if state_is_unhappy:
+            logger.debug("There is a memorized tracker state {}, "
+                         "added `FormValidation(False)` event"
+                         "".format(self._modified_states(states)))
+
+        return state_is_unhappy
+
     def predict_action_probabilities(self,
                                      tracker: DialogueStateTracker,
                                      domain: Domain) -> List[float]:
@@ -92,18 +114,7 @@ def predict_action_probabilities(self,
                 # predict form action after user utterance
 
                 if tracker.active_form.get('rejected'):
-                    # since it is assumed that training stories contain
-                    # only unhappy paths, notify the form that
-                    # it should not be validated if predicted by other policy
-                    tracker_as_states = self.featurizer.prediction_states(
-                        [tracker], domain)
-                    states = tracker_as_states[0]
-                    memorized_form = self.recall(states, tracker, domain)
-
-                    if memorized_form == tracker.active_form['name']:
-                        logger.debug("There is a memorized tracker state {}, "
-                                     "added `FormValidation(False)` event"
-                                     "".format(self._modified_states(states)))
+                    if self.state_is_unhappy(tracker, domain):
                         tracker.update(FormValidation(False))
                         return result
 
diff --git a/rasa/core/test.py b/rasa/core/test.py
index 3e9476da8ddc..013b5ac902fd 100644
--- a/rasa/core/test.py
+++ b/rasa/core/test.py
@@ -284,14 +284,42 @@ def _collect_user_uttered_predictions(event,
     return user_uttered_eval_store
 
 
+def _emulate_form_rejection(processor, partial_tracker):
+    from rasa.core.policies.form_policy import FormPolicy
+    from rasa.core.events import ActionExecutionRejected
+
+    if partial_tracker.active_form.get("name"):
+        for p in processor.policy_ensemble.policies:
+            if isinstance(p, FormPolicy):
+                # emulate form rejection
+                partial_tracker.update(ActionExecutionRejected(
+                    partial_tracker.active_form["name"]))
+                # check if unhappy path is covered by the train stories
+                if not p.state_is_unhappy(partial_tracker, processor.domain):
+                    # this state is not covered by the stories
+                    del partial_tracker.events[-1]
+                    partial_tracker.active_form['rejected'] = False
+
+
 def _collect_action_executed_predictions(processor, partial_tracker, event,
                                          fail_on_prediction_errors):
+    from rasa.core.policies.form_policy import FormPolicy
+
     action_executed_eval_store = EvaluationStore()
 
-    action, policy, confidence = processor.predict_next_action(partial_tracker)
+    gold = event.action_name
 
+    action, policy, confidence = processor.predict_next_action(partial_tracker)
     predicted = action.name()
-    gold = event.action_name
+
+    if predicted != gold and FormPolicy.__name__ in policy:
+        # FormPolicy predicted wrong action
+        # but it might be Ok if form action is rejected
+        _emulate_form_rejection(processor, partial_tracker)
+        # try again
+        action, policy, confidence = processor.predict_next_action(
+            partial_tracker)
+        predicted = action.name()
 
     action_executed_eval_store.add_to_store(action_predictions=predicted,
                                             action_targets=gold)
@@ -302,9 +330,15 @@ def _collect_action_executed_predictions(processor, partial_tracker, event,
                                                       event.confidence,
                                                       event.timestamp))
         if fail_on_prediction_errors:
-            raise ValueError(
-                "Model predicted a wrong action. Failed Story: "
-                "\n\n{}".format(partial_tracker.export_stories()))
+            error_msg = ("Model predicted a wrong action. Failed Story: "
+                         "\n\n{}".format(partial_tracker.export_stories()))
+            if FormPolicy.__name__ in policy:
+                error_msg += ("FormAction is not run during "
+                              "evaluation therefore it is impossible to know "
+                              "if validation failed or this story is wrong. "
+                              "If the story is correct, add it to the "
+                              "training stories and retrain.")
+            raise ValueError(error_msg)
     else:
         partial_tracker.update(event)
 


Code review comments:
diff_hunk: @@ -15,10 +15,10 @@ train-core:
 	python -m rasa.core.train -s data/stories.md -d domain.yml -o models/dialogue --debug
 
 run-core:
-	python -m rasa.core.run --core models/dialogue --nlu models/nlu/current --verbose --endpoints endpoints.yml
+	python -m rasa.core.run --core models/dialogue --verbose --endpoints endpoints.yml
path: examples/formbot/Makefile
text: @user1:
this is not identical -- why did you remove the nlu model :D 

@author:
🙈good spot