Repository: RasaHQ/rasa
Pull request: Fallback
**Proposed changes**:
- fixes #3187 

**Status (please check what you already did)**:
- [x] made PR ready for code review
- [ ] added some tests for the functionality
- [ ] updated the documentation
- [ ] updated the changelog


Problem statement:
evaluating core stories with two stage fallback causes an error
<!-- THIS INFORMATION IS MANDATORY - YOUR ISSUE WILL BE CLOSED IF IT IS MISSING. If you don't know your Rasa Core version, use `pip list | grep rasa`. Removing the below information is allowed for FEATURE REQUESTS. --> 

**Rasa version**: `master`

**Python version**: `3.6`

**Operating system** (windows, osx, ...): `mac`

**Issue**:
evaluating core stories with two stage fallback causes an error:
```
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ghost/Documents/rasa/rasa_core/rasa_core/evaluate.py", line 686, in <module>
    cmdline_arguments.e2e)
  File "/Users/ghost/Documents/rasa/rasa_core/rasa_core/evaluate.py", line 511, in run_story_evaluation
    use_e2e)
  File "/.../core/test.py", line 470, in collect_story_predictions
    in_training_data_fraction = _in_training_data_fraction(action_list)
  File "/...core/test.py", line 425, in _in_training_data_fraction
    a["action"] for a in action_list
  File "/.../core/test.py", line 426, in <listcomp>
    if not SimplePolicyEnsemble.is_not_memo_policy(a["policy"])
  File "/.../core/policies/ensemble.py", line 306, in is_not_memo_policy
    is_memo = best_policy_name.endswith(
AttributeError: 'NoneType' object has no attribute 'endswith'
```

Hints from issue discussion:

Commit message: Update test.py

Patch to review:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 8df6133e6ab5..8f4871976242 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -32,6 +32,7 @@ Removed
 
 Fixed
 -----
+-evaluating core stories with two stage fallback gave an error, trying to handle None for a policy
 - the ``/evaluate`` route for the Rasa NLU server now runs evaluation
   in a parallel process, which prevents the currently loaded model unloading
 - added missing implementation of the ``keys()`` function for the Redis Tracker
diff --git a/rasa/core/test.py b/rasa/core/test.py
index 7e4a29f83a4a..ed7180ed27ee 100644
--- a/rasa/core/test.py
+++ b/rasa/core/test.py
@@ -411,7 +411,7 @@ def _in_training_data_fraction(action_list):
     in_training_data = [
         a["action"]
         for a in action_list
-        if not SimplePolicyEnsemble.is_not_memo_policy(a["policy"])
+        if a["policy"] and not SimplePolicyEnsemble.is_not_memo_policy(a["policy"])
     ]
 
     return len(in_training_data) / len(action_list)


Code review comments:
diff_hunk: @@ -32,6 +32,7 @@ Removed
 
 Fixed
 -----
+-evaluating core stories with two stage fallback gave an error, trying to handle None for a policy
path: CHANGELOG.rst
text: @user1:
```suggestion
- evaluating core stories with two stage fallback gave an error, trying to handle None for a policy
```