Repository: rasahq/rasa
Issue: Response Selector evaluation fails when testing on a test split

Rasa Version     : 1.11.0a1
Rasa SDK Version : 1.10.0
Rasa X Version   : None
Python Version   : 3.7.7 (default, May  5 2020, 13:33:17) 
Operating System : Linux-4.15.0-1044-gke-x86_64-with-debian-buster-sid

**Issue**:
NLU test fails with dataset `Carbon Bot` and configuration file `Sparse + DIET + ResponseSelector(T2T)`.

**Error (including full traceback)**:
```
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Response selection evaluation results:
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Response Selection Evaluation: Only considering those 151 examples that have a defined response out of 515 examples.
/home/github/.cache/pypoetry/virtualenvs/rasa-6QvRaJbr-py3.7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/home/github/.cache/pypoetry/virtualenvs/rasa-6QvRaJbr-py3.7/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Classification report saved to results/public/carbon-bot/en/cvf_diet_responset2t.yml/response_selection_report.json.
2020-05-29 09:24:09 INFO     rasa.nlu.test  - Incorrect response predictions saved to results/public/carbon-bot/en/cvf_diet_responset2t.yml/response_selection_errors.json.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/github/rasa/__main__.py", line 108, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/github/rasa/cli/test.py", line 153, in run_nlu_test
    test_nlu(model_path, nlu_data, output, vars(args))
  File "/home/github/rasa/test.py", line 174, in test_nlu
    run_evaluation(nlu_data, nlu_model, output_directory=output_directory, **kwargs)
  File "/home/github/rasa/nlu/test.py", line 1562, in run_evaluation
    disable_plotting,
  File "/home/github/rasa/nlu/test.py", line 518, in evaluate_response_selections
    _labels = [response_to_intent_target[label] for label in labels]
  File "/home/github/rasa/nlu/test.py", line 518, in <listcomp>
    _labels = [response_to_intent_target[label] for label in labels]
KeyError: 'More and more companies and individuals are voluntarily offsetting their carbon emissions. According to goldstandard.org, individual offsets grew fourfold last year.'
```

**Command or request that led to error**:
```
rasa train nlu -u dataset/$DATASET/train -c dataset/configs/$CONFIG --out models/$DATASET/$CONFIG
rasa test nlu -u dataset/$DATASET/test -m models/$DATASET/$CONFIG --out results/$DATASET/$CONFIG
```


Changed files:
rasa/nlu/test.py

diff --git a/rasa/nlu/test.py b/rasa/nlu/test.py
index 8b1c93ba829..a0effebb026 100644
--- a/rasa/nlu/test.py
+++ b/rasa/nlu/test.py
@@ -450,7 +450,12 @@ def evaluate_response_selections(
             confusion_matrix_filename = os.path.join(
                 output_directory, confusion_matrix_filename
             )
-        _labels = [response_to_intent_target[label] for label in labels]
+        _labels = [
+            response_to_intent_target[label]
+            if label in response_to_intent_target
+            else f"'{label[:20]}...' (response not present in test data)"
+            for label in labels
+        ]
         plot_utils.plot_confusion_matrix(
             confusion_matrix,
             classes=_labels,