Repository: RasaHQ/rasa
Pull request: Do not filter training data in model.py but on component side
**Proposed changes**:
closes #6919 

**Status (please check what you already did)**:
- [ ] added some tests for the functionality
- [ ] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
sklearn.exceptions.NotFittedError: Vocabulary not fitted or provided
**Rasa version**: 2.0.0rc4

**Issue**:
Core training fails on the demo bot (`rasa init`) after changing the config to the one below.

**Error (including full traceback)**:
```
(rasa) ➜  ~/Repositories/rasa/demo_bot git:(core-entity-roles-groups) ✔ rasa train
The configuration for policies was chosen automatically. It was written into the config file at 'config.yml'.
Training NLU model...
2020-10-06 09:34:09 INFO     rasa.shared.nlu.training_data.training_data  - Training data stats:
2020-10-06 09:34:09 INFO     rasa.shared.nlu.training_data.training_data  - Number of intent examples: 69 (7 distinct intents)

2020-10-06 09:34:09 INFO     rasa.shared.nlu.training_data.training_data  -   Found intents: 'deny', 'mood_great', 'mood_unhappy', 'affirm', 'greet', 'bot_challenge', 'goodbye'
2020-10-06 09:34:09 INFO     rasa.shared.nlu.training_data.training_data  - Number of response examples: 0 (6 distinct responses)
2020-10-06 09:34:09 INFO     rasa.shared.nlu.training_data.training_data  - Number of entity examples: 0 (0 distinct entities)
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component RegexEntityExtractor
/home/tanja/Repositories/rasa/rasa/shared/utils/io.py:88: UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
/home/tanja/.virtualenv/rasa/lib/python3.6/site-packages/sklearn/feature_extraction/text.py:501: UserWarning: The parameter 'token_pattern' will not be used since 'analyzer' != 'word'
  warnings.warn("The parameter 'token_pattern' will not be used"
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:09 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
/home/tanja/Repositories/rasa/rasa/shared/utils/io.py:88: UserWarning: You specified 'DIET' to train entities, but no entities are present in the training data. Skip training of entities.
Epochs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:18<00:00,  5.46it/s, t_loss=1.504, i_acc=0.986]
2020-10-06 09:34:36 INFO     rasa.utils.tensorflow.models  - Finished training.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
2020-10-06 09:34:36 INFO     rasa.nlu.selectors.response_selector  - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Starting to train component FallbackClassifier
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Finished training component.
2020-10-06 09:34:36 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmp1hnllvoo/nlu'
NLU model training completed.
Training Core model...
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 2265.97it/s, # trackers=1]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 798.51it/s, # trackers=3]
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 269.55it/s, # trackers=12]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 83.32it/s, # trackers=39]
Processed rules: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2965.22it/s, # trackers=1]
Processed trackers: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 911.94it/s, # actions=12]
Processed actions: 12it [00:00, 9763.66it/s, # examples=12]
Processed trackers: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:00<00:00, 230.98it/s, # actions=30]
Traceback (most recent call last):
  File "/home/tanja/.virtualenv/rasa/bin/rasa", line 11, in <module>
    load_entry_point('rasa', 'console_scripts', 'rasa')()
  File "/home/tanja/Repositories/rasa/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/tanja/Repositories/rasa/rasa/cli/train.py", line 90, in train
    nlu_additional_arguments=extract_nlu_additional_arguments(args),
  File "/home/tanja/Repositories/rasa/rasa/train.py", line 55, in train
    loop,
  File "/home/tanja/Repositories/rasa/rasa/utils/common.py", line 300, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/tanja/Repositories/rasa/rasa/train.py", line 110, in train_async
    nlu_additional_arguments=nlu_additional_arguments,
  File "/home/tanja/Repositories/rasa/rasa/train.py", line 207, in _train_async_internal
    old_model_zip_path=old_model,
  File "/home/tanja/Repositories/rasa/rasa/train.py", line 263, in _do_training
    or _interpreter_from_previous_model(old_model_zip_path),
  File "/home/tanja/Repositories/rasa/rasa/train.py", line 409, in _train_core_with_validated_data
    interpreter=interpreter,
  File "/home/tanja/Repositories/rasa/rasa/core/train.py", line 67, in train
    agent.train(training_data, **additional_arguments)
  File "/home/tanja/Repositories/rasa/rasa/core/agent.py", line 724, in train
    training_trackers, self.domain, interpreter=self.interpreter, **kwargs
  File "/home/tanja/Repositories/rasa/rasa/core/policies/ensemble.py", line 189, in train
    trackers_to_train, domain, interpreter=interpreter, **kwargs
  File "/home/tanja/Repositories/rasa/rasa/core/policies/ted_policy.py", line 332, in train
    training_trackers, domain, interpreter, **kwargs
  File "/home/tanja/Repositories/rasa/rasa/core/policies/policy.py", line 165, in featurize_for_training
    training_trackers, domain, interpreter
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/tracker_featurizers.py", line 140, in featurize_trackers
    tracker_state_features = self._featurize_states(trackers_as_states, interpreter)
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/tracker_featurizers.py", line 73, in _featurize_states
    for tracker_states in trackers_as_states
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/tracker_featurizers.py", line 73, in <listcomp>
    for tracker_states in trackers_as_states
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/tracker_featurizers.py", line 71, in <listcomp>
    for state in tracker_states
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/single_state_featurizer.py", line 201, in encode_state
    self._extract_state_features(sub_state, interpreter, sparse=True)
  File "/home/tanja/Repositories/rasa/rasa/core/featurizers/single_state_featurizer.py", line 169, in _extract_state_features
    parsed_message = interpreter.featurize_message(message)
  File "/home/tanja/Repositories/rasa/rasa/core/interpreter.py", line 158, in featurize_message
    result = self.interpreter.featurize_message(message)
  File "/home/tanja/Repositories/rasa/rasa/nlu/model.py", line 418, in featurize_message
    component.process(message, **self.context)
  File "/home/tanja/Repositories/rasa/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py", line 562, in process
    attribute, [message_tokens]
  File "/home/tanja/Repositories/rasa/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py", line 438, in _create_features
    seq_vec = self.vectorizers[attribute].transform(tokens)
  File "/home/tanja/.virtualenv/rasa/lib/python3.6/site-packages/sklearn/feature_extraction/text.py", line 1247, in transform
    self._check_vocabulary()
  File "/home/tanja/.virtualenv/rasa/lib/python3.6/site-packages/sklearn/feature_extraction/text.py", line 467, in _check_vocabulary
    raise NotFittedError("Vocabulary not fitted or provided")
sklearn.exceptions.NotFittedError: Vocabulary not fitted or provided
```

**Command or request that led to error**:
```
rasa train
```

**Content of configuration file (config.yml)** (if relevant):
```yml
language: en

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexEntityExtractor
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
  - name: FallbackClassifier
    threshold: 0.3
    ambiguity_threshold: 0.1

policies:
```

**Content of domain file (domain.yml)** (if relevant):
```yml

```

Hints from issue discussion:
If you move the `RegexFeaturizer` after the `CountVectorsFeaturizer`s, core training works.
> If you move the `RegexFeaturizer` after the `CountVectorsFeaturizer`s, core training works.

You mean `RegexEntityExtractor` right?
Yes, if you move the `RegexEntityExtractor` after the `CountVectorsFeaturizers`, core training works.
This line is causing the error: https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/model.py#L204

Commit message: formatting

Patch to review:
diff --git a/changelog/6919.bugfix.md b/changelog/6919.bugfix.md
new file mode 100644
index 000000000000..d5b2e5d02a0b
--- /dev/null
+++ b/changelog/6919.bugfix.md
@@ -0,0 +1 @@
+Do not filter training data in model.py but on component side.
diff --git a/rasa/nlu/classifiers/diet_classifier.py b/rasa/nlu/classifiers/diet_classifier.py
index 83c5ab87fe7c..ef0525dd10d1 100644
--- a/rasa/nlu/classifiers/diet_classifier.py
+++ b/rasa/nlu/classifiers/diet_classifier.py
@@ -35,6 +35,8 @@
     ENTITY_ATTRIBUTE_GROUP,
     ENTITY_ATTRIBUTE_ROLE,
     NO_ENTITY_TAG,
+    ACTION_TEXT,
+    ACTION_NAME,
 )
 from rasa.nlu.config import RasaNLUModelConfig, InvalidConfigError
 from rasa.shared.nlu.training_data.training_data import TrainingData
@@ -618,6 +620,10 @@ def _create_model_data(
         label_ids = []
 
         for example in training_data:
+            # Do not train on e2e examples
+            if example.is_e2e_message():
+                continue
+
             if label_attribute is None or example.get(label_attribute):
                 text_features = self._extract_features(example, TEXT)
                 for feature_key, feature_value in text_features.items():
diff --git a/rasa/nlu/classifiers/keyword_intent_classifier.py b/rasa/nlu/classifiers/keyword_intent_classifier.py
index 34cd61d772a8..61320a9fcb75 100644
--- a/rasa/nlu/classifiers/keyword_intent_classifier.py
+++ b/rasa/nlu/classifiers/keyword_intent_classifier.py
@@ -46,7 +46,7 @@ def train(
     ) -> None:
 
         duplicate_examples = set()
-        for ex in training_data.training_examples:
+        for ex in training_data.intent_examples:
             if (
                 ex.get(TEXT) in self.intent_keyword_map.keys()
                 and ex.get(INTENT) != self.intent_keyword_map[ex.get(TEXT)]
diff --git a/rasa/nlu/extractors/crf_entity_extractor.py b/rasa/nlu/extractors/crf_entity_extractor.py
index 3204a0c431da..113d4337c8db 100644
--- a/rasa/nlu/extractors/crf_entity_extractor.py
+++ b/rasa/nlu/extractors/crf_entity_extractor.py
@@ -172,9 +172,7 @@ def train(
         self._update_crf_order(training_data)
 
         # filter out pre-trained entity examples
-        entity_examples = self.filter_trainable_entities(
-            training_data.training_examples
-        )
+        entity_examples = self.filter_trainable_entities(training_data.entity_examples)
 
         dataset = [self._convert_to_crf_tokens(example) for example in entity_examples]
 
diff --git a/rasa/nlu/extractors/mitie_entity_extractor.py b/rasa/nlu/extractors/mitie_entity_extractor.py
index ea275834a2de..d3870210a4ba 100644
--- a/rasa/nlu/extractors/mitie_entity_extractor.py
+++ b/rasa/nlu/extractors/mitie_entity_extractor.py
@@ -82,7 +82,7 @@ def train(
 
         # filter out pre-trained entity examples
         filtered_entity_examples = self.filter_trainable_entities(
-            training_data.training_examples
+            training_data.entity_examples
         )
 
         for example in filtered_entity_examples:
diff --git a/rasa/nlu/extractors/regex_entity_extractor.py b/rasa/nlu/extractors/regex_entity_extractor.py
index 8871dd1c80ba..0bc3d57fe13b 100644
--- a/rasa/nlu/extractors/regex_entity_extractor.py
+++ b/rasa/nlu/extractors/regex_entity_extractor.py
@@ -4,7 +4,6 @@
 from typing import Any, Dict, List, Optional, Text
 
 import rasa.shared.utils.io
-import rasa.utils.io as io_utils
 import rasa.nlu.utils.pattern_utils as pattern_utils
 from rasa.nlu.model import Metadata
 from rasa.nlu.config import RasaNLUModelConfig
diff --git a/rasa/nlu/model.py b/rasa/nlu/model.py
index 231c8c26f0da..2314f12b1186 100644
--- a/rasa/nlu/model.py
+++ b/rasa/nlu/model.py
@@ -201,9 +201,6 @@ def train(self, data: TrainingData, **kwargs: Any) -> "Interpreter":
         working_data: TrainingData = copy.deepcopy(data)
 
         for i, component in enumerate(self.pipeline):
-            if isinstance(component, (EntityExtractor, IntentClassifier)):
-                working_data = working_data.without_empty_e2e_examples()
-
             logger.info(f"Starting to train component {component.name}")
             component.prepare_partial_processing(self.pipeline[:i], context)
             updates = component.train(working_data, self.config, **context)
diff --git a/rasa/shared/nlu/training_data/message.py b/rasa/shared/nlu/training_data/message.py
index 63b3afc447a1..bcc272d2ddd6 100644
--- a/rasa/shared/nlu/training_data/message.py
+++ b/rasa/shared/nlu/training_data/message.py
@@ -321,6 +321,20 @@ def _combine_features(
 
         return combined_features
 
+    def is_e2e_message(self) -> bool:
+        return (
+            self.data.get(ACTION_NAME) is not None
+            or self.data.get(ACTION_TEXT)
+            or (
+                (self.data.get(INTENT) or self.data.get(RESPONSE))
+                and not self.data.get(TEXT)
+            )
+            or (
+                self.data.get(TEXT)
+                and not (self.data.get(INTENT) or self.data.get(RESPONSE))
+            )
+        )
+
 
 def ordered(obj: Any) -> Any:
     if isinstance(obj, dict):
diff --git a/tests/shared/nlu/training_data/test_message.py b/tests/shared/nlu/training_data/test_message.py
index 734efed7e1c6..c24a451bd519 100644
--- a/tests/shared/nlu/training_data/test_message.py
+++ b/tests/shared/nlu/training_data/test_message.py
@@ -11,6 +11,8 @@
     FEATURE_TYPE_SEQUENCE,
     ACTION_TEXT,
     ACTION_NAME,
+    INTENT,
+    RESPONSE,
 )
 import rasa.shared.nlu.training_data.message
 from rasa.shared.nlu.training_data.message import Message
@@ -264,3 +266,20 @@ def test_build_from_action():
     assert Message.build_from_action(
         action_text=test_action_text, action_name=test_action_name
     ) == Message(data={ACTION_NAME: test_action_name, ACTION_TEXT: test_action_text})
+
+
+@pytest.mark.parametrize(
+    "message, e2e_message",
+    [
+        (Message({INTENT: "intent", TEXT: "text"}), False),
+        (Message({RESPONSE: "response", TEXT: "text"}), False),
+        (Message({INTENT: "intent"}), True),
+        (Message({ACTION_TEXT: "action text", ACTION_NAME: ""}), True),
+        (Message({ACTION_NAME: "action"}), True),
+        (Message({TEXT: "text"}), True),
+    ],
+)
+def test_is_e2e_message(
+    message: Message, e2e_message: bool,
+):
+    assert e2e_message == message.is_e2e_message()


Code review comments:
diff_hunk: @@ -321,6 +321,20 @@ def _combine_features(
 
         return combined_features
 
+    def is_e2e_message(self) -> bool:
path: rasa/shared/nlu/training_data/message.py
text: @user1:
let's rename it to smth like `is_core_message`, because it's not only about e2e

@user1:
also, since it is public, I think doctoring is needed

diff_hunk: @@ -35,6 +35,8 @@
     ENTITY_ATTRIBUTE_GROUP,
     ENTITY_ATTRIBUTE_ROLE,
     NO_ENTITY_TAG,
+    ACTION_TEXT,
+    ACTION_NAME,
path: rasa/nlu/classifiers/diet_classifier.py
text: @user1:
this seems to be not used

@user1:
```suggestion
```