Repository: RasaHQ/rasa
Pull request: Updated featurizers
**Proposed changes**:
- Add option ``use_cls_token`` to all tokenizers. If it is set to ``True``, the token ``__CLS__`` will be added to the end of the list of tokens.
- Add option ``return_sequence`` to all featurizers. By default all featurizers return a matrix of size
(1 x feature-dimension). If the option ``return_sequence`` is set to ``True``, the corresponding featurizer will return a matrix of size (token-length x feature-dimension).
- Split featurizers into sparse and dense featurizers.
- Remove ``NGramFeaturizer``. Please use ``CountVectorsFeaturizer`` instead.
- To use custom features in the ``CRFEntityExtractor`` use ``text_dense_features`` instead of ``ner_features``. If ``text_dense_features`` are present in the feature set, the ``CRFEntityExtractor`` will automatically make use of them. Just make sure to add a dense featurizer in front of the ``CRFEntityExtractor`` in your pipeline and set the flag ``return_sequence`` to ``True`` for that featurizer.

closes https://github.com/RasaHQ/rasa/issues/4957
part of https://github.com/RasaHQ/research/issues/54

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
Replace ner_features by text_dense_features
If you wanted to use additional features in the CRFEntityExtractor you had to add ``ner_features`` to the feature list of the entity extractor. Additionally, you need a featurizer that actually sets those features. The main difference of those features are, that they should come as a sequence.

With the upcoming changes it will be possible to easily define for every featruizer whether it should return its features as sequence or not. Also, we are planning to add more featurizers that return embeddings from pre-trained language models. 

To be able to easily make use of any of those features, we should exchange the logic of ``ner_features``. You should be able to use the features any featurizer, that returns word embeddings, in the CRFEntityExtractor. Those features will be stored in the attribute ``text_dense_features`` of the message. The featurizer needs to return a sequence of features. Otherwise, the features cannot be used and a warning should be printed.

Hints from issue discussion:

Commit message: update model version

Patch to review:
diff --git a/changelog/4935.feature.rst b/changelog/4935.feature.rst
new file mode 100644
index 000000000000..33c5ee7cd50e
--- /dev/null
+++ b/changelog/4935.feature.rst
@@ -0,0 +1,13 @@
+Preparation for an upcoming change in the ``EmbeddingIntentClassifier``:
+
+Add option ``use_cls_token`` to all tokenizers. If it is set to ``True``, the token ``__CLS__`` will be added to
+the end of the list of tokens. Default is set to ``False``. No need to change the default value for now.
+
+Add option ``return_sequence`` to all featurizers. By default all featurizers return a matrix of size
+(1 x feature-dimension). If the option ``return_sequence`` is set to ``True``, the corresponding featurizer will return
+a matrix of size (token-length x feature-dimension). See https://rasa.com/docs/rasa/nlu/components/#featurizers.
+Default value is set to ``False``. However, you might want to set it to ``True`` if you want to use custom features
+in the ``CRFEntityExtractor``.
+See https://rasa.com/docs/rasa/nlu/entity-extraction/#passing-custom-features-to-crfentityextractor.
+
+These changes are model breaking!
\ No newline at end of file
diff --git a/changelog/4935.removal.rst b/changelog/4935.removal.rst
new file mode 100644
index 000000000000..dd1b98565419
--- /dev/null
+++ b/changelog/4935.removal.rst
@@ -0,0 +1,8 @@
+Remove ``NGramFeaturizer``. Please use ``CountVectorsFeaturizer`` instead.
+The following settings match the previous ``NGramFeaturizer``:
+- name: 'CountVectorsFeaturizer'
+  analyzer: 'char_wb'
+  min_ngram: 3
+  max_ngram: 17
+  max_features: 10
+  min_df: 5
\ No newline at end of file
diff --git a/changelog/4957.removal.rst b/changelog/4957.removal.rst
new file mode 100644
index 000000000000..3ae9e979f438
--- /dev/null
+++ b/changelog/4957.removal.rst
@@ -0,0 +1,5 @@
+To use custom features in the ``CRFEntityExtractor`` use ``text_dense_features`` instead of ``ner_features``. If
+``text_dense_features`` are present in the feature set, the ``CRFEntityExtractor`` will automatically make use of
+them. Just make sure to add a dense featurizer in front of the ``CRFEntityExtractor`` in your pipeline and set the
+flag ``return_sequence`` to ``True`` for that featurizer.
+See https://rasa.com/docs/rasa/nlu/entity-extraction/#passing-custom-features-to-crfentityextractor.
\ No newline at end of file
diff --git a/docs/api/featurization.rst b/docs/api/core-featurization.rst
similarity index 98%
rename from docs/api/featurization.rst
rename to docs/api/core-featurization.rst
index 90021cb68b07..2267b6882f80 100644
--- a/docs/api/featurization.rst
+++ b/docs/api/core-featurization.rst
@@ -1,10 +1,10 @@
 :desc: Find out how to apply machine learning algorithms to conversational AI
        using vector representations of conversations with Rasa.
 
-.. _featurization:
+.. _featurization_conversations:
 
-Featurization
-==============
+Featurization of Conversations
+==============================
 
 .. edit-link::
 
diff --git a/docs/core/policies.rst b/docs/core/policies.rst
index 1934deb67a66..6285df3d2ea8 100644
--- a/docs/core/policies.rst
+++ b/docs/core/policies.rst
@@ -70,7 +70,7 @@ in the policy configuration yaml file.
 
     Only the ``MaxHistoryTrackerFeaturizer`` uses a max history,
     whereas the ``FullDialogueTrackerFeaturizer`` always looks at
-    the full conversation history. See :ref:`featurization` for details.
+    the full conversation history. See :ref:`featurization_conversations` for details.
 
 As an example, let's say you have an ``out_of_scope`` intent which
 describes off-topic user messages. If your bot sees this intent multiple
@@ -218,7 +218,7 @@ following steps:
 
 It is recommended to use
 ``state_featurizer=LabelTokenizerSingleStateFeaturizer(...)``
-(see :ref:`featurization` for details).
+(see :ref:`featurization_conversations` for details).
 
 **Configuration:**
 
@@ -308,7 +308,7 @@ It is recommended to use
         Default ``max_history`` for this policy is ``None`` which means it'll use
         the ``FullDialogueTrackerFeaturizer``. We recommend to set ``max_history`` to
         some finite value in order to use ``MaxHistoryTrackerFeaturizer``
-        for **faster training**. See :ref:`featurization` for details.
+        for **faster training**. See :ref:`featurization_conversations` for details.
         We recommend to increase ``batch_size`` for ``MaxHistoryTrackerFeaturizer``
         (e.g. ``"batch_size": [32, 64]``)
 
diff --git a/docs/index.rst b/docs/index.rst
index 79df7016a952..9b8e824f90d2 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -91,7 +91,7 @@ Understand messages, hold conversations, and connect to messaging channels and A
    api/event-brokers
    api/lock-stores
    api/training-data-importers
-   api/featurization
+   api/core-featurization
    migration-guide
    changelog
 
diff --git a/docs/migration-guide.rst b/docs/migration-guide.rst
index fe933e449a30..a8776048f36c 100644
--- a/docs/migration-guide.rst
+++ b/docs/migration-guide.rst
@@ -37,7 +37,7 @@ General
 - Default ``max_history`` for ``EmbeddingPolicy`` is ``None`` which means it'll use
   the ``FullDialogueTrackerFeaturizer``. We recommend to set ``max_history`` to
   some finite value in order to use ``MaxHistoryTrackerFeaturizer``
-  for **faster training**. See :ref:`featurization` for details.
+  for **faster training**. See :ref:`featurization_conversations` for details.
   We recommend to increase ``batch_size`` for ``MaxHistoryTrackerFeaturizer``
   (e.g. ``"batch_size": [32, 64]``)
 - **Compare** mode of ``rasa train core`` allows the whole core config comparison.
diff --git a/docs/nlu/components.rst b/docs/nlu/components.rst
index 4b6b004300f3..3557f4c12ff9 100644
--- a/docs/nlu/components.rst
+++ b/docs/nlu/components.rst
@@ -82,8 +82,22 @@ SpacyNLP
           # between these two words, therefore setting this to `true`.
           case_sensitive: false
 
-Featurizers
------------
+Text Featurizers
+----------------
+
+Text featurizers are divided into two different categories: sparse featurizers and dense featurizers.
+Sparse featurizers are featurizers that return feature vectors with a lot of missing values, e.g. zeros.
+As those feature vectors would normally take up a lot of memory, we store them as sparse features.
+Sparse features only store the values that are non zero and their positions in the vector.
+Thus, we save a lot of memroy and are able to train on larger datasets.
+
+By default all featurizers will return a matrix of length (1 x feature-dimension).
+All featurizer (except the ``ConveRTFeaturizer``) have the option to return a sequence instead.
+In case the flag ``"return_sequence"`` is set to ``True`` the returned matrix of a featurizer will have the size
+(token-length x feature-dimension).
+So, the returned matrix will have an entry for every token.
+Otherwise, the matrix will just have one entry for the complete utterance.
+
 
 MitieFeaturizer
 ~~~~~~~~~~~~~~~
@@ -91,6 +105,7 @@ MitieFeaturizer
 :Short: MITIE intent featurizer
 :Outputs: nothing, used as an input to intent classifiers that need intent features (e.g. ``SklearnIntentClassifier``)
 :Requires: :ref:`MitieNLP`
+:Type: Dense featurizer
 :Description:
     Creates feature for intent classification using the MITIE featurizer.
 
@@ -114,40 +129,53 @@ SpacyFeaturizer
 :Short: spacy intent featurizer
 :Outputs: nothing, used as an input to intent classifiers that need intent features (e.g. ``SklearnIntentClassifier``)
 :Requires: :ref:`SpacyNLP`
+:Type: Dense featurizer
 :Description:
     Creates feature for intent classification using the spacy featurizer.
-    Optionally adds word vectors for each ``token`` to ``ner_features``, which can be referenced in ``CRFEntityExtractor``
 :Configuration:
 
     .. code-block:: yaml
 
         pipeline:
         - name: "SpacyFeaturizer"
-          # Whether to add word vectors to ``ner_features`` (default: False)
-          ner_feature_vectors: True
 
-NGramFeaturizer
-~~~~~~~~~~~~~~~
 
-:Short: Appends char-ngram features to feature vector
-:Outputs: nothing, appends its features to an existing feature vector generated by another intent featurizer
-:Requires: :ref:`SpacyNLP`
+ConveRTFeaturizer
+~~~~~~~~~~~~~~~~~
+
+:Short:
+    Creates a vector representation of user message and response (if specified) using
+    `ConveRT <https://github.com/PolyAI-LDN/polyai-models>`_ model.
+:Outputs:
+    nothing, used as an input to intent classifiers and response selectors that need intent features and response
+    features respectively (e.g. ``EmbeddingIntentClassifier`` and ``ResponseSelector``)
+:Requires: nothing
+:Type: Dense featurizer
 :Description:
-    This featurizer appends character ngram features to a feature vector. During training the component looks for the
-    most common character sequences (e.g. ``app`` or ``ing``). The added features represent a boolean flag if the
-    character sequence is present in the word sequence or not.
+    Creates features for intent classification and response selection.
+    Uses the `default signature <https://github.com/PolyAI-LDN/polyai-models#tfhub-signatures>`_ to compute vector
+    representations of input text.
 
-    .. note:: There needs to be another intent featurizer previous to this one in the pipeline!
+    .. warning::
+        Since ``ConveRT`` model is trained only on an english corpus of conversations, this featurizer should only
+        be used if your training data is in english language.
+
+    .. note::
+        To use ``ConveRTFeaturizer`` you need to install additional tensorflow libraries (``tensorflow_text`` and
+        ``tensorflow_hub``). You should do a pip install of Rasa with ``pip install rasa[convert]`` to install those.
+
+    .. warning::
+        If you set the option ``"return_sequence"`` to ``True``, Rasa will raise an error informing you that this
+        option is currently not supported. Do not use this featurizer in combination with any other featurizer that
+        has the option ``"return_sequence"`` set to ``True`` as training will fail. However, you can use this
+        featurizer with any other featurizer as long as ``"return_sequence"`` is set to ``False`` for all of them.  
 
 :Configuration:
 
     .. code-block:: yaml
 
         pipeline:
-        - name: "NGramFeaturizer"
-          # Maximum number of ngrams to use when augmenting
-          # feature vectors with character ngrams
-          max_number_of_ngrams: 10
+        - name: "ConveRTFeaturizer"
 
 
 RegexFeaturizer
@@ -156,12 +184,15 @@ RegexFeaturizer
 :Short: regex feature creation to support intent and entity classification
 :Outputs: ``text_features`` and ``tokens.pattern``
 :Requires: nothing
+:Type: Sparse featurizer
 :Description:
-    During training, the regex intent featurizer creates a list of `regular expressions` defined in the training data format.
-    For each regex, a feature will be set marking whether this expression was found in the input, which will later be fed into intent classifier / entity
-    extractor to simplify classification (assuming the classifier has learned during the training phase, that this set
-    feature indicates a certain intent). Regex features for entity extraction are currently only supported by the
-    ``CRFEntityExtractor`` component!
+    Creates features for entity extraction and intent classification.
+    During training, the regex intent featurizer creates a list of `regular expressions` defined in the training
+    data format.
+    For each regex, a feature will be set marking whether this expression was found in the input, which will later
+    be fed into intent classifier / entity extractor to simplify classification (assuming the classifier has learned
+    during the training phase, that this set feature indicates a certain intent).
+    Regex features for entity extraction are currently only supported by the ``CRFEntityExtractor`` component!
 
     .. note:: There needs to be a tokenizer previous to this featurizer in the pipeline!
 
@@ -169,13 +200,15 @@ RegexFeaturizer
 CountVectorsFeaturizer
 ~~~~~~~~~~~~~~~~~~~~~~
 
-:Short: Creates bag-of-words representation of user message and label(intent and response) features
+:Short: Creates bag-of-words representation of user message and label (intent and response) features
 :Outputs:
    nothing, used as an input to intent classifiers that
    need bag-of-words representation of intent features
    (e.g. ``EmbeddingIntentClassifier``)
 :Requires: nothing
+:Type: Sparse featurizer
 :Description:
+    Creates features for intent classification and response selection.
     Creates bag-of-words representation of user message and label features using
     `sklearn's CountVectorizer <http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html>`_.
     All tokens which consist only of digits (e.g. 123 and 99 but not a123d) will be assigned to the same feature.
@@ -208,25 +241,31 @@ CountVectorsFeaturizer
 
         Since the training is performed on limited vocabulary data, it cannot be guaranteed that during prediction
         an algorithm will not encounter an unknown word (a word that were not seen during training).
-        In order to teach an algorithm how to treat unknown words, some words in training data can be substituted by generic word ``OOV_token``.
+        In order to teach an algorithm how to treat unknown words, some words in training data can be substituted
+        by generic word ``OOV_token``.
         In this case during prediction all unknown words will be treated as this generic word ``OOV_token``.
 
-        For example, one might create separate intent ``outofscope`` in the training data containing messages of different number of ``OOV_token`` s and
-        maybe some additional general words. Then an algorithm will likely classify a message with unknown words as this intent ``outofscope``.
+        For example, one might create separate intent ``outofscope`` in the training data containing messages of
+        different number of ``OOV_token`` s and maybe some additional general words.
+        Then an algorithm will likely classify a message with unknown words as this intent ``outofscope``.
 
         .. note::
             This featurizer creates a bag-of-words representation by **counting** words,
             so the number of ``OOV_token`` in the sentence might be important.
 
-            - ``OOV_token`` set a keyword for unseen words; if training data contains ``OOV_token`` as words in some messages,
-              during prediction the words that were not seen during training will be substituted with provided ``OOV_token``;
-              if ``OOV_token=None`` (default behaviour) words that were not seen during training will be ignored during prediction time;
-            - ``OOV_words`` set a list of words to be treated as ``OOV_token`` during training; if a list of words that should be treated
-              as Out-Of-Vacabulary is known, it can be set to ``OOV_words`` instead of manually changing it in trainig data or using custom preprocessor.
+            - ``OOV_token`` set a keyword for unseen words; if training data contains ``OOV_token`` as words in some
+              messages, during prediction the words that were not seen during training will be substituted with
+              provided ``OOV_token``; if ``OOV_token=None`` (default behaviour) words that were not seen during
+              training will be ignored during prediction time;
+            - ``OOV_words`` set a list of words to be treated as ``OOV_token`` during training; if a list of words
+              that should be treated as Out-Of-Vacabulary is known, it can be set to ``OOV_words`` instead of manually
+              changing it in trainig data or using custom preprocessor.
 
         .. note::
-            Providing ``OOV_words`` is optional, training data can contain ``OOV_token`` input manually or by custom additional preprocessor.
-            Unseen words will be substituted with ``OOV_token`` **only** if this token is present in the training data or ``OOV_words`` list is provided.
+            Providing ``OOV_words`` is optional, training data can contain ``OOV_token`` input manually or by custom
+            additional preprocessor.
+            Unseen words will be substituted with ``OOV_token`` **only** if this token is present in the training
+            data or ``OOV_words`` list is provided.
 
     Sharing Vocabulary between user message and labels:
 
@@ -273,29 +312,6 @@ CountVectorsFeaturizer
           OOV_words: []  # list of strings
 
 
-ConveRTFeaturizer
-~~~~~~~~~~~~~~~~~
-
-:Short: Creates a vector representation of user message and response (if specified) using `ConveRT <https://github.com/PolyAI-LDN/polyai-models>`_ model.
-:Outputs: nothing, used as an input to intent classifiers and response selectors that need intent features and response features respectively(e.g. ``EmbeddingIntentClassifier`` and ``ResponseSelector``)
-:Requires: nothing
-:Description:
-    Creates features for intent classification and response selection.
-    Uses the `default signature <https://github.com/PolyAI-LDN/polyai-models#tfhub-signatures>`_ to compute vector representations of input text.
-
-    .. warning::
-        Since ``ConveRT`` model is trained only on an english corpus of conversations, this featurizer should only be used if your training data is in english language.
-
-    .. note::
-        To use ``ConveRTFeaturizer`` you should install ``tensorflow_text==1.15.1`` and ``tensorflow_hub==0.6.0``. Otherwise, you can also do a pip install of Rasa with ``pip install rasa[convert]``
-
-:Configuration:
-
-    .. code-block:: yaml
-
-        pipeline:
-        - name: "ConveRTFeaturizer"
-
 Intent Classifiers
 ------------------
 
@@ -303,7 +319,9 @@ Intent Classifiers
 MitieIntentClassifier
 ~~~~~~~~~~~~~~~~~~~~~
 
-:Short: MITIE intent classifier (using a `text categorizer <https://github.com/mit-nlp/MITIE/blob/master/examples/python/text_categorizer_pure_model.py>`_)
+:Short:
+    MITIE intent classifier (using a
+    `text categorizer <https://github.com/mit-nlp/MITIE/blob/master/examples/python/text_categorizer_pure_model.py>`_)
 :Outputs: ``intent``
 :Requires: A tokenizer and a featurizer
 :Output-Example:
@@ -316,7 +334,8 @@ MitieIntentClassifier
 
 :Description:
     This classifier uses MITIE to perform intent classification. The underlying classifier
-    is using a multi-class linear SVM with a sparse linear kernel (see `MITIE trainer code <https://github.com/mit-nlp/MITIE/blob/master/mitielib/src/text_categorizer_trainer.cpp#L222>`_).
+    is using a multi-class linear SVM with a sparse linear kernel (see
+    `MITIE trainer code <https://github.com/mit-nlp/MITIE/blob/master/mitielib/src/text_categorizer_trainer.cpp#L222>`_).
 
 :Configuration:
 
@@ -352,7 +371,8 @@ SklearnIntentClassifier
 :Description:
     The sklearn intent classifier trains a linear SVM which gets optimized using a grid search. In addition
     to other classifiers it also provides rankings of the labels that did not "win". The spacy intent classifier
-    needs to be preceded by a featurizer in the pipeline. This featurizer creates the features used for the classification.
+    needs to be preceded by a featurizer in the pipeline. This featurizer creates the features used for the
+    classification.
 
 :Configuration:
     During the training of the SVM a hyperparameter search is run to
@@ -475,7 +495,8 @@ EmbeddingIntentClassifier
 
     .. note:: For ``cosine`` similarity ``mu_pos`` and ``mu_neg`` should be between ``-1`` and ``1``.
 
-    .. note:: There is an option to use linearly increasing batch size. The idea comes from `<https://arxiv.org/abs/1711.00489>`_.
+    .. note:: There is an option to use linearly increasing batch size. The idea comes from
+              `<https://arxiv.org/abs/1711.00489>`_.
               In order to do it pass a list to ``batch_size``, e.g. ``"batch_size": [64, 256]`` (default behaviour).
               If constant ``batch_size`` is required, pass an ``int``, e.g. ``"batch_size": 64``.
 
@@ -511,7 +532,8 @@ KeywordIntentClassifier
 
 :Description:
     This classifier works by searching a message for keywords.
-    The matching is case sensitive by default and searches only for exact matches of the keyword-string in the user message.
+    The matching is case sensitive by default and searches only for exact matches of the keyword-string in the user
+    message.
     The keywords for an intent are the examples of that intent in the NLU training data.
     This means the entire example is the keyword, not the individual words in the example.
 
@@ -805,8 +827,10 @@ CRFEntityExtractor
     and the states are entity classes. Features of the words (capitalisation, POS tagging,
     etc.) give probabilities to certain entity classes, as are transitions between
     neighbouring entity tags: the most likely set of tags is then calculated and returned.
-    If POS features are used (pos or pos2), spaCy has to be installed. To use custom features
-    made available by Featurizers, use ``"ner_features"``.
+    If POS features are used (pos or pos2), spaCy has to be installed. If you want to use
+    additional features, such as pre-trained word embeddings, from any provided dense
+    featurizer, use ``"text_dense_features"``. Make sure to set ``"return_sequence"`` to
+    ``True`` in the corresponding featurizer.
 :Configuration:
    .. code-block:: yaml
 
@@ -820,7 +844,7 @@ CRFEntityExtractor
           # Available features are:
           # ``low``, ``title``, ``suffix5``, ``suffix3``, ``suffix2``,
           # ``suffix1``, ``pos``, ``pos2``, ``prefix5``, ``prefix2``,
-          # ``bias``, ``upper``, ``digit``, ``pattern``, and ``ner_features``
+          # ``bias``, ``upper``, ``digit``, ``pattern``, and ``text_dense_features``
           features: [["low", "title"], ["bias", "suffix3"], ["upper", "pos", "pos2"]]
 
           # The flag determines whether to use BILOU tagging or not. BILOU
diff --git a/docs/nlu/entity-extraction.rst b/docs/nlu/entity-extraction.rst
index 35edf433e665..c76305d5e0e1 100644
--- a/docs/nlu/entity-extraction.rst
+++ b/docs/nlu/entity-extraction.rst
@@ -154,7 +154,12 @@ as a postprocessing step after receiving the response from Rasa NLU.
 Passing Custom Features to ``CRFEntityExtractor``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If you want to pass custom features to ``CRFEntityExtractor``, you can create a ``Featurizer`` that provides ``ner_features``.
-If you do, ``ner_features`` should be an iterable of ``len(tokens)``, where each entry is a vector.
-If ``CRFEntityExtractor`` finds ``"ner_features"`` in one of the arrays in ``features`` in the config, it will pass the ``ner_features`` vectors to ``sklearn_crfsuite``.
-The simplest example of this is to pass word vectors as features, which you can do using :ref:``SpacyFeaturizer``.
+If you want to pass custom features, such as pre-trained word embeddings, to ``CRFEntityExtractor``, you can
+add any dense featurizer (except ``ConveRTFeaturizer``) to the pipeline before the ``CRFEntityExtractor``.
+Make sure to set ``"return_sequence"`` to ``True`` for the corresponding dense featurizer.
+``CRFEntityExtractor`` automatically finds the additional dense features and checks if the dense features are an
+iterable of ``len(tokens)``, where each entry is a vector.
+A warning will be shown in case the check fails.
+However, ``CRFEntityExtractor`` will continue to train just without the additional custom features.
+In case dense features are present, ``CRFEntityExtractor`` will pass the dense features to ``sklearn_crfsuite``
+and use them for training.
diff --git a/examples/restaurantbot/config.yml b/examples/restaurantbot/config.yml
index 58e9f0be7209..b06666b8f0dd 100644
--- a/examples/restaurantbot/config.yml
+++ b/examples/restaurantbot/config.yml
@@ -6,23 +6,24 @@ pipeline:
   - name: "SpacyFeaturizer"
   - name: "SklearnIntentClassifier"
   - name: "CRFEntityExtractor"
-    features: [ ["low", "title", "upper"],
+    features: [
+      ["low", "title", "upper"],
       [
-          "bias",
-          "low",
-          "prefix5",
-          "prefix2",
-          "suffix5",
-          "suffix3",
-          "suffix2",
-          "upper",
-          "title",
-          "digit",
-          "pattern",
-          "ner_features",
+        "bias",
+        "low",
+        "prefix5",
+        "prefix2",
+        "suffix5",
+        "suffix3",
+        "suffix2",
+        "upper",
+        "title",
+        "digit",
+        "pattern",
+        "text_dense_features"
       ],
-     ["low", "title", "upper"]]
-
+      ["low", "title", "upper"],
+    ]
   - name: "EntitySynonymMapper"
 
 policies:
diff --git a/rasa/constants.py b/rasa/constants.py
index a270ab695e18..7b88ca8ae8ba 100644
--- a/rasa/constants.py
+++ b/rasa/constants.py
@@ -33,7 +33,7 @@
 CONFIG_MANDATORY_KEYS_NLU = ["language", "pipeline"]
 CONFIG_MANDATORY_KEYS = CONFIG_MANDATORY_KEYS_CORE + CONFIG_MANDATORY_KEYS_NLU
 
-MINIMUM_COMPATIBLE_VERSION = "1.3.0a2"
+MINIMUM_COMPATIBLE_VERSION = "1.6.0a1"
 
 GLOBAL_USER_CONFIG_PATH = os.path.expanduser("~/.config/rasa/global.yml")
 
diff --git a/rasa/core/actions/action.py b/rasa/core/actions/action.py
index cfac96eff227..3e00ff24b73b 100644
--- a/rasa/core/actions/action.py
+++ b/rasa/core/actions/action.py
@@ -19,7 +19,7 @@
 from rasa.nlu.constants import (
     DEFAULT_OPEN_UTTERANCE_TYPE,
     OPEN_UTTERANCE_PREDICTION_KEY,
-    MESSAGE_SELECTOR_PROPERTY_NAME,
+    RESPONSE_SELECTOR_PROPERTY_NAME,
 )
 
 from rasa.core.events import (
@@ -197,7 +197,7 @@ async def run(
         """Query the appropriate response and create a bot utterance with that."""
 
         response_selector_properties = tracker.latest_message.parse_data[
-            MESSAGE_SELECTOR_PROPERTY_NAME
+            RESPONSE_SELECTOR_PROPERTY_NAME
         ]
 
         if self.intent_name_from_action() in response_selector_properties:
diff --git a/rasa/core/policies/embedding_policy.py b/rasa/core/policies/embedding_policy.py
index d599959e77c4..e817f1010171 100644
--- a/rasa/core/policies/embedding_policy.py
+++ b/rasa/core/policies/embedding_policy.py
@@ -252,25 +252,25 @@ def _label_features_for_Y(self, label_ids: "np.ndarray") -> "np.ndarray":
     # noinspection PyPep8Naming
     def _create_session_data(
         self, data_X: "np.ndarray", data_Y: Optional["np.ndarray"] = None
-    ) -> "train_utils.SessionData":
-        """Combine all tf session related data into a named tuple"""
-
+    ) -> "train_utils.SessionDataType":
+        """Combine all tf session related data into dict."""
         if data_Y is not None:
             # training time
             label_ids = self._label_ids_for_Y(data_Y)
             Y = self._label_features_for_Y(label_ids)
-
-            # idea taken from sklearn's stratify split
-            if label_ids.ndim == 2:
-                # for multi-label y, map each distinct row to a string repr
-                # using join because str(row) uses an ellipsis if len(row) > 1000
-                label_ids = np.array([" ".join(row.astype("str")) for row in label_ids])
+            # explicitly add last dimension to label_ids
+            # to track correctly dynamic sequences
+            label_ids = np.expand_dims(label_ids, -1)
         else:
             # prediction time
             label_ids = None
             Y = None
 
-        return train_utils.SessionData(X=data_X, Y=Y, label_ids=label_ids)
+        return {
+            "dialogue_features": [data_X],
+            "bot_features": [Y],
+            "action_ids": [label_ids],
+        }
 
     def _create_tf_bot_embed(self, b_in: "tf.Tensor") -> "tf.Tensor":
         """Create embedding bot vector."""
@@ -331,9 +331,9 @@ def _create_tf_dial(self, a_in) -> Tuple["tf.Tensor", "tf.Tensor"]:
 
     def _build_tf_train_graph(self) -> Tuple["tf.Tensor", "tf.Tensor"]:
         """Bulid train graph using iterator."""
+        # iterator returns a_in, b_in, action_ids
+        self.a_in, self.b_in, _ = self._iterator.get_next()
 
-        # session data are int counts but we need a float tensors
-        self.a_in, self.b_in = self._iterator.get_next()
         if isinstance(self.featurizer, MaxHistoryTrackerFeaturizer):
             # add time dimension if max history featurizer is used
             self.b_in = self.b_in[:, tf.newaxis, :]
@@ -364,23 +364,25 @@ def _build_tf_train_graph(self) -> Tuple["tf.Tensor", "tf.Tensor"]:
         )
 
     # prepare for prediction
-    def _create_tf_placeholders(self, session_data: "train_utils.SessionData") -> None:
+    def _create_tf_placeholders(
+        self, session_data: "train_utils.SessionDataType"
+    ) -> None:
         """Create placeholders for prediction."""
 
         dialogue_len = None  # use dynamic time
         self.a_in = tf.placeholder(
             dtype=tf.float32,
-            shape=(None, dialogue_len, session_data.X.shape[-1]),
+            shape=(None, dialogue_len, session_data["dialogue_features"][0].shape[-1]),
             name="a",
         )
         self.b_in = tf.placeholder(
             dtype=tf.float32,
-            shape=(None, dialogue_len, None, session_data.Y.shape[-1]),
+            shape=(None, dialogue_len, None, session_data["bot_features"][0].shape[-1]),
             name="b",
         )
 
     def _build_tf_pred_graph(
-        self, session_data: "train_utils.SessionData"
+        self, session_data: "train_utils.SessionDataType"
     ) -> "tf.Tensor":
         """Rebuild tf graph for prediction."""
 
@@ -440,7 +442,10 @@ def train(
 
         if self.evaluate_on_num_examples:
             session_data, eval_session_data = train_utils.train_val_split(
-                session_data, self.evaluate_on_num_examples, self.random_seed
+                session_data,
+                self.evaluate_on_num_examples,
+                self.random_seed,
+                label_key="action_ids",
             )
         else:
             eval_session_data = None
@@ -458,7 +463,11 @@ def train(
                 train_init_op,
                 eval_init_op,
             ) = train_utils.create_iterator_init_datasets(
-                session_data, eval_session_data, batch_size_in, self.batch_strategy
+                session_data,
+                eval_session_data,
+                batch_size_in,
+                self.batch_strategy,
+                label_key="action_ids",
             )
 
             self._is_training = tf.placeholder_with_default(False, shape=())
@@ -512,7 +521,9 @@ def continue_training(
                 session_data = self._create_session_data(
                     training_data.X, training_data.y
                 )
-                train_dataset = train_utils.create_tf_dataset(session_data, batch_size)
+                train_dataset = train_utils.create_tf_dataset(
+                    session_data, batch_size, label_key="action_ids"
+                )
                 train_init_op = self._iterator.make_initializer(train_dataset)
                 self.session.run(train_init_op)
 
@@ -535,7 +546,7 @@ def tf_feed_dict_for_prediction(
         data_X = self.featurizer.create_X([tracker], domain)
         session_data = self._create_session_data(data_X)
 
-        return {self.a_in: session_data.X}
+        return {self.a_in: session_data["dialogue_features"][0]}
 
     def predict_action_probabilities(
         self, tracker: "DialogueStateTracker", domain: "Domain"
diff --git a/rasa/nlu/classifiers/embedding_intent_classifier.py b/rasa/nlu/classifiers/embedding_intent_classifier.py
index 50767d9b5a0a..ccaa6cf4ba26 100644
--- a/rasa/nlu/classifiers/embedding_intent_classifier.py
+++ b/rasa/nlu/classifiers/embedding_intent_classifier.py
@@ -1,18 +1,22 @@
 import logging
+
 import numpy as np
 import os
 import pickle
+import scipy.sparse
 import typing
-from typing import Any, Dict, List, Optional, Text, Tuple
+from typing import Any, Dict, List, Optional, Text, Tuple, Union
 import warnings
 
 from rasa.nlu.classifiers import LABEL_RANKING_LENGTH
 from rasa.nlu.components import Component
 from rasa.utils import train_utils
+from rasa.utils.train_utils import SessionDataType
 from rasa.nlu.constants import (
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_VECTOR_FEATURE_NAMES,
+    INTENT_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
+    DENSE_FEATURE_NAMES,
 )
 
 import tensorflow as tf
@@ -50,7 +54,7 @@ class EmbeddingIntentClassifier(Component):
 
     provides = ["intent", "intent_ranking"]
 
-    requires = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = []
 
     # default properties (DOC MARKER - don't remove)
     defaults = {
@@ -74,6 +78,8 @@ class EmbeddingIntentClassifier(Component):
         # set random seed to any int to get reproducible results
         "random_seed": None,
         # embedding parameters
+        # default dense dimension used if no dense features are present
+        "dense_dim": {"text": 512, "intent": 20},
         # dimension size of embedding vectors
         "embed_dim": 20,
         # the type of the similarity
@@ -107,53 +113,9 @@ class EmbeddingIntentClassifier(Component):
     }
     # end default properties (DOC MARKER - don't remove)
 
-    def __init__(
-        self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        inverted_label_dict: Optional[Dict[int, Text]] = None,
-        session: Optional["tf.Session"] = None,
-        graph: Optional["tf.Graph"] = None,
-        message_placeholder: Optional["tf.Tensor"] = None,
-        label_placeholder: Optional["tf.Tensor"] = None,
-        similarity_all: Optional["tf.Tensor"] = None,
-        pred_confidence: Optional["tf.Tensor"] = None,
-        similarity: Optional["tf.Tensor"] = None,
-        message_embed: Optional["tf.Tensor"] = None,
-        label_embed: Optional["tf.Tensor"] = None,
-        all_labels_embed: Optional["tf.Tensor"] = None,
-    ) -> None:
-        """Declare instant variables with default values"""
-
-        super().__init__(component_config)
-
-        self._load_params()
-
-        # transform numbers to labels
-        self.inverted_label_dict = inverted_label_dict
-        # encode all label_ids with numbers
-        self._encoded_all_label_ids = None
-
-        # tf related instances
-        self.session = session
-        self.graph = graph
-        self.a_in = message_placeholder
-        self.b_in = label_placeholder
-        self.sim_all = similarity_all
-        self.pred_confidence = pred_confidence
-        self.sim = similarity
-
-        # persisted embeddings
-        self.message_embed = message_embed
-        self.label_embed = label_embed
-        self.all_labels_embed = all_labels_embed
-
-        # internal tf instances
-        self._iterator = None
-        self._train_op = None
-        self._is_training = None
-
-    # config migration warning
-    def _check_old_config_variables(self, config: Dict[Text, Any]) -> None:
+    @staticmethod
+    def _check_old_config_variables(config: Dict[Text, Any]) -> None:
+        """Config migration warning"""
 
         removed_tokenization_params = [
             "intent_tokenization_flag",
@@ -174,21 +136,21 @@ def _check_old_config_variables(self, config: Dict[Text, Any]) -> None:
     # init helpers
     def _load_nn_architecture_params(self, config: Dict[Text, Any]) -> None:
         self.hidden_layer_sizes = {
-            "a": config["hidden_layers_sizes_a"],
-            "b": config["hidden_layers_sizes_b"],
+            "text": config["hidden_layers_sizes_a"],
+            "intent": config["hidden_layers_sizes_b"],
         }
         self.share_hidden_layers = config["share_hidden_layers"]
         if (
             self.share_hidden_layers
-            and self.hidden_layer_sizes["a"] != self.hidden_layer_sizes["b"]
+            and self.hidden_layer_sizes["text"] != self.hidden_layer_sizes["intent"]
         ):
             raise ValueError(
                 "If hidden layer weights are shared,"
                 "hidden_layer_sizes for a and b must coincide."
             )
 
-        self.batch_size = config["batch_size"]
-        self.batch_strategy = config["batch_strategy"]
+        self.batch_in_size = config["batch_size"]
+        self.batch_in_strategy = config["batch_strategy"]
 
         self.epochs = config["epochs"]
 
@@ -197,6 +159,7 @@ def _load_nn_architecture_params(self, config: Dict[Text, Any]) -> None:
     def _load_embedding_params(self, config: Dict[Text, Any]) -> None:
         self.embed_dim = config["embed_dim"]
         self.num_neg = config["num_neg"]
+        self.dense_dim = config["dense_dim"]
 
         self.similarity_type = config["similarity_type"]
         self.loss_type = config["loss_type"]
@@ -237,6 +200,53 @@ def _load_params(self) -> None:
     def required_packages(cls) -> List[Text]:
         return ["tensorflow"]
 
+    def __init__(
+        self,
+        component_config: Optional[Dict[Text, Any]] = None,
+        inverted_label_dict: Optional[Dict[int, Text]] = None,
+        session: Optional["tf.Session"] = None,
+        graph: Optional["tf.Graph"] = None,
+        batch_placeholder: Optional["tf.Tensor"] = None,
+        similarity_all: Optional["tf.Tensor"] = None,
+        pred_confidence: Optional["tf.Tensor"] = None,
+        similarity: Optional["tf.Tensor"] = None,
+        message_embed: Optional["tf.Tensor"] = None,
+        label_embed: Optional["tf.Tensor"] = None,
+        all_labels_embed: Optional["tf.Tensor"] = None,
+        batch_tuple_sizes: Optional[Dict] = None,
+    ) -> None:
+        """Declare instance variables with default values"""
+
+        super().__init__(component_config)
+
+        self._load_params()
+
+        # transform numbers to labels
+        self.inverted_label_dict = inverted_label_dict
+        # encode all label_ids with numbers
+        self._label_data = None
+
+        # tf related instances
+        self.session = session
+        self.graph = graph
+        self.batch_in = batch_placeholder
+        self.sim_all = similarity_all
+        self.pred_confidence = pred_confidence
+        self.sim = similarity
+
+        # persisted embeddings
+        self.message_embed = message_embed
+        self.label_embed = label_embed
+        self.all_labels_embed = all_labels_embed
+
+        # keep the input tuple sizes in self.batch_in
+        self.batch_tuple_sizes = batch_tuple_sizes
+
+        # internal tf instances
+        self._iterator = None
+        self._train_op = None
+        self._is_training = None
+
     # training data helpers:
     @staticmethod
     def _create_label_id_dict(
@@ -252,7 +262,9 @@ def _create_label_id_dict(
         }
 
     @staticmethod
-    def _find_example_for_label(label, examples, attribute):
+    def _find_example_for_label(
+        label: Text, examples: List["Message"], attribute: Text
+    ) -> Optional["Message"]:
         for ex in examples:
             if ex.get(attribute) == label:
                 return ex
@@ -260,96 +272,209 @@ def _find_example_for_label(label, examples, attribute):
 
     @staticmethod
     def _check_labels_features_exist(
-        labels_example: List[Tuple[int, "Message"]], attribute_feature_name: Text
+        labels_example: List["Message"], attribute: Text
     ) -> bool:
         """Check if all labels have features set"""
-        for (label_idx, label_example) in labels_example:
-            if label_example.get(attribute_feature_name) is None:
+
+        for label_example in labels_example:
+            if (
+                label_example.get(SPARSE_FEATURE_NAMES[attribute]) is None
+                and label_example.get(DENSE_FEATURE_NAMES[attribute]) is None
+            ):
                 return False
         return True
 
     @staticmethod
+    def _extract_and_add_features(
+        message: "Message", attribute: Text
+    ) -> Tuple[Optional[scipy.sparse.spmatrix], Optional[np.ndarray]]:
+        sparse_features = None
+        dense_features = None
+
+        if message.get(SPARSE_FEATURE_NAMES[attribute]) is not None:
+            sparse_features = message.get(SPARSE_FEATURE_NAMES[attribute])
+
+        if message.get(DENSE_FEATURE_NAMES[attribute]) is not None:
+            dense_features = message.get(DENSE_FEATURE_NAMES[attribute])
+
+        if sparse_features is not None and dense_features is not None:
+            if sparse_features.shape[0] != dense_features.shape[0]:
+                raise ValueError(
+                    f"Sequence dimensions for sparse and dense features "
+                    f"don't coincide in '{message.text}' for attribute '{attribute}'."
+                )
+
+        return sparse_features, dense_features
+
     def _extract_labels_precomputed_features(
-        label_examples: List[Tuple[int, "Message"]], attribute_feature_name: Text
-    ) -> np.ndarray:
+        self, label_examples: List["Message"]
+    ) -> List[np.ndarray]:
+        """Collect precomputed encodings"""
 
-        # Collect precomputed encodings
-        encoded_id_labels = [
-            (label_idx, label_example.get(attribute_feature_name))
-            for (label_idx, label_example) in label_examples
-        ]
+        sparse_features = []
+        dense_features = []
 
-        # Sort the list of tuples based on label_idx
-        encoded_id_labels = sorted(encoded_id_labels, key=lambda x: x[0])
+        for e in label_examples:
+            _sparse, _dense = self._extract_and_add_features(e, INTENT_ATTRIBUTE)
+            if _sparse is not None:
+                sparse_features.append(_sparse)
+            if _dense is not None:
+                dense_features.append(_dense)
 
-        encoded_all_labels = [encoding for (index, encoding) in encoded_id_labels]
+        sparse_features = np.array(sparse_features)
+        dense_features = np.array(dense_features)
 
-        return np.array(encoded_all_labels)
+        return [sparse_features, dense_features]
 
+    @staticmethod
     def _compute_default_label_features(
-        self, labels_example: List[Tuple[int, "Message"]]
-    ) -> np.ndarray:
+        labels_example: List["Message"],
+    ) -> List[np.ndarray]:
         """Compute one-hot representation for the labels"""
 
-        return np.eye(len(labels_example))
+        return [
+            np.array(
+                [
+                    np.expand_dims(a, 0)
+                    for a in np.eye(len(labels_example), dtype=np.float32)
+                ]
+            )
+        ]
 
-    def _create_encoded_label_ids(
+    def _create_label_data(
         self,
         training_data: "TrainingData",
         label_id_dict: Dict[Text, int],
         attribute: Text,
-        attribute_feature_name: Text,
-    ) -> np.ndarray:
-        """Create matrix with label_ids encoded in rows as bag of words. If the features are already computed, fetch
-        them from the message object else compute a one hot encoding for the label as the feature vector
-        Find a training example for each label and get the encoded features from the corresponding Message object"""
+    ) -> "SessionDataType":
+        """Create matrix with label_ids encoded in rows as bag of words.
 
-        labels_example = []
+        Find a training example for each label and get the encoded features
+        from the corresponding Message object.
+        If the features are already computed, fetch them from the message object
+        else compute a one hot encoding for the label as the feature vector.
+        """
 
         # Collect one example for each label
+        labels_idx_example = []
         for label_name, idx in label_id_dict.items():
             label_example = self._find_example_for_label(
                 label_name, training_data.intent_examples, attribute
             )
-            labels_example.append((idx, label_example))
+            labels_idx_example.append((idx, label_example))
+
+        # Sort the list of tuples based on label_idx
+        labels_idx_example = sorted(labels_idx_example, key=lambda x: x[0])
+        labels_example = [example for (_, example) in labels_idx_example]
 
         # Collect features, precomputed if they exist, else compute on the fly
-        if self._check_labels_features_exist(labels_example, attribute_feature_name):
-            encoded_id_labels = self._extract_labels_precomputed_features(
-                labels_example, attribute_feature_name
-            )
+        if self._check_labels_features_exist(labels_example, attribute):
+            features = self._extract_labels_precomputed_features(labels_example)
         else:
-            encoded_id_labels = self._compute_default_label_features(labels_example)
+            features = self._compute_default_label_features(labels_example)
+
+        label_data = {}
+        self._add_to_session_data(label_data, "intent_features", features)
+        self._add_mask_to_session_data(label_data, "intent_mask", "intent_features")
 
-        return encoded_id_labels
+        return label_data
+
+    def _use_default_label_features(self, label_ids: np.ndarray) -> List[np.ndarray]:
+        return [
+            np.array(
+                [
+                    self._label_data["intent_features"][0][label_id]
+                    for label_id in label_ids
+                ]
+            )
+        ]
 
     # noinspection PyPep8Naming
     def _create_session_data(
         self,
-        training_data: "TrainingData",
-        label_id_dict: Dict[Text, int],
-        attribute: Text,
-    ) -> "train_utils.SessionData":
-        """Prepare data for training and create a SessionData object"""
-
-        X = []
+        training_data: List["Message"],
+        label_id_dict: Optional[Dict[Text, int]] = None,
+        label_attribute: Optional[Text] = None,
+    ) -> "SessionDataType":
+        """Prepare data for training and create a SessionDataType object"""
+
+        X_sparse = []
+        X_dense = []
+        Y_sparse = []
+        Y_dense = []
         label_ids = []
-        Y = []
 
-        for e in training_data.intent_examples:
-            if e.get(attribute):
-                X.append(e.get(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]))
-                label_ids.append(label_id_dict[e.get(attribute)])
-
-        X = np.array(X)
+        for e in training_data:
+            _sparse, _dense = self._extract_and_add_features(e, TEXT_ATTRIBUTE)
+            if _sparse is not None:
+                X_sparse.append(_sparse)
+            if _dense is not None:
+                X_dense.append(_dense)
+
+            _sparse, _dense = self._extract_and_add_features(e, INTENT_ATTRIBUTE)
+            if _sparse is not None:
+                Y_sparse.append(_sparse)
+            if _dense is not None:
+                Y_dense.append(_dense)
+
+            if label_attribute and e.get(label_attribute):
+                label_ids.append(label_id_dict[e.get(label_attribute)])
+
+        X_sparse = np.array(X_sparse)
+        X_dense = np.array(X_dense)
+        Y_sparse = np.array(Y_sparse)
+        Y_dense = np.array(Y_dense)
         label_ids = np.array(label_ids)
 
-        for label_id_idx in label_ids:
-            Y.append(self._encoded_all_label_ids[label_id_idx])
+        session_data = {}
+        self._add_to_session_data(session_data, "text_features", [X_sparse, X_dense])
+        self._add_to_session_data(session_data, "intent_features", [Y_sparse, Y_dense])
+        # explicitly add last dimension to label_ids
+        # to track correctly dynamic sequences
+        self._add_to_session_data(
+            session_data, "intent_ids", [np.expand_dims(label_ids, -1)]
+        )
 
-        Y = np.array(Y)
+        if label_attribute and (
+            "intent_features" not in session_data or not session_data["intent_features"]
+        ):
+            # no label features are present, get default features from _label_data
+            session_data["intent_features"] = self._use_default_label_features(
+                label_ids
+            )
+
+        self._add_mask_to_session_data(session_data, "text_mask", "text_features")
+        self._add_mask_to_session_data(session_data, "intent_mask", "intent_features")
 
-        return train_utils.SessionData(X=X, Y=Y, label_ids=label_ids)
+        return session_data
+
+    @staticmethod
+    def _add_to_session_data(
+        session_data: SessionDataType, key: Text, features: List[np.ndarray]
+    ):
+        if not features:
+            return
+
+        session_data[key] = []
+
+        for data in features:
+            if data.size > 0:
+                session_data[key].append(data)
+
+    @staticmethod
+    def _add_mask_to_session_data(
+        session_data: SessionDataType, key: Text, from_key: Text
+    ):
+
+        session_data[key] = []
+
+        for data in session_data[from_key]:
+            if data.size > 0:
+                # explicitly add last dimension to mask
+                # to track correctly dynamic sequences
+                mask = np.array([np.ones((x.shape[0], 1)) for x in data])
+                session_data[key].append(mask)
+                break
 
     # tf helpers:
     def _create_tf_embed_fnn(
@@ -377,39 +502,82 @@ def _create_tf_embed_fnn(
             layer_name_suffix=embed_name,
         )
 
-    def _build_tf_train_graph(self) -> Tuple["tf.Tensor", "tf.Tensor"]:
-        self.a_in, self.b_in = self._iterator.get_next()
+    def _combine_sparse_dense_features(
+        self,
+        features: List[Union[tf.Tensor, tf.SparseTensor]],
+        mask: tf.Tensor,
+        name: Text,
+    ) -> tf.Tensor:
+        dense_features = []
+
+        dense_dim = self.dense_dim[name]
+        # if dense features are present use the feature dimension of the dense features
+        for f in features:
+            if not isinstance(f, tf.SparseTensor):
+                dense_dim = f.shape[-1]
+                break
+
+        for f in features:
+            if isinstance(f, tf.SparseTensor):
+                dense_features.append(
+                    train_utils.tf_dense_layer_for_sparse(f, dense_dim, name, self.C2)
+                )
+            else:
+                dense_features.append(f)
+
+        output = tf.concat(dense_features, axis=-1) * mask
+        # apply mean to convert sequence to sentence features
+        output = tf.reduce_sum(output, axis=1) / tf.reduce_sum(mask, axis=1)
+        return output
 
-        all_label_ids = tf.constant(
-            self._encoded_all_label_ids, dtype=tf.float32, name="all_label_ids"
+    def _build_tf_train_graph(
+        self, session_data: SessionDataType
+    ) -> Tuple["tf.Tensor", "tf.Tensor"]:
+
+        # get in tensors from generator
+        self.batch_in = self._iterator.get_next()
+        # convert encoded all labels into the batch format
+        label_batch = train_utils.prepare_batch(self._label_data)
+
+        # convert batch format into sparse and dense tensors
+        batch_data, _ = train_utils.batch_to_session_data(self.batch_in, session_data)
+        label_data, _ = train_utils.batch_to_session_data(label_batch, self._label_data)
+
+        a = self._combine_sparse_dense_features(
+            batch_data["text_features"], batch_data["text_mask"][0], "text"
+        )
+        b = self._combine_sparse_dense_features(
+            batch_data["intent_features"], batch_data["intent_mask"][0], "intent"
+        )
+        all_bs = self._combine_sparse_dense_features(
+            label_data["intent_features"], label_data["intent_mask"][0], "intent"
         )
 
         self.message_embed = self._create_tf_embed_fnn(
-            self.a_in,
-            self.hidden_layer_sizes["a"],
-            fnn_name="a_b" if self.share_hidden_layers else "a",
-            embed_name="a",
+            a,
+            self.hidden_layer_sizes["text"],
+            fnn_name="text_intent" if self.share_hidden_layers else "text",
+            embed_name="text",
         )
-
         self.label_embed = self._create_tf_embed_fnn(
-            self.b_in,
-            self.hidden_layer_sizes["b"],
-            fnn_name="a_b" if self.share_hidden_layers else "b",
-            embed_name="b",
+            b,
+            self.hidden_layer_sizes["intent"],
+            fnn_name="text_intent" if self.share_hidden_layers else "intent",
+            embed_name="intent",
         )
         self.all_labels_embed = self._create_tf_embed_fnn(
-            all_label_ids,
-            self.hidden_layer_sizes["b"],
-            fnn_name="a_b" if self.share_hidden_layers else "b",
-            embed_name="b",
+            all_bs,
+            self.hidden_layer_sizes["intent"],
+            fnn_name="text_intent" if self.share_hidden_layers else "intent",
+            embed_name="intent",
         )
 
         return train_utils.calculate_loss_acc(
             self.message_embed,
             self.label_embed,
-            self.b_in,
+            b,
             self.all_labels_embed,
-            all_label_ids,
+            all_bs,
             self.num_neg,
             None,
             self.loss_type,
@@ -420,21 +588,34 @@ def _build_tf_train_graph(self) -> Tuple["tf.Tensor", "tf.Tensor"]:
             self.scale_loss,
         )
 
-    def _build_tf_pred_graph(
-        self, session_data: "train_utils.SessionData"
-    ) -> "tf.Tensor":
-        self.a_in = tf.placeholder(
-            tf.float32, (None, session_data.X.shape[-1]), name="a"
+    def _build_tf_pred_graph(self, session_data: "SessionDataType") -> "tf.Tensor":
+
+        shapes, types = train_utils.get_shapes_types(session_data)
+
+        batch_placeholder = []
+        for s, t in zip(shapes, types):
+            batch_placeholder.append(tf.placeholder(t, s))
+
+        self.batch_in = tf.tuple(batch_placeholder)
+
+        batch_data, self.batch_tuple_sizes = train_utils.batch_to_session_data(
+            self.batch_in, session_data
         )
-        self.b_in = tf.placeholder(
-            tf.float32, (None, None, session_data.Y.shape[-1]), name="b"
+
+        a = self._combine_sparse_dense_features(
+            batch_data["text_features"], batch_data["text_mask"][0], "text"
+        )
+        b = self._combine_sparse_dense_features(
+            batch_data["intent_features"], batch_data["intent_mask"][0], "intent"
         )
 
+        self.all_labels_embed = tf.constant(self.session.run(self.all_labels_embed))
+
         self.message_embed = self._create_tf_embed_fnn(
-            self.a_in,
-            self.hidden_layer_sizes["a"],
-            fnn_name="a_b" if self.share_hidden_layers else "a",
-            embed_name="a",
+            a,
+            self.hidden_layer_sizes["text"],
+            fnn_name="text_intent" if self.share_hidden_layers else "text",
+            embed_name="text",
         )
 
         self.sim_all = train_utils.tf_raw_sim(
@@ -444,10 +625,10 @@ def _build_tf_pred_graph(
         )
 
         self.label_embed = self._create_tf_embed_fnn(
-            self.b_in,
-            self.hidden_layer_sizes["b"],
-            fnn_name="a_b" if self.share_hidden_layers else "b",
-            embed_name="b",
+            b,
+            self.hidden_layer_sizes["intent"],
+            fnn_name="text_intent" if self.share_hidden_layers else "intent",
+            embed_name="intent",
         )
 
         self.sim = train_utils.tf_raw_sim(
@@ -456,53 +637,59 @@ def _build_tf_pred_graph(
 
         return train_utils.confidence_from_sim(self.sim_all, self.similarity_type)
 
-    def check_input_dimension_consistency(self, session_data):
+    @staticmethod
+    def _get_num_of_features(session_data: "SessionDataType", key: Text) -> int:
+        num_features = 0
+        for data in session_data[key]:
+            if data.size > 0:
+                num_features += data[0].shape[-1]
+        return num_features
+
+    def check_input_dimension_consistency(self, session_data: "SessionDataType"):
+        """Check if text features and intent features have the same dimension."""
 
         if self.share_hidden_layers:
-            if session_data.X[0].shape[-1] != session_data.Y[0].shape[-1]:
+            num_text_features = self._get_num_of_features(session_data, "text_features")
+            num_intent_features = self._get_num_of_features(
+                session_data, "intent_features"
+            )
+
+            if num_text_features != num_intent_features:
                 raise ValueError(
                     "If embeddings are shared, "
                     "text features and label features "
                     "must coincide. Check the output dimensions of previous components."
                 )
 
-    def preprocess_train_data(self, training_data):
-        """Performs sanity checks on training data, extracts encodings for labels and prepares data for training"""
+    def preprocess_train_data(self, training_data: "TrainingData"):
+        """Prepares data for training.
+
+        Performs sanity checks on training data, extracts encodings for labels.
+        """
 
         label_id_dict = self._create_label_id_dict(
-            training_data, attribute=MESSAGE_INTENT_ATTRIBUTE
+            training_data, attribute=INTENT_ATTRIBUTE
         )
 
         self.inverted_label_dict = {v: k for k, v in label_id_dict.items()}
-        self._encoded_all_label_ids = self._create_encoded_label_ids(
-            training_data,
-            label_id_dict,
-            attribute=MESSAGE_INTENT_ATTRIBUTE,
-            attribute_feature_name=MESSAGE_VECTOR_FEATURE_NAMES[
-                MESSAGE_INTENT_ATTRIBUTE
-            ],
-        )
 
-        # check if number of negatives is less than number of label_ids
-        logger.debug(
-            f"Check if num_neg {self.num_neg} is smaller than "
-            f"number of label_ids {self._encoded_all_label_ids.shape[0]}, "
-            f"else set num_neg to the number of label_ids - 1."
+        self._label_data = self._create_label_data(
+            training_data, label_id_dict, attribute=INTENT_ATTRIBUTE
         )
-        # noinspection PyAttributeOutsideInit
-        self.num_neg = min(self.num_neg, self._encoded_all_label_ids.shape[0] - 1)
 
         session_data = self._create_session_data(
-            training_data, label_id_dict, attribute=MESSAGE_INTENT_ATTRIBUTE
+            training_data.intent_examples,
+            label_id_dict,
+            label_attribute=INTENT_ATTRIBUTE,
         )
 
         self.check_input_dimension_consistency(session_data)
 
         return session_data
 
-    def _check_enough_labels(self, session_data) -> bool:
-
-        return len(np.unique(session_data.label_ids)) >= 2
+    @staticmethod
+    def _check_enough_labels(session_data: "SessionDataType") -> bool:
+        return len(np.unique(session_data["intent_ids"])) >= 2
 
     def train(
         self,
@@ -510,7 +697,7 @@ def train(
         cfg: Optional["RasaNLUModelConfig"] = None,
         **kwargs: Any,
     ) -> None:
-        """Train the embedding label classifier on a data set."""
+        """Train the embedding intent classifier on a data set."""
 
         logger.debug("Started training embedding classifier.")
 
@@ -531,7 +718,10 @@ def train(
 
         if self.evaluate_on_num_examples:
             session_data, eval_session_data = train_utils.train_val_split(
-                session_data, self.evaluate_on_num_examples, self.random_seed
+                session_data,
+                self.evaluate_on_num_examples,
+                self.random_seed,
+                label_key="intent_ids",
             )
         else:
             eval_session_data = None
@@ -549,12 +739,16 @@ def train(
                 train_init_op,
                 eval_init_op,
             ) = train_utils.create_iterator_init_datasets(
-                session_data, eval_session_data, batch_size_in, self.batch_strategy
+                session_data,
+                eval_session_data,
+                batch_size_in,
+                self.batch_in_strategy,
+                label_key="intent_ids",
             )
 
             self._is_training = tf.placeholder_with_default(False, shape=())
 
-            loss, acc = self._build_tf_train_graph()
+            loss, acc = self._build_tf_train_graph(session_data)
 
             # define which optimizer to use
             self._train_op = tf.train.AdamOptimizer().minimize(loss)
@@ -571,7 +765,7 @@ def train(
                 self.session,
                 self._is_training,
                 self.epochs,
-                self.batch_size,
+                self.batch_in_size,
                 self.evaluate_on_num_examples,
                 self.evaluate_every_num_epochs,
             )
@@ -581,10 +775,17 @@ def train(
 
     # process helpers
     # noinspection PyPep8Naming
-    def _calculate_message_sim(self, X: np.ndarray) -> Tuple[np.ndarray, List[float]]:
+    def _calculate_message_sim(
+        self, batch: Tuple[np.ndarray]
+    ) -> Tuple[np.ndarray, List[float]]:
         """Calculate message similarities"""
 
-        message_sim = self.session.run(self.pred_confidence, feed_dict={self.a_in: X})
+        message_sim = self.session.run(
+            self.pred_confidence,
+            feed_dict={
+                _x_in: _x for _x_in, _x in zip(self.batch_in, batch) if _x is not None
+            },
+        )
 
         message_sim = message_sim.flatten()  # sim is a matrix
 
@@ -594,40 +795,45 @@ def _calculate_message_sim(self, X: np.ndarray) -> Tuple[np.ndarray, List[float]
         # transform sim to python list for JSON serializing
         return label_ids, message_sim.tolist()
 
-    def predict_label(self, message):
+    def predict_label(
+        self, message: "Message"
+    ) -> Tuple[Dict[Text, Any], List[Dict[Text, Any]]]:
+        """Predicts the intent of the provided message."""
 
         label = {"name": None, "confidence": 0.0}
         label_ranking = []
+
         if self.session is None:
             logger.error(
                 "There is no trained tf.session: "
                 "component is either not trained or "
                 "didn't receive enough training data."
             )
+            return label, label_ranking
+
+        # create session data from message and convert it into a batch of 1
+        session_data = self._create_session_data([message])
+        batch = train_utils.prepare_batch(
+            session_data, tuple_sizes=self.batch_tuple_sizes
+        )
+
+        # load tf graph and session
+        label_ids, message_sim = self._calculate_message_sim(batch)
+
+        # if X contains all zeros do not predict some label
+        if label_ids.size > 0:
+            label = {
+                "name": self.inverted_label_dict[label_ids[0]],
+                "confidence": message_sim[0],
+            }
+
+            ranking = list(zip(list(label_ids), message_sim))
+            ranking = ranking[:LABEL_RANKING_LENGTH]
+            label_ranking = [
+                {"name": self.inverted_label_dict[label_idx], "confidence": score}
+                for label_idx, score in ranking
+            ]
 
-        else:
-            # get features (bag of words) for a message
-            # noinspection PyPep8Naming
-            X = message.get(
-                MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]
-            ).reshape(1, -1)
-
-            # load tf graph and session
-            label_ids, message_sim = self._calculate_message_sim(X)
-
-            # if X contains all zeros do not predict some label
-            if X.any() and label_ids.size > 0:
-                label = {
-                    "name": self.inverted_label_dict[label_ids[0]],
-                    "confidence": message_sim[0],
-                }
-
-                ranking = list(zip(list(label_ids), message_sim))
-                ranking = ranking[:LABEL_RANKING_LENGTH]
-                label_ranking = [
-                    {"name": self.inverted_label_dict[label_idx], "confidence": score}
-                    for label_idx, score in ranking
-                ]
         return label, label_ranking
 
     def process(self, message: "Message", **kwargs: Any) -> None:
@@ -658,8 +864,7 @@ def persist(self, file_name: Text, model_dir: Text) -> Dict[Text, Any]:
             if e.errno != errno.EEXIST:
                 raise
         with self.graph.as_default():
-            train_utils.persist_tensor("message_placeholder", self.a_in, self.graph)
-            train_utils.persist_tensor("label_placeholder", self.b_in, self.graph)
+            train_utils.persist_tensor("batch_placeholder", self.batch_in, self.graph)
 
             train_utils.persist_tensor("similarity_all", self.sim_all, self.graph)
             train_utils.persist_tensor(
@@ -684,6 +889,11 @@ def persist(self, file_name: Text, model_dir: Text) -> Dict[Text, Any]:
         with open(os.path.join(model_dir, file_name + ".tf_config.pkl"), "wb") as f:
             pickle.dump(self._tf_config, f)
 
+        with open(
+            os.path.join(model_dir, file_name + ".batch_tuple_sizes.pkl"), "wb"
+        ) as f:
+            pickle.dump(self.batch_tuple_sizes, f)
+
         return {"file": file_name}
 
     @classmethod
@@ -695,6 +905,7 @@ def load(
         cached_component: Optional["EmbeddingIntentClassifier"] = None,
         **kwargs: Any,
     ) -> "EmbeddingIntentClassifier":
+        """Loads the trained model from the provided directory."""
 
         if model_dir and meta.get("file"):
             file_name = meta.get("file")
@@ -710,8 +921,7 @@ def load(
 
                 saver.restore(session, checkpoint)
 
-                a_in = train_utils.load_tensor("message_placeholder")
-                b_in = train_utils.load_tensor("label_placeholder")
+                batch_in = train_utils.load_tensor("batch_placeholder")
 
                 sim_all = train_utils.load_tensor("similarity_all")
                 pred_confidence = train_utils.load_tensor("pred_confidence")
@@ -726,19 +936,24 @@ def load(
             ) as f:
                 inv_label_dict = pickle.load(f)
 
+            with open(
+                os.path.join(model_dir, file_name + ".batch_tuple_sizes.pkl"), "rb"
+            ) as f:
+                batch_tuple_sizes = pickle.load(f)
+
             return cls(
                 component_config=meta,
                 inverted_label_dict=inv_label_dict,
                 session=session,
                 graph=graph,
-                message_placeholder=a_in,
-                label_placeholder=b_in,
+                batch_placeholder=batch_in,
                 similarity_all=sim_all,
                 pred_confidence=pred_confidence,
                 similarity=sim,
                 message_embed=message_embed,
                 label_embed=label_embed,
                 all_labels_embed=all_labels_embed,
+                batch_tuple_sizes=batch_tuple_sizes,
             )
 
         else:
diff --git a/rasa/nlu/classifiers/mitie_intent_classifier.py b/rasa/nlu/classifiers/mitie_intent_classifier.py
index ad60b008cce1..7a95e8e26f8c 100644
--- a/rasa/nlu/classifiers/mitie_intent_classifier.py
+++ b/rasa/nlu/classifiers/mitie_intent_classifier.py
@@ -5,6 +5,7 @@
 from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.model import Metadata
+from rasa.nlu.constants import TOKENS_NAMES, TEXT_ATTRIBUTE
 from rasa.nlu.training_data import Message, TrainingData
 
 if typing.TYPE_CHECKING:
@@ -15,7 +16,7 @@ class MitieIntentClassifier(Component):
 
     provides = ["intent"]
 
-    requires = ["tokens", "mitie_feature_extractor", "mitie_file"]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE], "mitie_feature_extractor", "mitie_file"]
 
     def __init__(
         self, component_config: Optional[Dict[Text, Any]] = None, clf=None
@@ -78,7 +79,7 @@ def process(self, message: Message, **kwargs: Any) -> None:
 
     @staticmethod
     def _tokens_of_message(message) -> List[Text]:
-        return [token.text for token in message.get("tokens", [])]
+        return [token.text for token in message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])]
 
     @classmethod
     def load(
diff --git a/rasa/nlu/classifiers/sklearn_intent_classifier.py b/rasa/nlu/classifiers/sklearn_intent_classifier.py
index 377c9c95224d..bfdadd90c21b 100644
--- a/rasa/nlu/classifiers/sklearn_intent_classifier.py
+++ b/rasa/nlu/classifiers/sklearn_intent_classifier.py
@@ -5,13 +5,14 @@
 import typing
 from typing import Any, Dict, List, Optional, Text, Tuple
 
+from rasa.nlu.featurizers.featurzier import sequence_to_sentence_features
 from rasa.nlu import utils
 from rasa.nlu.classifiers import LABEL_RANKING_LENGTH
 from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.model import Metadata
 from rasa.nlu.training_data import Message, TrainingData
-from rasa.nlu.constants import MESSAGE_VECTOR_FEATURE_NAMES, MESSAGE_TEXT_ATTRIBUTE
+from rasa.nlu.constants import DENSE_FEATURE_NAMES, SPARSE_FEATURE_NAMES, TEXT_ATTRIBUTE
 
 logger = logging.getLogger(__name__)
 
@@ -24,7 +25,7 @@ class SklearnIntentClassifier(Component):
 
     provides = ["intent", "intent_ranking"]
 
-    requires = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = [DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]]
 
     defaults = {
         # C parameter of the svm - cross validation will select the best value
@@ -96,7 +97,9 @@ def train(
             y = self.transform_labels_str2num(labels)
             X = np.stack(
                 [
-                    example.get("text_features")
+                    sequence_to_sentence_features(
+                        example.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+                    )
                     for example in training_data.intent_examples
                 ]
             )
@@ -146,7 +149,9 @@ def process(self, message: Message, **kwargs: Any) -> None:
             intent = None
             intent_ranking = []
         else:
-            X = message.get("text_features").reshape(1, -1)
+            X = sequence_to_sentence_features(
+                message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+            ).reshape(1, -1)
             intent_ids, probabilities = self.predict(X)
             intents = self.transform_labels_num2str(np.ravel(intent_ids))
             # `predict` returns a matrix as it is supposed
diff --git a/rasa/nlu/components.py b/rasa/nlu/components.py
index 4b261a52e46e..d5c3786887b2 100644
--- a/rasa/nlu/components.py
+++ b/rasa/nlu/components.py
@@ -5,7 +5,7 @@
 
 from rasa.nlu.config import RasaNLUModelConfig, override_defaults
 from rasa.nlu.training_data import TrainingData, Message
-from rasa.nlu.constants import MESSAGE_RESPONSE_ATTRIBUTE
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE
 
 if typing.TYPE_CHECKING:
     from rasa.nlu.model import Metadata
@@ -90,7 +90,7 @@ def validate_required_components_from_data(
     response_selector_exists = False
     for component in pipeline:
         # check if a response selector is part of NLU pipeline
-        if MESSAGE_RESPONSE_ATTRIBUTE in component.provides:
+        if RESPONSE_ATTRIBUTE in component.provides:
             response_selector_exists = True
 
     if len(data.response_examples) and not response_selector_exists:
diff --git a/rasa/nlu/constants.py b/rasa/nlu/constants.py
index 9e3b5f7a2e78..6fb2f023fe18 100644
--- a/rasa/nlu/constants.py
+++ b/rasa/nlu/constants.py
@@ -1,43 +1,40 @@
-MESSAGE_TEXT_ATTRIBUTE = "text"
+TEXT_ATTRIBUTE = "text"
 
-MESSAGE_RESPONSE_KEY_ATTRIBUTE = "response_key"
+RESPONSE_KEY_ATTRIBUTE = "response_key"
 
-MESSAGE_INTENT_ATTRIBUTE = "intent"
+INTENT_ATTRIBUTE = "intent"
 
-MESSAGE_RESPONSE_ATTRIBUTE = "response"
+RESPONSE_ATTRIBUTE = "response"
 
-MESSAGE_ENTITIES_ATTRIBUTE = "entities"
+ENTITIES_ATTRIBUTE = "entities"
 
-MESSAGE_NER_FEATURES_ATTRIBUTE = "ner_features"
+CLS_TOKEN = "__CLS__"
 
-MESSAGE_ATTRIBUTES = [
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_RESPONSE_ATTRIBUTE,
-]
+MESSAGE_ATTRIBUTES = [TEXT_ATTRIBUTE, INTENT_ATTRIBUTE, RESPONSE_ATTRIBUTE]
 
-MESSAGE_TOKENS_NAMES = {
-    MESSAGE_TEXT_ATTRIBUTE: "tokens",
-    MESSAGE_INTENT_ATTRIBUTE: "intent_tokens",
-    MESSAGE_RESPONSE_ATTRIBUTE: "response_tokens",
-    MESSAGE_ENTITIES_ATTRIBUTE: "tokens",
+TOKENS_NAMES = {
+    TEXT_ATTRIBUTE: "tokens",
+    INTENT_ATTRIBUTE: "intent_tokens",
+    RESPONSE_ATTRIBUTE: "response_tokens",
 }
 
-MESSAGE_VECTOR_FEATURE_NAMES = {
-    MESSAGE_TEXT_ATTRIBUTE: "text_features",
-    MESSAGE_INTENT_ATTRIBUTE: "intent_features",
-    MESSAGE_RESPONSE_ATTRIBUTE: "response_features",
-    MESSAGE_ENTITIES_ATTRIBUTE: "ner_features",
+SPARSE_FEATURE_NAMES = {
+    TEXT_ATTRIBUTE: "text_sparse_features",
+    INTENT_ATTRIBUTE: "intent_sparse_features",
+    RESPONSE_ATTRIBUTE: "response_sparse_features",
 }
 
-MESSAGE_SPACY_FEATURES_NAMES = {
-    MESSAGE_TEXT_ATTRIBUTE: "spacy_doc",
-    MESSAGE_RESPONSE_ATTRIBUTE: "response_spacy_doc",
+DENSE_FEATURE_NAMES = {
+    TEXT_ATTRIBUTE: "text_dense_features",
+    INTENT_ATTRIBUTE: "intent_dense_features",
+    RESPONSE_ATTRIBUTE: "response_dense_features",
 }
 
-SPACY_FEATURIZABLE_ATTRIBUTES = [MESSAGE_TEXT_ATTRIBUTE, MESSAGE_RESPONSE_ATTRIBUTE]
+SPACY_DOCS = {TEXT_ATTRIBUTE: "spacy_doc", RESPONSE_ATTRIBUTE: "response_spacy_doc"}
 
-MESSAGE_SELECTOR_PROPERTY_NAME = "response_selector"
+DENSE_FEATURIZABLE_ATTRIBUTES = [TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE]
+
+RESPONSE_SELECTOR_PROPERTY_NAME = "response_selector"
 DEFAULT_OPEN_UTTERANCE_TYPE = "default"
 OPEN_UTTERANCE_PREDICTION_KEY = "response"
 OPEN_UTTERANCE_RANKING_KEY = "ranking"
diff --git a/rasa/nlu/extractors/crf_entity_extractor.py b/rasa/nlu/extractors/crf_entity_extractor.py
index e5915e6dd2f7..99854c01d3da 100644
--- a/rasa/nlu/extractors/crf_entity_extractor.py
+++ b/rasa/nlu/extractors/crf_entity_extractor.py
@@ -2,13 +2,21 @@
 import warnings
 import os
 import typing
-from typing import Any, Dict, List, Optional, Text, Tuple, Union
+import numpy as np
+from typing import Any, Dict, List, Optional, Text, Tuple, Union, NamedTuple
 
 from rasa.nlu.config import InvalidConfigError, RasaNLUModelConfig
 from rasa.nlu.extractors import EntityExtractor
 from rasa.nlu.model import Metadata
-from rasa.nlu.tokenizers import Token
+from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.constants import (
+    TOKENS_NAMES,
+    TEXT_ATTRIBUTE,
+    DENSE_FEATURE_NAMES,
+    SPACY_DOCS,
+    ENTITIES_ATTRIBUTE,
+)
 from rasa.constants import DOCS_BASE_URL
 
 try:
@@ -23,11 +31,19 @@
     from spacy.tokens import Doc
 
 
+class CRFToken(NamedTuple):
+    text: Text
+    tag: Text
+    entity: Text
+    pattern: Dict[Text, Any]
+    dense_features: np.ndarray
+
+
 class CRFEntityExtractor(EntityExtractor):
 
-    provides = ["entities"]
+    provides = [ENTITIES_ATTRIBUTE]
 
-    requires = ["tokens"]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE]]
 
     defaults = {
         # BILOU_flag determines whether to use BILOU tagging or not.
@@ -66,21 +82,21 @@ class CRFEntityExtractor(EntityExtractor):
     }
 
     function_dict = {
-        "low": lambda doc: doc[0].lower(),  # pytype: disable=attribute-error
-        "title": lambda doc: doc[0].istitle(),  # pytype: disable=attribute-error
-        "prefix5": lambda doc: doc[0][:5],
-        "prefix2": lambda doc: doc[0][:2],
-        "suffix5": lambda doc: doc[0][-5:],
-        "suffix3": lambda doc: doc[0][-3:],
-        "suffix2": lambda doc: doc[0][-2:],
-        "suffix1": lambda doc: doc[0][-1:],
-        "pos": lambda doc: doc[1],
-        "pos2": lambda doc: doc[1][:2],
-        "bias": lambda doc: "bias",
-        "upper": lambda doc: doc[0].isupper(),  # pytype: disable=attribute-error
-        "digit": lambda doc: doc[0].isdigit(),  # pytype: disable=attribute-error
-        "pattern": lambda doc: doc[3],
-        "ner_features": lambda doc: doc[4],
+        "low": lambda crf_token: crf_token.text.lower(),  # pytype: disable=attribute-error
+        "title": lambda crf_token: crf_token.text.istitle(),  # pytype: disable=attribute-error
+        "prefix5": lambda crf_token: crf_token.text[:5],
+        "prefix2": lambda crf_token: crf_token.text[:2],
+        "suffix5": lambda crf_token: crf_token.text[-5:],
+        "suffix3": lambda crf_token: crf_token.text[-3:],
+        "suffix2": lambda crf_token: crf_token.text[-2:],
+        "suffix1": lambda crf_token: crf_token.text[-1:],
+        "pos": lambda crf_token: crf_token.tag,
+        "pos2": lambda crf_token: crf_token.tag[:2],
+        "bias": lambda crf_token: "bias",
+        "upper": lambda crf_token: crf_token.text.isupper(),  # pytype: disable=attribute-error
+        "digit": lambda crf_token: crf_token.text.isdigit(),  # pytype: disable=attribute-error
+        "pattern": lambda crf_token: crf_token.pattern,
+        "text_dense_features": lambda crf_token: crf_token.dense_features,
     }
 
     def __init__(
@@ -96,8 +112,6 @@ def __init__(
         self._validate_configuration()
 
         self._check_pos_features_and_spacy()
-        # possibly add a check here to ensure ner_features iff custom_extractor
-        self._check_ner_features()
 
     def _check_pos_features_and_spacy(self) -> None:
         import itertools
@@ -118,13 +132,6 @@ def _check_spacy() -> None:
                 "instructions."
             )
 
-    def _check_ner_features(self) -> None:
-        import itertools
-
-        features = self.component_config.get("features", [])
-        used_features = set(itertools.chain.from_iterable(features))
-        self.use_ner_features = "ner_features" in used_features
-
     def _validate_configuration(self) -> None:
         if len(self.component_config.get("features", [])) % 2 != 1:
             raise ValueError(
@@ -156,27 +163,17 @@ def train(
 
             self._train_model(dataset)
 
-    def _create_dataset(
-        self, examples: List[Message]
-    ) -> List[
-        List[
-            Tuple[
-                Optional[Text],
-                Optional[Text],
-                Text,
-                Dict[Text, Any],
-                Optional[Dict[Text, Any]],
-            ]
-        ]
-    ]:
+    def _create_dataset(self, examples: List[Message]) -> List[List[CRFToken]]:
         dataset = []
+
         for example in examples:
             entity_offsets = self._convert_example(example)
             dataset.append(self._from_json_to_crf(example, entity_offsets))
+
         return dataset
 
     def _check_spacy_doc(self, message) -> None:
-        if self.pos_features and message.get("spacy_doc") is None:
+        if self.pos_features and message.get(SPACY_DOCS[TEXT_ATTRIBUTE]) is None:
             raise InvalidConfigError(
                 "Could not find `spacy_doc` attribute for "
                 "message {}\n"
@@ -192,7 +189,9 @@ def process(self, message: Message, **kwargs: Any) -> None:
 
         extracted = self.add_extractor_name(self.extract_entities(message))
         message.set(
-            "entities", message.get("entities", []) + extracted, add_to_output=True
+            ENTITIES_ATTRIBUTE,
+            message.get(ENTITIES_ATTRIBUTE, []) + extracted,
+            add_to_output=True,
         )
 
     @staticmethod
@@ -200,7 +199,7 @@ def _convert_example(example: Message) -> List[Tuple[int, int, Text]]:
         def convert_entity(entity):
             return entity["start"], entity["end"], entity["entity"]
 
-        return [convert_entity(ent) for ent in example.get("entities", [])]
+        return [convert_entity(ent) for ent in example.get(ENTITIES_ATTRIBUTE, [])]
 
     def extract_entities(self, message: Message) -> List[Dict[Text, Any]]:
         """Take a sentence and return entities in json format"""
@@ -335,9 +334,9 @@ def _from_crf_to_json(
     ) -> List[Dict[Text, Any]]:
 
         if self.pos_features:
-            tokens = message.get("spacy_doc")
+            tokens = message.get(SPACY_DOCS[TEXT_ATTRIBUTE])
         else:
-            tokens = message.get("tokens")
+            tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
 
         if len(tokens) != len(entities):
             raise Exception(
@@ -433,18 +432,7 @@ def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]
 
         return {"file": file_name}
 
-    def _sentence_to_features(
-        self,
-        sentence: List[
-            Tuple[
-                Optional[Text],
-                Optional[Text],
-                Text,
-                Dict[Text, Any],
-                Optional[Dict[Text, Any]],
-            ]
-        ],
-    ) -> List[Dict[Text, Any]]:
+    def _sentence_to_features(self, sentence: List[CRFToken]) -> List[Dict[Text, Any]]:
         """Convert a word into discrete features in self.crf_features,
         including word before and word after."""
 
@@ -503,25 +491,17 @@ def _sentence_to_labels(
 
     def _from_json_to_crf(
         self, message: Message, entity_offsets: List[Tuple[int, int, Text]]
-    ) -> List[
-        Tuple[
-            Optional[Text],
-            Optional[Text],
-            Text,
-            Dict[Text, Any],
-            Optional[Dict[Text, Any]],
-        ]
-    ]:
+    ) -> List[CRFToken]:
         """Convert json examples to format of underlying crfsuite."""
 
         if self.pos_features:
             from spacy.gold import GoldParse  # pytype: disable=import-error
 
-            doc_or_tokens = message.get("spacy_doc")
+            doc_or_tokens = message.get(SPACY_DOCS[TEXT_ATTRIBUTE])
             gold = GoldParse(doc_or_tokens, entities=entity_offsets)
             ents = [l[5] for l in gold.orig_annot]
         else:
-            doc_or_tokens = message.get("tokens")
+            doc_or_tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
             ents = self._bilou_tags_from_offsets(doc_or_tokens, entity_offsets)
 
         # collect badly annotated examples
@@ -585,8 +565,8 @@ def _bilou_tags_from_offsets(tokens, entities, missing: Text = "O") -> List[Text
 
     @staticmethod
     def __pattern_of_token(message, i):
-        if message.get("tokens") is not None:
-            return message.get("tokens")[i].get("pattern", {})
+        if message.get(TOKENS_NAMES[TEXT_ATTRIBUTE]) is not None:
+            return message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])[i].get("pattern", {})
         else:
             return {}
 
@@ -598,14 +578,23 @@ def __tag_of_token(token):
             return token.tag_
 
     @staticmethod
-    def __additional_ner_features(message: Message) -> List[Any]:
-        features = message.get("ner_features", [])
-        tokens = message.get("tokens", [])
+    def __get_dense_features(message: Message) -> Optional[List[Any]]:
+        features = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+        if features is None:
+            return None
+
+        tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])
         if len(tokens) != len(features):
-            warn_string = "Number of custom NER features ({}) does not match number of tokens ({})".format(
-                len(features), len(tokens)
+            warnings.warn(
+                f"Number of features ({len(features)}) for attribute "
+                f"'{DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]}' "
+                f"does not match number of tokens ({len(tokens)}). Set "
+                f"'return_sequence' to true in the corresponding featurizer in order "
+                f"to make use of the features in 'CRFEntityExtractor'."
             )
-            raise Exception(warn_string)
+            return None
+
         # convert to python-crfsuite feature format
         features_out = []
         for feature in features:
@@ -613,53 +602,38 @@ def __additional_ner_features(message: Message) -> List[Any]:
                 str(index): token_features
                 for index, token_features in enumerate(feature)
             }
-            converted = {"custom_ner_features": feature_dict}
+            converted = {"text_dense_features": feature_dict}
             features_out.append(converted)
         return features_out
 
     def _from_text_to_crf(
         self, message: Message, entities: List[Text] = None
-    ) -> List[
-        Tuple[
-            Optional[Text],
-            Optional[Text],
-            Text,
-            Dict[Text, Any],
-            Optional[Dict[Text, Any]],
-        ]
-    ]:
+    ) -> List[CRFToken]:
         """Takes a sentence and switches it to crfsuite format."""
 
         crf_format = []
         if self.pos_features:
-            tokens = message.get("spacy_doc")
+            tokens = message.get(SPACY_DOCS[TEXT_ATTRIBUTE])
         else:
-            tokens = message.get("tokens")
-        ner_features = (
-            self.__additional_ner_features(message) if self.use_ner_features else None
-        )
+            tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+
+        text_dense_features = self.__get_dense_features(message)
+
         for i, token in enumerate(tokens):
             pattern = self.__pattern_of_token(message, i)
             entity = entities[i] if entities else "N/A"
             tag = self.__tag_of_token(token) if self.pos_features else None
-            custom_ner_features = ner_features[i] if self.use_ner_features else None
-            crf_format.append((token.text, tag, entity, pattern, custom_ner_features))
+            dense_features = (
+                text_dense_features[i] if text_dense_features is not None else []
+            )
+
+            crf_format.append(
+                CRFToken(token.text, tag, entity, pattern, dense_features)
+            )
+
         return crf_format
 
-    def _train_model(
-        self,
-        df_train: List[
-            List[
-                Tuple[
-                    Optional[Text],
-                    Optional[Text],
-                    Text,
-                    Dict[Text, Any],
-                    Optional[Dict[Text, Any]],
-                ]
-            ]
-        ],
-    ) -> None:
+    def _train_model(self, df_train: List[List[CRFToken]]) -> None:
         """Train the crf tagger based on the training data."""
         import sklearn_crfsuite
 
diff --git a/rasa/nlu/extractors/duckling_http_extractor.py b/rasa/nlu/extractors/duckling_http_extractor.py
index d4d987dc1064..e9c329acc159 100644
--- a/rasa/nlu/extractors/duckling_http_extractor.py
+++ b/rasa/nlu/extractors/duckling_http_extractor.py
@@ -6,6 +6,7 @@
 import requests
 from typing import Any, List, Optional, Text, Dict
 
+from rasa.nlu.constants import ENTITIES_ATTRIBUTE
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.extractors import EntityExtractor
 from rasa.nlu.model import Metadata
@@ -51,7 +52,7 @@ def convert_duckling_format_to_rasa(
 class DucklingHTTPExtractor(EntityExtractor):
     """Searches for structured entites, e.g. dates, using a duckling server."""
 
-    provides = ["entities"]
+    provides = [ENTITIES_ATTRIBUTE]
 
     defaults = {
         # by default all dimensions recognized by duckling are returned
@@ -186,7 +187,9 @@ def process(self, message: Message, **kwargs: Any) -> None:
 
         extracted = self.add_extractor_name(extracted)
         message.set(
-            "entities", message.get("entities", []) + extracted, add_to_output=True
+            ENTITIES_ATTRIBUTE,
+            message.get(ENTITIES_ATTRIBUTE, []) + extracted,
+            add_to_output=True,
         )
 
     @classmethod
diff --git a/rasa/nlu/extractors/entity_synonyms.py b/rasa/nlu/extractors/entity_synonyms.py
index c0b7c930dcc1..500027711621 100644
--- a/rasa/nlu/extractors/entity_synonyms.py
+++ b/rasa/nlu/extractors/entity_synonyms.py
@@ -2,7 +2,7 @@
 import warnings
 from typing import Any, Dict, Optional, Text
 
-from rasa.nlu import utils
+from rasa.nlu.constants import ENTITIES_ATTRIBUTE
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.extractors import EntityExtractor
 from rasa.nlu.model import Metadata
@@ -13,7 +13,7 @@
 
 class EntitySynonymMapper(EntityExtractor):
 
-    provides = ["entities"]
+    provides = [ENTITIES_ATTRIBUTE]
 
     def __init__(
         self,
@@ -33,15 +33,15 @@ def train(
             self.add_entities_if_synonyms(key, value)
 
         for example in training_data.entity_examples:
-            for entity in example.get("entities", []):
+            for entity in example.get(ENTITIES_ATTRIBUTE, []):
                 entity_val = example.text[entity["start"] : entity["end"]]
                 self.add_entities_if_synonyms(entity_val, str(entity.get("value")))
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
-        updated_entities = message.get("entities", [])[:]
+        updated_entities = message.get(ENTITIES_ATTRIBUTE, [])[:]
         self.replace_synonyms(updated_entities)
-        message.set("entities", updated_entities, add_to_output=True)
+        message.set(ENTITIES_ATTRIBUTE, updated_entities, add_to_output=True)
 
     def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]:
 
diff --git a/rasa/nlu/extractors/mitie_entity_extractor.py b/rasa/nlu/extractors/mitie_entity_extractor.py
index 56a2639fb7f7..671b312e499f 100644
--- a/rasa/nlu/extractors/mitie_entity_extractor.py
+++ b/rasa/nlu/extractors/mitie_entity_extractor.py
@@ -4,10 +4,11 @@
 import typing
 from typing import Any, Dict, List, Optional, Text
 
+from rasa.nlu.constants import ENTITIES_ATTRIBUTE, TOKENS_NAMES, TEXT_ATTRIBUTE
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.extractors import EntityExtractor
 from rasa.nlu.model import Metadata
-from rasa.nlu.tokenizers import Token
+from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu.training_data import Message, TrainingData
 
 logger = logging.getLogger(__name__)
@@ -18,9 +19,9 @@
 
 class MitieEntityExtractor(EntityExtractor):
 
-    provides = ["entities"]
+    provides = [ENTITIES_ATTRIBUTE]
 
-    requires = ["tokens", "mitie_feature_extractor", "mitie_file"]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE], "mitie_feature_extractor", "mitie_file"]
 
     def __init__(self, component_config: Optional[Dict[Text, Any]] = None, ner=None):
         """Construct a new intent classifier using the sklearn framework."""
@@ -92,9 +93,9 @@ def _prepare_mitie_sample(self, training_example) -> Any:
         import mitie
 
         text = training_example.text
-        tokens = training_example.get("tokens")
+        tokens = training_example.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
         sample = mitie.ner_training_instance([t.text for t in tokens])
-        for ent in training_example.get("entities", []):
+        for ent in training_example.get(ENTITIES_ATTRIBUTE, []):
             try:
                 # if the token is not aligned an exception will be raised
                 start, end = MitieEntityExtractor.find_entity(ent, text, tokens)
@@ -124,11 +125,15 @@ def process(self, message: Message, **kwargs: Any) -> None:
             )
 
         ents = self.extract_entities(
-            message.text, message.get("tokens"), mitie_feature_extractor
+            message.text,
+            message.get(TOKENS_NAMES[TEXT_ATTRIBUTE]),
+            mitie_feature_extractor,
         )
         extracted = self.add_extractor_name(ents)
         message.set(
-            "entities", message.get("entities", []) + extracted, add_to_output=True
+            ENTITIES_ATTRIBUTE,
+            message.get(ENTITIES_ATTRIBUTE, []) + extracted,
+            add_to_output=True,
         )
 
     @classmethod
diff --git a/rasa/nlu/extractors/spacy_entity_extractor.py b/rasa/nlu/extractors/spacy_entity_extractor.py
index 3a79d69bb01c..8c0bf9e79322 100644
--- a/rasa/nlu/extractors/spacy_entity_extractor.py
+++ b/rasa/nlu/extractors/spacy_entity_extractor.py
@@ -1,6 +1,7 @@
 import typing
 from typing import Any, Dict, List, Text, Optional
 
+from rasa.nlu.constants import ENTITIES_ATTRIBUTE
 from rasa.nlu.extractors import EntityExtractor
 from rasa.nlu.training_data import Message
 
@@ -10,7 +11,7 @@
 
 class SpacyEntityExtractor(EntityExtractor):
 
-    provides = ["entities"]
+    provides = [ENTITIES_ATTRIBUTE]
 
     requires = ["spacy_nlp"]
 
@@ -35,7 +36,9 @@ def process(self, message: Message, **kwargs: Any) -> None:
             all_extracted, dimensions
         )
         message.set(
-            "entities", message.get("entities", []) + extracted, add_to_output=True
+            ENTITIES_ATTRIBUTE,
+            message.get(ENTITIES_ATTRIBUTE, []) + extracted,
+            add_to_output=True,
         )
 
     @staticmethod
diff --git a/rasa/nlu/featurizers/__init__.py b/rasa/nlu/featurizers/__init__.py
index 75480178ebf3..e69de29bb2d1 100644
--- a/rasa/nlu/featurizers/__init__.py
+++ b/rasa/nlu/featurizers/__init__.py
@@ -1,21 +0,0 @@
-import numpy as np
-
-from typing import Any, Text
-from rasa.nlu.training_data import Message
-from rasa.nlu.components import Component
-from rasa.nlu.constants import MESSAGE_VECTOR_FEATURE_NAMES, MESSAGE_TEXT_ATTRIBUTE
-
-
-class Featurizer(Component):
-    @staticmethod
-    def _combine_with_existing_features(
-        message: Message,
-        additional_features: np.array,
-        feature_name: Text = MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-    ) -> np.array:
-        if message.get(feature_name) is not None:
-            return np.concatenate(
-                (message.get(feature_name), additional_features), axis=-1
-            )
-        else:
-            return additional_features
diff --git a/rasa/nlu/featurizers/dense_featurizer/__init__.py b/rasa/nlu/featurizers/dense_featurizer/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/rasa/nlu/featurizers/convert_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
similarity index 54%
rename from rasa/nlu/featurizers/convert_featurizer.py
rename to rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
index 8633323fb3da..4b4b9999727e 100644
--- a/rasa/nlu/featurizers/convert_featurizer.py
+++ b/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
@@ -1,12 +1,13 @@
 import logging
-from rasa.nlu.featurizers import Featurizer
+import warnings
+from rasa.nlu.featurizers.featurzier import Featurizer
 from typing import Any, Dict, List, Optional, Text, Tuple
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.training_data import Message, TrainingData
 from rasa.nlu.constants import (
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-    SPACY_FEATURIZABLE_ATTRIBUTES,
+    TEXT_ATTRIBUTE,
+    DENSE_FEATURE_NAMES,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
 )
 import numpy as np
 import tensorflow as tf
@@ -17,12 +18,19 @@
 class ConveRTFeaturizer(Featurizer):
 
     provides = [
-        MESSAGE_VECTOR_FEATURE_NAMES[attribute]
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
+        DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
     ]
 
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
     def _load_model(self) -> None:
 
+        # needed in order to load model
         import tensorflow_text
         import tensorflow_hub as tfhub
 
@@ -44,12 +52,30 @@ def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
 
         self._load_model()
 
+        self.return_sequence = self.component_config["return_sequence"]
+
+        if self.return_sequence:
+            raise NotImplementedError(
+                f"ConveRTFeaturizer always returns a feature vector of size "
+                f"(1 x feature-dimensions). It cannot return a proper sequence "
+                f"right now. ConveRTFeaturizer can only be used "
+                f"with 'return_sequence' set to False. Also, any other featurizer "
+                f"used next to ConveRTFeaturizer should have the flag "
+                f"'return_sequence' set to False."
+            )
+
+        logger.debug(
+            f"ConveRTFeaturizer always returns a feature vector of size "
+            f"(1 x feature-dimensions). If you use any other featurizer with "
+            f"'return_sequence' equal to True, training will fail."
+        )
+
     @classmethod
     def required_packages(cls) -> List[Text]:
         return ["tensorflow_text", "tensorflow_hub"]
 
     def _compute_features(
-        self, batch_examples: List[Message], attribute: Text = MESSAGE_TEXT_ATTRIBUTE
+        self, batch_examples: List[Message], attribute: Text = TEXT_ATTRIBUTE
     ) -> np.ndarray:
 
         # Get text for attribute of each example
@@ -72,9 +98,17 @@ def train(
         **kwargs: Any,
     ) -> None:
 
+        if config is not None and config.language != "en":
+            warnings.warn(
+                f"Since ``ConveRT`` model is trained only on an english "
+                f"corpus of conversations, this featurizer should only be "
+                f"used if your training data is in english language. "
+                f"However, you are training in '{config.language}'."
+            )
+
         batch_size = 64
 
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES:
+        for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
 
             non_empty_examples = list(
                 filter(lambda x: x.get(attribute), training_data.training_examples)
@@ -96,11 +130,11 @@ def train(
                 for index, ex in enumerate(batch_examples):
 
                     ex.set(
-                        MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                        self._combine_with_existing_features(
+                        DENSE_FEATURE_NAMES[attribute],
+                        self._combine_with_existing_dense_features(
                             ex,
-                            batch_features[index],
-                            MESSAGE_VECTOR_FEATURE_NAMES[attribute],
+                            np.expand_dims(batch_features[index], axis=0),
+                            DENSE_FEATURE_NAMES[attribute],
                         ),
                     )
 
@@ -110,8 +144,10 @@ def process(self, message: Message, **kwargs: Any) -> None:
 
         feats = self._compute_features([message])[0]
         message.set(
-            MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-            self._combine_with_existing_features(
-                message, feats, MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]
+            DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
+            self._combine_with_existing_dense_features(
+                message,
+                np.expand_dims(feats, axis=0),
+                DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
             ),
         )
diff --git a/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
new file mode 100644
index 000000000000..9347b9296a34
--- /dev/null
+++ b/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
@@ -0,0 +1,131 @@
+import numpy as np
+import typing
+from typing import Any, List, Text, Dict
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    import mitie
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
+    MESSAGE_ATTRIBUTES,
+    DENSE_FEATURE_NAMES,
+    CLS_TOKEN,
+)
+
+
+class MitieFeaturizer(Featurizer):
+
+    provides = [DENSE_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    requires = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES] + [
+        "mitie_feature_extractor"
+    ]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    @classmethod
+    def required_packages(cls) -> List[Text]:
+        return ["mitie", "numpy"]
+
+    def ndim(self, feature_extractor: "mitie.total_word_feature_extractor"):
+
+        return feature_extractor.num_dimensions
+
+    def get_tokens_by_attribute(self, example, attribute) -> Any:
+
+        return example.get(TOKENS_NAMES[attribute])
+
+    def train(
+        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
+    ) -> None:
+
+        mitie_feature_extractor = self._mitie_feature_extractor(**kwargs)
+        for example in training_data.intent_examples:
+            for attribute in MESSAGE_ATTRIBUTES:
+                self.process_training_example(
+                    example, attribute, mitie_feature_extractor
+                )
+
+    def process_training_example(
+        self, example: Message, attribute: Text, mitie_feature_extractor: Any
+    ):
+        attribute_tokens = self.get_tokens_by_attribute(example, attribute)
+        if attribute_tokens is not None:
+            features = self.features_for_tokens(
+                attribute_tokens, mitie_feature_extractor
+            )
+            example.set(
+                DENSE_FEATURE_NAMES[attribute],
+                self._combine_with_existing_dense_features(
+                    example, features, DENSE_FEATURE_NAMES[attribute]
+                ),
+            )
+
+    def process(self, message: Message, **kwargs: Any) -> None:
+
+        mitie_feature_extractor = self._mitie_feature_extractor(**kwargs)
+        features = self.features_for_tokens(
+            message.get(TOKENS_NAMES[TEXT_ATTRIBUTE]), mitie_feature_extractor
+        )
+        message.set(
+            DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
+            self._combine_with_existing_dense_features(
+                message, features, DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+            ),
+        )
+
+    def _mitie_feature_extractor(self, **kwargs) -> Any:
+        mitie_feature_extractor = kwargs.get("mitie_feature_extractor")
+        if not mitie_feature_extractor:
+            raise Exception(
+                "Failed to train 'MitieFeaturizer'. "
+                "Missing a proper MITIE feature extractor. "
+                "Make sure this component is preceded by "
+                "the 'MitieNLP' component in the pipeline "
+                "configuration."
+            )
+        return mitie_feature_extractor
+
+    def features_for_tokens(
+        self,
+        tokens: List[Token],
+        feature_extractor: "mitie.total_word_feature_extractor",
+    ) -> np.ndarray:
+        cls_token_used = tokens[-1].text == CLS_TOKEN if tokens else False
+
+        tokens_without_cls = tokens
+        if cls_token_used:
+            tokens_without_cls = tokens[:-1]
+
+        # calculate features
+        features = []
+        for token in tokens_without_cls:
+            features.append(feature_extractor.get_feature_vector(token.text))
+        features = np.array(features)
+
+        if cls_token_used and self.return_sequence:
+            # cls token is used, need to append a vector
+            cls_token_vec = np.mean(features, axis=0, keepdims=True)
+            features = np.concatenate([features, cls_token_vec])
+
+        if not self.return_sequence:
+            features = np.mean(features, axis=0, keepdims=True)
+
+        return features
diff --git a/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
new file mode 100644
index 000000000000..3578f585ca71
--- /dev/null
+++ b/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
@@ -0,0 +1,88 @@
+import numpy as np
+import typing
+from typing import Any, Optional, Dict, Text
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    from spacy.tokens import Doc
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    SPACY_DOCS,
+    DENSE_FEATURE_NAMES,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
+    TOKENS_NAMES,
+    CLS_TOKEN,
+)
+
+
+class SpacyFeaturizer(Featurizer):
+
+    provides = [
+        DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ]
+
+    requires = [
+        SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ] + [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    def _features_for_doc(self, doc: "Doc") -> np.ndarray:
+        """Feature vector for a single document / sentence / tokens."""
+        if self.return_sequence:
+            return np.array([t.vector for t in doc])
+        return np.array([doc.vector])
+
+    def train(
+        self,
+        training_data: TrainingData,
+        config: Optional[RasaNLUModelConfig],
+        **kwargs: Any,
+    ) -> None:
+
+        for example in training_data.intent_examples:
+            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
+                self._set_spacy_features(example, attribute)
+
+    def get_doc(self, message, attribute):
+
+        return message.get(SPACY_DOCS[attribute])
+
+    def process(self, message: Message, **kwargs: Any) -> None:
+
+        self._set_spacy_features(message)
+
+    def _set_spacy_features(self, message, attribute=TEXT_ATTRIBUTE):
+        """Adds the spacy word vectors to the messages features."""
+
+        message_attribute_doc = self.get_doc(message, attribute)
+        tokens = message.get(TOKENS_NAMES[attribute])
+        cls_token_used = tokens[-1].text == CLS_TOKEN if tokens else False
+
+        if message_attribute_doc is not None:
+            features = self._features_for_doc(message_attribute_doc)
+
+            if cls_token_used and self.return_sequence:
+                # cls token is used, need to append a vector
+                cls_token_vec = np.mean(features, axis=0, keepdims=True)
+                features = np.concatenate([features, cls_token_vec])
+
+            features = self._combine_with_existing_dense_features(
+                message, features, DENSE_FEATURE_NAMES[attribute]
+            )
+            message.set(DENSE_FEATURE_NAMES[attribute], features)
diff --git a/rasa/nlu/featurizers/featurzier.py b/rasa/nlu/featurizers/featurzier.py
new file mode 100644
index 000000000000..334d6fb61fc1
--- /dev/null
+++ b/rasa/nlu/featurizers/featurzier.py
@@ -0,0 +1,75 @@
+import numpy as np
+import scipy.sparse
+from typing import Any, Text, List, Union, Optional, Dict
+from rasa.nlu.training_data import Message
+from rasa.nlu.components import Component
+from rasa.nlu.constants import SPARSE_FEATURE_NAMES, DENSE_FEATURE_NAMES, TEXT_ATTRIBUTE
+
+
+def sequence_to_sentence_features(
+    features: Union[np.ndarray, scipy.sparse.spmatrix]
+) -> Optional[Union[np.ndarray, scipy.sparse.spmatrix]]:
+    if features is None:
+        return None
+
+    if isinstance(features, scipy.sparse.spmatrix):
+        return scipy.sparse.coo_matrix(features.sum(axis=0))
+
+    return np.mean(features, axis=0)
+
+
+class Featurizer(Component):
+    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
+        super(Featurizer, self).__init__(component_config)
+
+        try:
+            self.return_sequence = self.component_config["return_sequence"]
+        except KeyError:
+            raise KeyError(
+                "No default value for 'return_sequence' was set. Please, "
+                "add it to the default dict of the featurizer."
+            )
+
+    @staticmethod
+    def _combine_with_existing_dense_features(
+        message: Message,
+        additional_features: Any,
+        feature_name: Text = DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
+    ) -> Any:
+        if message.get(feature_name) is not None:
+
+            if len(message.get(feature_name)) != len(additional_features):
+                raise ValueError(
+                    f"Cannot concatenate dense features as sequence dimension does not "
+                    f"match: {len(message.get(feature_name))} != "
+                    f"{len(additional_features)}. "
+                    f"Make sure to set 'return_sequence' to the same value for all your "
+                    f"featurizers."
+                )
+
+            return np.concatenate(
+                (message.get(feature_name), additional_features), axis=-1
+            )
+        else:
+            return additional_features
+
+    @staticmethod
+    def _combine_with_existing_sparse_features(
+        message: Message,
+        additional_features: Any,
+        feature_name: Text = SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
+    ) -> Any:
+        if message.get(feature_name) is not None:
+            from scipy.sparse import hstack
+
+            if message.get(feature_name).shape[0] != additional_features.shape[0]:
+                raise ValueError(
+                    f"Cannot concatenate sparse features as sequence dimension does not "
+                    f"match: {message.get(feature_name).shape[0]} != "
+                    f"{additional_features.shape[0]}. "
+                    f"Make sure to set 'return_sequence' to the same value for all your "
+                    f"featurizers."
+                )
+            return hstack([message.get(feature_name), additional_features])
+        else:
+            return additional_features
diff --git a/rasa/nlu/featurizers/mitie_featurizer.py b/rasa/nlu/featurizers/mitie_featurizer.py
deleted file mode 100644
index c570d10dce4f..000000000000
--- a/rasa/nlu/featurizers/mitie_featurizer.py
+++ /dev/null
@@ -1,106 +0,0 @@
-import numpy as np
-import typing
-from typing import Any, List, Text
-
-from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.featurizers import Featurizer
-from rasa.nlu.tokenizers import Token
-from rasa.nlu.training_data import Message, TrainingData
-
-if typing.TYPE_CHECKING:
-    import mitie
-
-from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-)
-
-
-class MitieFeaturizer(Featurizer):
-
-    provides = [
-        MESSAGE_VECTOR_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES
-    ]
-
-    requires = [MESSAGE_TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES] + [
-        "mitie_feature_extractor"
-    ]
-
-    @classmethod
-    def required_packages(cls) -> List[Text]:
-        return ["mitie", "numpy"]
-
-    def ndim(self, feature_extractor: "mitie.total_word_feature_extractor"):
-
-        return feature_extractor.num_dimensions
-
-    def get_tokens_by_attribute(self, example, attribute) -> Any:
-
-        return example.get(MESSAGE_TOKENS_NAMES[attribute])
-
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-
-        mitie_feature_extractor = self._mitie_feature_extractor(**kwargs)
-        for example in training_data.intent_examples:
-
-            for attribute in MESSAGE_ATTRIBUTES:
-
-                attribute_tokens = self.get_tokens_by_attribute(example, attribute)
-                if attribute_tokens is not None:
-
-                    features = self.features_for_tokens(
-                        attribute_tokens, mitie_feature_extractor
-                    )
-                    example.set(
-                        MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                        self._combine_with_existing_features(
-                            example, features, MESSAGE_VECTOR_FEATURE_NAMES[attribute]
-                        ),
-                    )
-
-    def process(self, message: Message, **kwargs: Any) -> None:
-
-        mitie_feature_extractor = self._mitie_feature_extractor(**kwargs)
-        features = self.features_for_tokens(
-            message.get(MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE]),
-            mitie_feature_extractor,
-        )
-        message.set(
-            MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-            self._combine_with_existing_features(
-                message, features, MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]
-            ),
-        )
-
-    def _mitie_feature_extractor(self, **kwargs) -> Any:
-        mitie_feature_extractor = kwargs.get("mitie_feature_extractor")
-        if not mitie_feature_extractor:
-            raise Exception(
-                "Failed to train 'MitieFeaturizer'. "
-                "Missing a proper MITIE feature extractor. "
-                "Make sure this component is preceded by "
-                "the 'MitieNLP' component in the pipeline "
-                "configuration."
-            )
-        return mitie_feature_extractor
-
-    def features_for_tokens(
-        self,
-        tokens: List[Token],
-        feature_extractor: "mitie.total_word_feature_extractor",
-    ) -> np.ndarray:
-
-        vec = np.zeros(self.ndim(feature_extractor))
-        for token in tokens:
-            vec += feature_extractor.get_feature_vector(token.text)
-        if tokens:
-            return vec / len(tokens)
-        else:
-            return vec
diff --git a/rasa/nlu/featurizers/ngram_featurizer.py b/rasa/nlu/featurizers/ngram_featurizer.py
deleted file mode 100644
index 7fd49200a23e..000000000000
--- a/rasa/nlu/featurizers/ngram_featurizer.py
+++ /dev/null
@@ -1,433 +0,0 @@
-import time
-from collections import Counter
-
-import logging
-import numpy as np
-import os
-import typing
-import warnings
-from string import punctuation
-from typing import Any, Dict, List, Optional, Text
-
-from rasa.nlu import utils
-from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.featurizers import Featurizer
-from rasa.nlu.training_data import Message, TrainingData
-from rasa.nlu.utils import write_json_to_file
-import rasa.utils.io
-from rasa.nlu.constants import (
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-)
-
-logger = logging.getLogger(__name__)
-
-if typing.TYPE_CHECKING:
-    from rasa.nlu.model import Metadata
-
-
-class NGramFeaturizer(Featurizer):
-
-    provides = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
-
-    requires = [MESSAGE_SPACY_FEATURES_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
-
-    defaults = {
-        # defines the maximum number of ngrams to collect and add
-        # to the featurization of a sentence
-        "max_number_of_ngrams": 10,
-        # the minimal length in characters of an ngram to be eligible
-        "ngram_min_length": 3,
-        # the maximal length in characters of an ngram to be eligible
-        "ngram_max_length": 17,
-        # the minimal number of times an ngram needs to occur in the
-        # training data to be considered as a feature
-        "ngram_min_occurrences": 5,
-        # during cross validation (used to detect which ngrams are most
-        # valuable) every intent with fever examples than this config
-        # value will be excluded
-        "min_intent_examples": 4,
-    }
-
-    def __init__(
-        self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        all_ngrams: Optional[List[Text]] = None,
-        best_num_ngrams: Optional[int] = None,
-    ):
-        super().__init__(component_config)
-
-        self.best_num_ngrams = best_num_ngrams
-        self.all_ngrams = all_ngrams
-
-    @classmethod
-    def required_packages(cls) -> List[Text]:
-        return ["spacy", "sklearn"]
-
-    def train(
-        self, training_data: TrainingData, cfg: RasaNLUModelConfig, **kwargs: Any
-    ):
-
-        start = time.time()
-        self.train_on_sentences(training_data.intent_examples)
-        logger.debug("Ngram collection took {} seconds".format(time.time() - start))
-
-        for example in training_data.training_examples:
-            updated = self._text_features_with_ngrams(example, self.best_num_ngrams)
-            example.set(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE], updated)
-
-    def process(self, message: Message, **kwargs: Any):
-
-        updated = self._text_features_with_ngrams(message, self.best_num_ngrams)
-        message.set(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE], updated)
-
-    def _text_features_with_ngrams(self, message, max_ngrams) -> np.array:
-
-        ngrams_to_use = self._ngrams_to_use(max_ngrams)
-
-        if ngrams_to_use is not None:
-            extras = np.array(self._ngrams_in_sentence(message, ngrams_to_use))
-            return self._combine_with_existing_features(message, extras)
-        else:
-            return message.get(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE])
-
-    @classmethod
-    def load(
-        cls,
-        meta: Dict[Text, Any],
-        model_dir: Optional[Text] = None,
-        model_metadata: Optional["Metadata"] = None,
-        cached_component: Optional["NGramFeaturizer"] = None,
-        **kwargs: Any,
-    ) -> "NGramFeaturizer":
-
-        file_name = meta.get("file")
-        featurizer_file = os.path.join(model_dir, file_name)
-
-        if os.path.exists(featurizer_file):
-            data = rasa.utils.io.read_json_file(featurizer_file)
-            return NGramFeaturizer(meta, data["all_ngrams"], data["best_num_ngrams"])
-        else:
-            return NGramFeaturizer(meta)
-
-    def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]:
-        """Persist this model into the passed directory."""
-
-        file_name = file_name + ".json"
-        featurizer_file = os.path.join(model_dir, file_name)
-        data = {"all_ngrams": self.all_ngrams, "best_num_ngrams": self.best_num_ngrams}
-
-        write_json_to_file(featurizer_file, data, separators=(",", ": "))
-
-        return {"file": file_name}
-
-    def train_on_sentences(self, examples) -> None:
-        labels = [e.get("intent") for e in examples]
-        self.all_ngrams = self._get_best_ngrams(examples, labels)
-        self.best_num_ngrams = self._cross_validation(examples, labels)
-
-    def _ngrams_to_use(self, num_ngrams: Optional[int]) -> List[Text]:
-        if num_ngrams == 0 or self.all_ngrams is None:
-            return []
-        elif num_ngrams is not None:
-            return self.all_ngrams[:num_ngrams]
-        else:
-            return self.all_ngrams
-
-    def _get_best_ngrams(self, examples, labels) -> list:
-        """Return an ordered list of the best character ngrams."""
-
-        oov_strings = self._remove_in_vocab_words(examples)
-        ngrams = self._generate_all_ngrams(
-            oov_strings, self.component_config["ngram_min_length"]
-        )
-        return self._sort_applicable_ngrams(ngrams, examples, labels)
-
-    def _remove_in_vocab_words(self, examples) -> List[Text]:
-        """Automatically removes words with digits in them, that may be a
-        hyperlink or that _are_ in vocabulary for the nlp."""
-
-        new_sents = []
-        for example in examples:
-            new_sents.append(self._remove_in_vocab_words_from_sentence(example))
-        return new_sents
-
-    @staticmethod
-    def _is_ngram_worthy(token) -> bool:
-        """Decide if we should use this token for ngram counting.
-
-        Excludes every word with digits in them, hyperlinks or
-        an assigned word vector."""
-        return (
-            not token.has_vector
-            and not token.like_url
-            and not token.like_num
-            and not token.like_email
-            and not token.is_punct
-        )
-
-    def _remove_in_vocab_words_from_sentence(self, example) -> Text:
-        """Filter for words that do not have a word vector."""
-
-        cleaned_tokens = [
-            token
-            for token in example.get(
-                MESSAGE_SPACY_FEATURES_NAMES[MESSAGE_TEXT_ATTRIBUTE]
-            )
-            if self._is_ngram_worthy(token)
-        ]
-
-        # keep only out-of-vocab 'non_word' words
-        non_words = " ".join([t.text for t in cleaned_tokens])
-
-        # remove digits and extra spaces
-        non_words = "".join([letter for letter in non_words if not letter.isdigit()])
-        non_words = " ".join([word for word in non_words.split(" ") if word != ""])
-
-        # add cleaned sentence to list of these sentences
-        return non_words
-
-    def _intents_with_enough_examples(self, labels, examples) -> list:
-        """Filter examples where we do not have a min number of examples."""
-
-        min_intent_examples = self.component_config["min_intent_examples"]
-        usable_labels = []
-
-        for label in np.unique(labels):
-            lab_sents = np.array(examples)[np.array(labels) == label]
-            if len(lab_sents) < min_intent_examples:
-                continue
-            usable_labels.append(label)
-
-        return usable_labels
-
-    def _rank_ngrams_using_cv(self, examples, labels, list_of_ngrams) -> list:
-        from sklearn import linear_model
-
-        X = np.array(self._ngrams_in_sentences(examples, list_of_ngrams))
-        y = self.encode_labels(labels)
-
-        clf = linear_model.RandomizedLogisticRegression(C=1)
-        clf.fit(X, y)
-
-        # sort the ngrams according to the classification score
-        scores = clf.scores_
-        sorted_idxs = sorted(enumerate(scores), key=lambda x: -1 * x[1])
-        sorted_ngrams = [list_of_ngrams[i[0]] for i in sorted_idxs]
-
-        return sorted_ngrams
-
-    def _sort_applicable_ngrams(self, ngrams_list, examples, labels) -> list:
-        """Given an intent classification problem and a list of ngrams,
-
-        creates ordered list of most useful ngrams."""
-
-        if not ngrams_list:
-            return []
-
-        # make sure we have enough labeled instances for cv
-        usable_labels = self._intents_with_enough_examples(labels, examples)
-
-        mask = [label in usable_labels for label in labels]
-        if any(mask) and len(usable_labels) >= 2:
-            try:
-                examples = np.array(examples)[mask]
-                labels = np.array(labels)[mask]
-
-                return self._rank_ngrams_using_cv(examples, labels, ngrams_list)
-            except ValueError as e:
-                if "needs samples of at least 2 classes" in str(e):
-                    # we got unlucky during the random
-                    # sampling :( and selected a slice that
-                    # only contains one class
-                    return []
-                else:
-                    raise e
-        else:
-            # there is no example we can use for the cross validation
-            return []
-
-    def _ngrams_in_sentences(self, examples, ngrams) -> list:
-        """Given a set of sentences, returns a feature vector for each sentence.
-
-        The first $k$ elements are from the `intent_features`,
-        the rest are {1,0} elements denoting whether an ngram is in sentence."""
-
-        all_vectors = []
-        for example in examples:
-            presence_vector = self._ngrams_in_sentence(example, ngrams)
-            all_vectors.append(presence_vector)
-        return all_vectors
-
-    def _ngrams_in_sentence(self, example, ngrams) -> np.ndarray:
-        """Given a set of sentences, return a vector indicating ngram presence.
-
-        The vector will return 1 entries if the corresponding ngram is
-        present in the sentence and 0 if it is not."""
-
-        cleaned_sentence = self._remove_in_vocab_words_from_sentence(example)
-        presence_vector = np.zeros(len(ngrams))
-        idx_array = [
-            idx for idx in range(len(ngrams)) if ngrams[idx] in cleaned_sentence
-        ]
-        presence_vector[idx_array] = 1
-        return presence_vector
-
-    def _generate_all_ngrams(self, list_of_strings, ngram_min_length) -> list:
-        """Takes a list of strings and generates all character ngrams.
-
-        Generated ngrams are at least 3 characters (and at most 17),
-        occur at least 5 times and occur independently of longer
-        superset ngrams at least once."""
-
-        features = {}
-        counters = {ngram_min_length - 1: Counter()}
-        max_length = self.component_config["ngram_max_length"]
-
-        for n in range(ngram_min_length, max_length):
-            candidates = []
-            features[n] = []
-            counters[n] = Counter()
-
-            # generate all possible n length ngrams
-            for text in list_of_strings:
-                text = text.replace(punctuation, " ")
-                for word in text.lower().split(" "):
-                    cands = [word[i : i + n] for i in range(len(word) - n)]
-                    for cand in cands:
-                        counters[n][cand] += 1
-                        if cand not in candidates:
-                            candidates.append(cand)
-
-            min_count = self.component_config["ngram_min_occurrences"]
-            # iterate over these candidates picking only the applicable ones
-            for can in candidates:
-                if counters[n][can] >= min_count:
-                    features[n].append(can)
-                    begin = can[:-1]
-                    end = can[1:]
-                    if n >= ngram_min_length:
-                        if (
-                            counters[n - 1][begin] == counters[n][can]
-                            and begin in features[n - 1]
-                        ):
-                            features[n - 1].remove(begin)
-                        if (
-                            counters[n - 1][end] == counters[n][can]
-                            and end in features[n - 1]
-                        ):
-                            features[n - 1].remove(end)
-
-        return [item for sublist in list(features.values()) for item in sublist]
-
-    @staticmethod
-    def _collect_features(examples) -> Optional[np.ndarray]:
-        if examples:
-            collected_features = [
-                e.get(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE])
-                for e in examples
-                if e.get(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE])
-                is not None
-            ]
-        else:
-            collected_features = []
-
-        if collected_features:
-            return np.stack(collected_features)
-        else:
-            return None
-
-    def _append_ngram_features(
-        self, examples, existing_features, max_ngrams
-    ) -> np.array:
-        ngrams_to_use = self._ngrams_to_use(max_ngrams)
-        extras = np.array(self._ngrams_in_sentences(examples, ngrams_to_use))
-        if existing_features is not None:
-            return np.hstack((existing_features, extras))
-        else:
-            return extras
-
-    @staticmethod
-    def _num_cv_splits(y: np.array) -> int:
-        return min(10, np.min(np.bincount(y))) if y.size > 0 else 0
-
-    @staticmethod
-    def encode_labels(labels: np.array) -> np.array:
-        from sklearn import preprocessing
-
-        intent_encoder = preprocessing.LabelEncoder()
-        intent_encoder.fit(labels)
-        return intent_encoder.transform(labels)
-
-    def _score_ngram_selection(
-        self, examples, y, existing_text_features, cv_splits, max_ngrams
-    ) -> np.array:
-        from sklearn.model_selection import cross_val_score
-        from sklearn.linear_model import LogisticRegression
-
-        if existing_text_features is None:
-            return 0.0
-
-        clf = LogisticRegression(class_weight="balanced")
-
-        no_ngrams_X = self._append_ngram_features(
-            examples, existing_text_features, max_ngrams
-        )
-        return np.mean(cross_val_score(clf, no_ngrams_X, y, cv=cv_splits))
-
-    @staticmethod
-    def _generate_test_points(max_ngrams) -> np.array:
-        """Generate a list of increasing numbers.
-
-        They are used to take the best n ngrams and evaluate them. This n
-        is varied to find the best number of ngrams to use. This function
-        defines the number of ngrams that get tested."""
-
-        possible_ngrams = np.linspace(0, max_ngrams, 8)
-        return np.unique(list(map(int, np.floor(possible_ngrams))))
-
-    def _cross_validation(self, examples, labels) -> int:
-        """Choose the best number of ngrams to include in bow.
-
-        Given an intent classification problem and a set of ordered ngrams
-        (ordered in terms of importance by pick_applicable_ngrams) we
-        choose the best number of ngrams to include in our bow vecs
-        by cross validation."""
-
-        max_ngrams = self.component_config["max_number_of_ngrams"]
-
-        if not self.all_ngrams:
-            logger.debug("Found no ngrams. Using existing features.")
-            return 0
-
-        existing_text_features = self._collect_features(examples)
-
-        y = self.encode_labels(labels)
-        cv_splits = self._num_cv_splits(y)
-
-        if cv_splits >= 3:
-            logger.debug(
-                "Started ngram cross-validation to find b"
-                "est number of ngrams to use..."
-            )
-
-            scores = []
-            num_ngrams = self._generate_test_points(max_ngrams)
-            for n in num_ngrams:
-                score = self._score_ngram_selection(
-                    examples, y, existing_text_features, cv_splits, max_ngrams=n
-                )
-                scores.append(score)
-                logger.debug(f"Evaluating usage of {n} ngrams. Score: {score}")
-
-            n_top = num_ngrams[np.argmax(scores)]
-            logger.info("Best score with {} ngrams: {}".format(n_top, np.max(scores)))
-            return n_top.item()
-        else:
-            warnings.warn(
-                "Can't cross-validate ngram featurizer. "
-                "There aren't enough examples per intent "
-                "(at least 3)"
-            )
-            return max_ngrams
diff --git a/rasa/nlu/featurizers/spacy_featurizer.py b/rasa/nlu/featurizers/spacy_featurizer.py
deleted file mode 100644
index 9a5973d75e23..000000000000
--- a/rasa/nlu/featurizers/spacy_featurizer.py
+++ /dev/null
@@ -1,105 +0,0 @@
-import numpy as np
-import typing
-from typing import Any, Dict, Text, Optional
-
-from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.featurizers import Featurizer
-from rasa.nlu.training_data import Message, TrainingData
-
-if typing.TYPE_CHECKING:
-    from spacy.language import Language
-    from spacy.tokens import Doc
-
-from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-    MESSAGE_ENTITIES_ATTRIBUTE,
-    MESSAGE_NER_FEATURES_ATTRIBUTE,
-    SPACY_FEATURIZABLE_ATTRIBUTES,
-)
-
-
-def ndim(spacy_nlp: "Language") -> int:
-    """Number of features used to represent a document / sentence."""
-    return spacy_nlp.vocab.vectors_length
-
-
-def features_for_doc(doc: "Doc") -> np.ndarray:
-    """Feature vector for a single document / sentence."""
-    return doc.vector
-
-
-class SpacyFeaturizer(Featurizer):
-
-    provides = [
-        MESSAGE_VECTOR_FEATURE_NAMES[attribute]
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
-    ] + [MESSAGE_NER_FEATURES_ATTRIBUTE]
-
-    requires = [
-        MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
-    ]
-
-    defaults = {"ner_feature_vectors": False}
-
-    def __init__(
-        self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        known_patterns=None,
-        lookup_tables=None,
-    ) -> None:
-
-        super().__init__(component_config)
-
-        self.ner_feature_vectors = self.component_config["ner_feature_vectors"]
-
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-
-        for example in training_data.intent_examples:
-            for attribute in SPACY_FEATURIZABLE_ATTRIBUTES:
-                self._set_spacy_features(example, attribute)
-            self._set_spacy_ner_features(example)
-
-    def get_doc(self, message, attribute) -> Any:
-
-        return message.get(MESSAGE_SPACY_FEATURES_NAMES[attribute])
-
-    def process(self, message: Message, **kwargs: Any) -> None:
-
-        self._set_spacy_features(message)
-        self._set_spacy_ner_features(message)
-
-    def _set_spacy_ner_features(self, message: Message):
-        """If we want to use spacy as an NER featurizer, set token vectors"""
-        doc = message.get(MESSAGE_SPACY_FEATURES_NAMES[MESSAGE_TEXT_ATTRIBUTE])
-        if self.ner_feature_vectors:
-            ner_features = np.array([t.vector for t in doc])
-        else:
-            ner_features = np.array([[] for t in doc])
-        combined_features = self._combine_with_existing_features(
-            message,
-            ner_features,
-            MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_ENTITIES_ATTRIBUTE],
-        )
-        message.set(
-            MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_ENTITIES_ATTRIBUTE], combined_features
-        )
-
-    def _set_spacy_features(self, message, attribute=MESSAGE_TEXT_ATTRIBUTE) -> None:
-        """Adds the spacy word vectors to the messages features."""
-
-        message_attribute_doc = self.get_doc(message, attribute)
-        if message_attribute_doc is not None:
-            fs = features_for_doc(message_attribute_doc)
-            features = self._combine_with_existing_features(
-                message, fs, MESSAGE_VECTOR_FEATURE_NAMES[attribute]
-            )
-            message.set(MESSAGE_VECTOR_FEATURE_NAMES[attribute], features)
diff --git a/rasa/nlu/featurizers/sparse_featurizer/__init__.py b/rasa/nlu/featurizers/sparse_featurizer/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/rasa/nlu/featurizers/count_vectors_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
similarity index 60%
rename from rasa/nlu/featurizers/count_vectors_featurizer.py
rename to rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
index 8f9907cd15e3..5a853a431cdd 100644
--- a/rasa/nlu/featurizers/count_vectors_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
@@ -1,36 +1,30 @@
 import logging
-import warnings
 import os
 import re
+import scipy.sparse
 from typing import Any, Dict, List, Optional, Text
-import numpy as np
 
 from sklearn.feature_extraction.text import CountVectorizer
 from rasa.nlu import utils
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.featurizers import Featurizer
+from rasa.nlu.featurizers.featurzier import Featurizer
 from rasa.nlu.model import Metadata
 from rasa.nlu.training_data import Message, TrainingData
-
-logger = logging.getLogger(__name__)
-
 from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
     MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-    SPACY_FEATURIZABLE_ATTRIBUTES,
+    SPARSE_FEATURE_NAMES,
+    INTENT_ATTRIBUTE,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
 )
 
+logger = logging.getLogger(__name__)
+
 
 class CountVectorsFeaturizer(Featurizer):
-    """Bag of words featurizer
+    """Creates a sequence of token counts features based on sklearn's `CountVectorizer`.
 
-    Creates bag-of-words representation of intent features
-    using sklearn's `CountVectorizer`.
     All tokens which consist only of digits (e.g. 123 and 99
     but not ab12d) will be represented by a single feature.
 
@@ -39,11 +33,9 @@ class CountVectorsFeaturizer(Featurizer):
     from https://arxiv.org/abs/1810.07150.
     """
 
-    provides = [
-        MESSAGE_VECTOR_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES
-    ]
+    provides = [SPARSE_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
 
-    requires = []
+    requires = [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
 
     defaults = {
         # whether to use a shared vocab
@@ -80,6 +72,10 @@ class CountVectorsFeaturizer(Featurizer):
         # will be converted to lowercase if lowercase is True
         "OOV_token": None,  # string or None
         "OOV_words": [],  # string or list of strings
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False,
     }
 
     @classmethod
@@ -120,6 +116,9 @@ def _load_count_vect_params(self) -> None:
         # if convert all characters to lowercase
         self.lowercase = self.component_config["lowercase"]
 
+        # whether to return a sequence or not
+        self.return_sequence = self.component_config["return_sequence"]
+
     # noinspection PyPep8Naming
     def _load_OOV_params(self) -> None:
         self.OOV_token = self.component_config["OOV_token"]
@@ -184,12 +183,12 @@ def _check_analyzer(self) -> None:
                 )
 
     @staticmethod
-    def _attributes(analyzer):
+    def _attributes_for(analyzer):
         """Create a list of attributes that should be featurized."""
 
         # intents should be featurized only by word level count vectorizer
         return (
-            MESSAGE_ATTRIBUTES if analyzer == "word" else SPACY_FEATURIZABLE_ATTRIBUTES
+            MESSAGE_ATTRIBUTES if analyzer == "word" else DENSE_FEATURIZABLE_ATTRIBUTES
         )
 
     def __init__(
@@ -211,154 +210,144 @@ def __init__(
         self._check_analyzer()
 
         # set which attributes to featurize
-        self._attributes = self._attributes(self.analyzer)
+        self._attributes = self._attributes_for(self.analyzer)
 
         # declare class instance for CountVectorizer
         self.vectorizers = vectorizers
 
-    def _get_message_text_by_attribute(
-        self, message: "Message", attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
-        """Get processed text of attribute of a message"""
-
-        if message.get(attribute) is None:
-            # return empty string since sklearn countvectorizer does not like None object while training and predicting
-            return ""
-
-        tokens = self._get_message_tokens_by_attribute(message, attribute)
-
-        text = self._process_text(tokens, attribute)
-
-        text = self._replace_with_oov_token(text, attribute)
+    @staticmethod
+    def _get_message_tokens_by_attribute(
+        message: "Message", attribute: Text
+    ) -> List[Text]:
+        """Get text tokens of an attribute of a message"""
+        if message.get(TOKENS_NAMES[attribute]):
+            return [t.lemma for t in message.get(TOKENS_NAMES[attribute])]
 
-        return text
+        return message.get(attribute).split()
 
-    def _process_text(
-        self, tokens: List[Text], attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
+    def _process_tokens(
+        self, tokens: List[Text], attribute: Text = TEXT_ATTRIBUTE
+    ) -> List[Text]:
         """Apply processing and cleaning steps to text"""
 
-        text = " ".join(tokens)
-
-        if attribute == MESSAGE_INTENT_ATTRIBUTE:
+        if attribute == INTENT_ATTRIBUTE:
             # Don't do any processing for intent attribute. Treat them as whole labels
-            return text
+            return tokens
 
         # replace all digits with NUMBER token
-        text = re.sub(r"\b[0-9]+\b", "__NUMBER__", text)
+        tokens = [re.sub(r"\b[0-9]+\b", "__NUMBER__", text) for text in tokens]
 
         # convert to lowercase if necessary
         if self.lowercase:
-            text = text.lower()
-        return text
+            tokens = [text.lower() for text in tokens]
+
+        return tokens
 
-    def _replace_with_oov_token(self, text: Text, attribute: Text) -> Text:
+    def _replace_with_oov_token(
+        self, tokens: List[Text], attribute: Text
+    ) -> List[Text]:
         """Replace OOV words with OOV token"""
 
         if self.OOV_token and self.analyzer == "word":
-            text_tokens = text.split()
-            if self._check_attribute_vocabulary(
+            vocabulary_exists = self._check_attribute_vocabulary(attribute)
+            if vocabulary_exists and self.OOV_token in self._get_attribute_vocabulary(
                 attribute
-            ) and self.OOV_token in self._get_attribute_vocabulary(attribute):
+            ):
                 # CountVectorizer is trained, process for prediction
-                text_tokens = [
+                tokens = [
                     t
                     if t in self._get_attribute_vocabulary_tokens(attribute)
                     else self.OOV_token
-                    for t in text_tokens
+                    for t in tokens
                 ]
             elif self.OOV_words:
                 # CountVectorizer is not trained, process for train
-                text_tokens = [
-                    self.OOV_token if t in self.OOV_words else t for t in text_tokens
-                ]
-            text = " ".join(text_tokens)
-        return text
+                tokens = [self.OOV_token if t in self.OOV_words else t for t in tokens]
 
-    @staticmethod
-    def _get_message_tokens_by_attribute(
-        message: "Message", attribute: Text
+        return tokens
+
+    def _get_processed_message_tokens_by_attribute(
+        self, message: "Message", attribute: Text = TEXT_ATTRIBUTE
     ) -> List[Text]:
-        """Get text tokens of an attribute of a message"""
+        """Get processed text of attribute of a message"""
+
+        if message.get(attribute) is None:
+            # return empty string since sklearn countvectorizer does not like None
+            # object while training and predicting
+            return [""]
+
+        tokens = self._get_message_tokens_by_attribute(message, attribute)
+        tokens = self._process_tokens(tokens, attribute)
+        tokens = self._replace_with_oov_token(tokens, attribute)
 
-        if attribute in SPACY_FEATURIZABLE_ATTRIBUTES and message.get(
-            MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        ):  # if lemmatize is possible
-            tokens = [
-                t.lemma_ for t in message.get(MESSAGE_SPACY_FEATURES_NAMES[attribute])
-            ]
-        elif message.get(
-            MESSAGE_TOKENS_NAMES[attribute]
-        ):  # if directly tokens is provided
-            tokens = [t.text for t in message.get(MESSAGE_TOKENS_NAMES[attribute])]
-        else:
-            tokens = message.get(attribute).split()
         return tokens
 
     # noinspection PyPep8Naming
-    def _check_OOV_present(self, examples) -> None:
+    def _check_OOV_present(self, all_tokens: List[List[Text]]):
         """Check if an OOV word is present"""
-        if self.OOV_token and not self.OOV_words:
-            for t in examples:
-                if (
-                    t is None
-                    or self.OOV_token in t
-                    or (self.lowercase and self.OOV_token in t.lower())
-                ):
-                    return
-
-            warnings.warn(
-                f"OOV_token='{self.OOV_token}' was given, but it is not present "
-                "in the training data. All unseen words "
-                "will be ignored during prediction."
-            )
-
-    def _set_attribute_features(
-        self,
-        attribute: Text,
-        attribute_features: np.ndarray,
-        training_data: "TrainingData",
-    ):
-        """Set computed features of the attribute to corresponding message objects"""
-        for i, example in enumerate(training_data.intent_examples):
-            # create bag for each example
-            example.set(
-                MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                self._combine_with_existing_features(
-                    example,
-                    attribute_features[i],
-                    MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                ),
-            )
+        if self.OOV_token and not self.OOV_words and all_tokens:
+            for tokens in all_tokens:
+                for text in tokens:
+                    if self.OOV_token in text or (
+                        self.lowercase and self.OOV_token in text.lower()
+                    ):
+                        return
+
+            if any(text for tokens in all_tokens for text in tokens):
+                # if there is some text in tokens, warn if there is no oov token
+                logger.warning(
+                    f"OOV_token='{self.OOV_token}' was given, but it is not present "
+                    "in the training data. All unseen words "
+                    "will be ignored during prediction."
+                )
 
-    def _get_all_attributes_processed_texts(
+    def _get_all_attributes_processed_tokens(
         self, training_data: "TrainingData"
-    ) -> Dict[Text, List[Text]]:
+    ) -> Dict[Text, List[List[Text]]]:
         """Get processed text for all attributes of examples in training data"""
 
-        processed_attribute_texts = {}
+        processed_attribute_tokens = {}
         for attribute in self._attributes:
-            attribute_texts = [
-                self._get_message_text_by_attribute(example, attribute)
-                for example in training_data.intent_examples
+            all_tokens = [
+                self._get_processed_message_tokens_by_attribute(example, attribute)
+                for example in training_data.training_examples
+            ]
+            if attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
+                # check for oov tokens only in text based attributes
+                self._check_OOV_present(all_tokens)
+            processed_attribute_tokens[attribute] = all_tokens
+
+        return processed_attribute_tokens
+
+    @staticmethod
+    def _convert_attribute_tokens_to_texts(
+        attribute_tokens: Dict[Text, List[List[Text]]]
+    ) -> Dict[Text, List[Text]]:
+        attribute_texts = {}
+
+        for attribute in attribute_tokens.keys():
+            attribute_texts[attribute] = [
+                " ".join(tokens) for tokens in attribute_tokens[attribute]
             ]
-            self._check_OOV_present(attribute_texts)
-            processed_attribute_texts[attribute] = attribute_texts
-        return processed_attribute_texts
+
+        return attribute_texts
 
     def _train_with_shared_vocab(self, attribute_texts: Dict[Text, List[Text]]):
         """Construct the vectorizers and train them with a shared vocab"""
 
         self.vectorizers = self._create_shared_vocab_vectorizers(
-            self.token_pattern,
-            self.strip_accents,
-            self.lowercase,
-            self.stop_words,
-            (self.min_ngram, self.max_ngram),
-            self.max_df,
-            self.min_df,
-            self.max_features,
-            self.analyzer,
+            {
+                "token_pattern": self.token_pattern,
+                "strip_accents": self.strip_accents,
+                "lowercase": self.lowercase,
+                "stop_words": self.stop_words,
+                "min_ngram": self.min_ngram,
+                "max_ngram": self.max_ngram,
+                "max_df": self.max_df,
+                "min_df": self.min_df,
+                "max_features": self.max_features,
+                "analyzer": self.analyzer,
+            }
         )
 
         combined_cleaned_texts = []
@@ -366,29 +355,33 @@ def _train_with_shared_vocab(self, attribute_texts: Dict[Text, List[Text]]):
             combined_cleaned_texts += attribute_texts[attribute]
 
         try:
-            self.vectorizers[MESSAGE_TEXT_ATTRIBUTE].fit(combined_cleaned_texts)
+            self.vectorizers[TEXT_ATTRIBUTE].fit(combined_cleaned_texts)
         except ValueError:
             logger.warning(
-                "Unable to train a shared CountVectorizer. Leaving an untrained CountVectorizer"
+                "Unable to train a shared CountVectorizer. "
+                "Leaving an untrained CountVectorizer"
             )
 
     @staticmethod
-    def _attribute_texts_is_non_empty(attribute_texts) -> bool:
+    def _attribute_texts_is_non_empty(attribute_texts: List[Text]) -> bool:
         return any(attribute_texts)
 
     def _train_with_independent_vocab(self, attribute_texts: Dict[Text, List[Text]]):
         """Construct the vectorizers and train them with an independent vocab"""
 
         self.vectorizers = self._create_independent_vocab_vectorizers(
-            self.token_pattern,
-            self.strip_accents,
-            self.lowercase,
-            self.stop_words,
-            (self.min_ngram, self.max_ngram),
-            self.max_df,
-            self.min_df,
-            self.max_features,
-            self.analyzer,
+            {
+                "token_pattern": self.token_pattern,
+                "strip_accents": self.strip_accents,
+                "lowercase": self.lowercase,
+                "stop_words": self.stop_words,
+                "min_ngram": self.min_ngram,
+                "max_ngram": self.max_ngram,
+                "max_df": self.max_df,
+                "min_df": self.min_df,
+                "max_features": self.max_features,
+                "analyzer": self.analyzer,
+            }
         )
 
         for attribute in self._attributes:
@@ -397,30 +390,60 @@ def _train_with_independent_vocab(self, attribute_texts: Dict[Text, List[Text]])
                     self.vectorizers[attribute].fit(attribute_texts[attribute])
                 except ValueError:
                     logger.warning(
-                        f"Unable to train CountVectorizer for message attribute "
-                        "{attribute}. "
-                        "Leaving an untrained CountVectorizer for it"
+                        f"Unable to train CountVectorizer for message "
+                        f"attribute {attribute}. Leaving an untrained "
+                        f"CountVectorizer for it."
                     )
             else:
                 logger.debug(
-                    "No text provided for {} attribute in any messages of training data. Skipping "
-                    "training a CountVectorizer for it.".format(attribute)
+                    f"No text provided for {attribute} attribute in any messages of "
+                    f"training data. Skipping training a CountVectorizer for it."
                 )
 
+    def _create_sequence(
+        self, attribute: Text, all_tokens: List[List[Text]]
+    ) -> List[scipy.sparse.coo_matrix]:
+        X = []
+
+        for i, tokens in enumerate(all_tokens):
+            # vectorizer.transform returns a sparse matrix of size
+            # [n_samples, n_features]
+            # set input to list of tokens if sequence should be returned
+            # otherwise join all tokens to a single string and pass that as a list
+            input = tokens
+            if not self.return_sequence:
+                input = [" ".join(tokens)]
+
+            x = self.vectorizers[attribute].transform(input)
+            x.sort_indices()
+            X.append(x.tocoo())
+
+        return X
+
     def _get_featurized_attribute(
-        self, attribute: Text, attribute_texts: List[Text]
-    ) -> Optional[np.ndarray]:
+        self, attribute: Text, all_tokens: List[List[Text]]
+    ) -> Optional[List[scipy.sparse.coo_matrix]]:
         """Return features of a particular attribute for complete data"""
 
         if self._check_attribute_vocabulary(attribute):
             # count vectorizer was trained
-            featurized_attributes = (
-                self.vectorizers[attribute].transform(attribute_texts).toarray()
-            )
-            return featurized_attributes
+            return self._create_sequence(attribute, all_tokens)
         else:
             return None
 
+    def _set_attribute_features(
+        self, attribute: Text, attribute_features: List, training_data: "TrainingData"
+    ):
+        """Set computed features of the attribute to corresponding message objects"""
+        for i, example in enumerate(training_data.training_examples):
+            # create bag for each example
+            example.set(
+                SPARSE_FEATURE_NAMES[attribute],
+                self._combine_with_existing_sparse_features(
+                    example, attribute_features[i], SPARSE_FEATURE_NAMES[attribute]
+                ),
+            )
+
     def train(
         self, training_data: TrainingData, cfg: RasaNLUModelConfig = None, **kwargs: Any
     ) -> None:
@@ -436,21 +459,24 @@ def train(
             self.OOV_words = [t.lemma_ for w in self.OOV_words for t in spacy_nlp(w)]
 
         # process sentences and collect data for all attributes
-        processed_attribute_texts = self._get_all_attributes_processed_texts(
+        processed_attribute_tokens = self._get_all_attributes_processed_tokens(
             training_data
         )
 
         # train for all attributes
+        attribute_texts = self._convert_attribute_tokens_to_texts(
+            processed_attribute_tokens
+        )
         if self.use_shared_vocab:
-            self._train_with_shared_vocab(processed_attribute_texts)
+            self._train_with_shared_vocab(attribute_texts)
         else:
-            self._train_with_independent_vocab(processed_attribute_texts)
+            self._train_with_independent_vocab(attribute_texts)
 
         # transform for all attributes
         for attribute in self._attributes:
 
             attribute_features = self._get_featurized_attribute(
-                attribute, processed_attribute_texts[attribute]
+                attribute, processed_attribute_tokens[attribute]
             )
 
             if attribute_features is not None:
@@ -467,25 +493,24 @@ def process(self, message: Message, **kwargs: Any) -> None:
                 "component is either not trained or "
                 "didn't receive enough training data"
             )
-        else:
-            message_text = self._get_message_text_by_attribute(
-                message, attribute=MESSAGE_TEXT_ATTRIBUTE
-            )
+            return
 
-            bag = (
-                self.vectorizers[MESSAGE_TEXT_ATTRIBUTE]
-                .transform([message_text])
-                .toarray()
-                .squeeze()
-            )
-            message.set(
-                MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-                self._combine_with_existing_features(
-                    message,
-                    bag,
-                    feature_name=MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-                ),
-            )
+        attribute = TEXT_ATTRIBUTE
+        message_tokens = self._get_processed_message_tokens_by_attribute(
+            message, attribute
+        )
+
+        # features shape (1, seq, dim)
+        features = self._create_sequence(attribute, [message_tokens])
+
+        message.set(
+            SPARSE_FEATURE_NAMES[attribute],
+            self._combine_with_existing_sparse_features(
+                message,
+                features[0],  # 0 -> batch dimension
+                feature_name=SPARSE_FEATURE_NAMES[attribute],
+            ),
+        )
 
     def _collect_vectorizer_vocabularies(self) -> Dict[Text, Optional[Dict[Text, int]]]:
         """Get vocabulary for all attributes"""
@@ -519,82 +544,64 @@ def persist(self, file_name: Text, model_dir: Text) -> Optional[Dict[Text, Any]]
                 featurizer_file = os.path.join(model_dir, file_name)
 
                 if self.use_shared_vocab:
-                    # Only persist vocabulary from one attribute. Can be loaded and distributed to all attributes.
-                    utils.json_pickle(
-                        featurizer_file, attribute_vocabularies[MESSAGE_TEXT_ATTRIBUTE]
-                    )
+                    # Only persist vocabulary from one attribute. Can be loaded and
+                    # distributed to all attributes.
+                    vocab = attribute_vocabularies[TEXT_ATTRIBUTE]
                 else:
-                    utils.json_pickle(featurizer_file, attribute_vocabularies)
+                    vocab = attribute_vocabularies
+
+                utils.json_pickle(featurizer_file, vocab)
+
         return {"file": file_name}
 
     @classmethod
     def _create_shared_vocab_vectorizers(
-        cls,
-        token_pattern,
-        strip_accents,
-        lowercase,
-        stop_words,
-        ngram_range,
-        max_df,
-        min_df,
-        max_features,
-        analyzer,
-        vocabulary=None,
+        cls, parameters: Dict[Text, Any], vocabulary: Optional[Any] = None
     ) -> Dict[Text, "CountVectorizer"]:
         """Create vectorizers for all attributes with shared vocabulary"""
 
         shared_vectorizer = CountVectorizer(
-            token_pattern=token_pattern,
-            strip_accents=strip_accents,
-            lowercase=lowercase,
-            stop_words=stop_words,
-            ngram_range=ngram_range,
-            max_df=max_df,
-            min_df=min_df,
-            max_features=max_features,
-            analyzer=analyzer,
+            token_pattern=parameters["token_pattern"],
+            strip_accents=parameters["strip_accents"],
+            lowercase=parameters["lowercase"],
+            stop_words=parameters["stop_words"],
+            ngram_range=(parameters["min_ngram"], parameters["max_ngram"]),
+            max_df=parameters["max_df"],
+            min_df=parameters["min_df"],
+            max_features=parameters["max_features"],
+            analyzer=parameters["analyzer"],
             vocabulary=vocabulary,
         )
 
         attribute_vectorizers = {}
 
-        for attribute in cls._attributes(analyzer):
+        for attribute in cls._attributes_for(parameters["analyzer"]):
             attribute_vectorizers[attribute] = shared_vectorizer
 
         return attribute_vectorizers
 
     @classmethod
     def _create_independent_vocab_vectorizers(
-        cls,
-        token_pattern,
-        strip_accents,
-        lowercase,
-        stop_words,
-        ngram_range,
-        max_df,
-        min_df,
-        max_features,
-        analyzer,
-        vocabulary=None,
+        cls, parameters: Dict[Text, Any], vocabulary: Optional[Any] = None
     ) -> Dict[Text, "CountVectorizer"]:
         """Create vectorizers for all attributes with independent vocabulary"""
 
         attribute_vectorizers = {}
 
-        for attribute in cls._attributes(analyzer):
+        for attribute in cls._attributes_for(parameters["analyzer"]):
 
             attribute_vocabulary = vocabulary[attribute] if vocabulary else None
 
             attribute_vectorizer = CountVectorizer(
-                token_pattern=token_pattern,
-                strip_accents=strip_accents,
-                lowercase=lowercase,
-                stop_words=stop_words,
-                ngram_range=ngram_range,
-                max_df=max_df,
-                min_df=min_df,
-                max_features=max_features,
-                analyzer=analyzer,
+                token_pattern=parameters["token_pattern"],
+                strip_accents=parameters["strip_accents"],
+                lowercase=parameters["lowercase"],
+                stop_words=parameters["stop_words"],
+                ngram_range=(parameters["min_ngram"], parameters["max_ngram"]),
+                max_df=parameters["max_df"],
+                min_df=parameters["min_df"],
+                max_features=parameters["max_features"],
+                analyzer=parameters["analyzer"],
                 vocabulary=attribute_vocabulary,
             )
             attribute_vectorizers[attribute] = attribute_vectorizer
@@ -614,38 +621,20 @@ def load(
         file_name = meta.get("file")
         featurizer_file = os.path.join(model_dir, file_name)
 
-        if os.path.exists(featurizer_file):
-            vocabulary = utils.json_unpickle(featurizer_file)
-
-            share_vocabulary = meta["use_shared_vocab"]
-
-            if share_vocabulary:
-                vectorizers = cls._create_shared_vocab_vectorizers(
-                    token_pattern=meta["token_pattern"],
-                    strip_accents=meta["strip_accents"],
-                    lowercase=meta["lowercase"],
-                    stop_words=meta["stop_words"],
-                    ngram_range=(meta["min_ngram"], meta["max_ngram"]),
-                    max_df=meta["max_df"],
-                    min_df=meta["min_df"],
-                    max_features=meta["max_features"],
-                    analyzer=meta["analyzer"],
-                    vocabulary=vocabulary,
-                )
-            else:
-                vectorizers = cls._create_independent_vocab_vectorizers(
-                    token_pattern=meta["token_pattern"],
-                    strip_accents=meta["strip_accents"],
-                    lowercase=meta["lowercase"],
-                    stop_words=meta["stop_words"],
-                    ngram_range=(meta["min_ngram"], meta["max_ngram"]),
-                    max_df=meta["max_df"],
-                    min_df=meta["min_df"],
-                    max_features=meta["max_features"],
-                    analyzer=meta["analyzer"],
-                    vocabulary=vocabulary,
-                )
+        if not os.path.exists(featurizer_file):
+            return cls(meta)
+
+        vocabulary = utils.json_unpickle(featurizer_file)
 
-            return cls(meta, vectorizers)
+        share_vocabulary = meta["use_shared_vocab"]
+
+        if share_vocabulary:
+            vectorizers = cls._create_shared_vocab_vectorizers(
+                meta, vocabulary=vocabulary
+            )
         else:
-            return cls(meta)
+            vectorizers = cls._create_independent_vocab_vectorizers(
+                meta, vocabulary=vocabulary
+            )
+
+        return cls(meta, vectorizers)
diff --git a/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
new file mode 100644
index 000000000000..8a1a438e42b5
--- /dev/null
+++ b/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
@@ -0,0 +1,26 @@
+import logging
+import warnings
+
+from typing import Any, Dict, Optional, Text
+
+from rasa.nlu.featurizers.featurzier import Featurizer
+
+logger = logging.getLogger(__name__)
+
+
+class NGramFeaturizer(Featurizer):
+    def __init__(self, component_config: Optional[Dict[Text, Any]] = None):
+        super(NGramFeaturizer, self).__init__(component_config)
+
+        raise NotImplementedError(
+            "REMOVAL warning: You cannot use `NGramFeaturizer` anymore. "
+            "Please use `CountVectorsFeaturizer` instead. The following settings"
+            "match the previous `NGramFeaturizer`:"
+            ""
+            "- name: 'CountVectorsFeaturizer'"
+            "  analyzer: 'char_wb'"
+            "  min_ngram: 3"
+            "  max_ngram: 17"
+            "  max_features: 10"
+            "  min_df: 5"
+        )
diff --git a/rasa/nlu/featurizers/regex_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
similarity index 65%
rename from rasa/nlu/featurizers/regex_featurizer.py
rename to rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
index 05e5944a1653..1b7a94e9b691 100644
--- a/rasa/nlu/featurizers/regex_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
@@ -5,17 +5,19 @@
 import os
 import re
 import typing
+import scipy.sparse
 from typing import Any, Dict, Optional, Text, Union, List
 
 from rasa.nlu import utils
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.featurizers import Featurizer
+from rasa.nlu.featurizers.featurzier import Featurizer
 from rasa.nlu.training_data import Message, TrainingData
 import rasa.utils.io
 from rasa.nlu.constants import (
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_VECTOR_FEATURE_NAMES,
+    TOKENS_NAMES,
+    TEXT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
 )
 from rasa.constants import DOCS_BASE_URL
 
@@ -27,15 +29,22 @@
 
 class RegexFeaturizer(Featurizer):
 
-    provides = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    provides = [SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]]
 
-    requires = [MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE]]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
 
     def __init__(
         self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        known_patterns=None,
-        lookup_tables=None,
+        component_config: Dict[Text, Any] = None,
+        known_patterns: List[Dict[Text, Text]] = None,
+        lookup_tables: List[Dict[Text, Union[Text, List]]] = None,
     ) -> None:
 
         super().__init__(component_config)
@@ -44,6 +53,8 @@ def __init__(
         lookup_tables = lookup_tables or []
         self._add_lookup_table_regexes(lookup_tables)
 
+        self.return_sequence = self.component_config["return_sequence"]
+
     def train(
         self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
     ) -> None:
@@ -52,56 +63,68 @@ def train(
         self._add_lookup_table_regexes(training_data.lookup_tables)
 
         for example in training_data.training_examples:
-            updated = self._text_features_with_regex(example)
-            example.set(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE], updated)
+            for attribute in [TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE]:
+                self._text_features_with_regex(example, attribute)
 
     def process(self, message: Message, **kwargs: Any) -> None:
+        self._text_features_with_regex(message, TEXT_ATTRIBUTE)
 
-        updated = self._text_features_with_regex(message)
-        message.set(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE], updated)
-
-    def _text_features_with_regex(self, message) -> Any:
+    def _text_features_with_regex(self, message: Message, attribute: Text) -> None:
         if self.known_patterns:
-            extras = self.features_for_patterns(message)
-            return self._combine_with_existing_features(message, extras)
-        else:
-            return message.get(MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE])
+            extras = self._features_for_patterns(message, attribute)
+            features = self._combine_with_existing_sparse_features(
+                message, extras, feature_name=SPARSE_FEATURE_NAMES[attribute]
+            )
+            message.set(SPARSE_FEATURE_NAMES[attribute], features)
 
-    def _add_lookup_table_regexes(self, lookup_tables) -> None:
-        # appends the regex features from the lookup tables to
-        # self.known_patterns
+    def _add_lookup_table_regexes(
+        self, lookup_tables: List[Dict[Text, Union[Text, List]]]
+    ) -> None:
+        """appends the regex features from the lookup tables to self.known_patterns"""
         for table in lookup_tables:
             regex_pattern = self._generate_lookup_regex(table)
             lookup_regex = {"name": table["name"], "pattern": regex_pattern}
             self.known_patterns.append(lookup_regex)
 
-    def features_for_patterns(self, message) -> np.array:
+    def _features_for_patterns(
+        self, message: Message, attribute: Text
+    ) -> scipy.sparse.coo_matrix:
         """Checks which known patterns match the message.
 
         Given a sentence, returns a vector of {1,0} values indicating which
         regexes did match. Furthermore, if the
         message is tokenized, the function will mark all tokens with a dict
         relating the name of the regex to whether it was matched."""
+        tokens = message.get(TOKENS_NAMES[attribute], [])
 
-        found_patterns = []
-        for exp in self.known_patterns:
-            matches = re.finditer(exp["pattern"], message.text)
+        if self.return_sequence:
+            seq_length = len(tokens)
+        else:
+            seq_length = 1
+
+        vec = np.zeros([seq_length, len(self.known_patterns)])
+
+        for pattern_index, pattern in enumerate(self.known_patterns):
+            matches = re.finditer(pattern["pattern"], message.text)
             matches = list(matches)
-            found_patterns.append(False)
-            for token_index, t in enumerate(
-                message.get(MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE], [])
-            ):
+
+            for token_index, t in enumerate(tokens):
                 patterns = t.get("pattern", default={})
-                patterns[exp["name"]] = False
+                patterns[pattern["name"]] = False
+
+                if self.return_sequence:
+                    seq_index = token_index
+                else:
+                    seq_index = 0
 
                 for match in matches:
                     if t.offset < match.end() and t.end > match.start():
-                        patterns[exp["name"]] = True
-                        found_patterns[-1] = True
+                        patterns[pattern["name"]] = True
+                        vec[seq_index][pattern_index] = 1.0
 
                 t.set("pattern", patterns)
 
-        return np.array(found_patterns).astype(float)
+        return scipy.sparse.coo_matrix(vec)
 
     def _generate_lookup_regex(
         self, lookup_table: Dict[Text, Union[Text, List[Text]]]
diff --git a/rasa/nlu/registry.py b/rasa/nlu/registry.py
index 55bad6668c31..34a17a930bcc 100644
--- a/rasa/nlu/registry.py
+++ b/rasa/nlu/registry.py
@@ -19,12 +19,13 @@
 from rasa.nlu.extractors.entity_synonyms import EntitySynonymMapper
 from rasa.nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
 from rasa.nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
-from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-from rasa.nlu.featurizers.mitie_featurizer import MitieFeaturizer
-from rasa.nlu.featurizers.ngram_featurizer import NGramFeaturizer
-from rasa.nlu.featurizers.regex_featurizer import RegexFeaturizer
-from rasa.nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-from rasa.nlu.featurizers.convert_featurizer import ConveRTFeaturizer
+from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+    CountVectorsFeaturizer,
+)
+from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
+from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+from rasa.nlu.featurizers.dense_featurizer.convert_featurizer import ConveRTFeaturizer
 from rasa.nlu.model import Metadata
 from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
 from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
@@ -62,7 +63,6 @@
     # featurizers
     SpacyFeaturizer,
     MitieFeaturizer,
-    NGramFeaturizer,
     RegexFeaturizer,
     CountVectorsFeaturizer,
     ConveRTFeaturizer,
diff --git a/rasa/nlu/selectors/embedding_response_selector.py b/rasa/nlu/selectors/embedding_response_selector.py
index c27c4f00ae2a..1c8fbac28b1c 100644
--- a/rasa/nlu/selectors/embedding_response_selector.py
+++ b/rasa/nlu/selectors/embedding_response_selector.py
@@ -1,20 +1,13 @@
 import logging
 import typing
-from typing import Any, Dict, Optional, Text
+from typing import Any, Dict, Text
 
 from rasa.nlu.classifiers.embedding_intent_classifier import EmbeddingIntentClassifier
-from rasa.core.actions.action import RESPOND_PREFIX
 from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-    OPEN_UTTERANCE_PREDICTION_KEY,
-    OPEN_UTTERANCE_RANKING_KEY,
-    MESSAGE_SELECTOR_PROPERTY_NAME,
+    RESPONSE_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
+    RESPONSE_SELECTOR_PROPERTY_NAME,
     DEFAULT_OPEN_UTTERANCE_TYPE,
 )
 
@@ -50,7 +43,7 @@ class ResponseSelector(EmbeddingIntentClassifier):
 
     provides = ["response", "response_ranking"]
 
-    requires = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = [SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]]
 
     # default properties (DOC MARKER - don't remove)
     defaults = {
@@ -74,6 +67,8 @@ class ResponseSelector(EmbeddingIntentClassifier):
         # set random seed to any int to get reproducible results
         "random_seed": None,
         # embedding parameters
+        # default dense dimension used if no dense features are present
+        "dense_dim": 512,
         # dimension size of embedding vectors
         "embed_dim": 20,
         # the type of the similarity
@@ -129,46 +124,33 @@ def _set_message_property(
         message: "Message", prediction_dict: Dict[Text, Any], selector_key: Text
     ):
 
-        message_selector_properties = message.get(MESSAGE_SELECTOR_PROPERTY_NAME, {})
+        message_selector_properties = message.get(RESPONSE_SELECTOR_PROPERTY_NAME, {})
         message_selector_properties[selector_key] = prediction_dict
         message.set(
-            MESSAGE_SELECTOR_PROPERTY_NAME,
+            RESPONSE_SELECTOR_PROPERTY_NAME,
             message_selector_properties,
             add_to_output=True,
         )
 
     def preprocess_train_data(self, training_data):
-        """Performs sanity checks on training data, extracts encodings for labels and prepares data for training"""
-
+        """Performs sanity checks on training data, extracts encodings for labels
+        and prepares data for training"""
         if self.retrieval_intent:
             training_data = training_data.filter_by_intent(self.retrieval_intent)
 
         label_id_dict = self._create_label_id_dict(
-            training_data, attribute=MESSAGE_RESPONSE_ATTRIBUTE
+            training_data, attribute=RESPONSE_ATTRIBUTE
         )
 
         self.inverted_label_dict = {v: k for k, v in label_id_dict.items()}
-        self._encoded_all_label_ids = self._create_encoded_label_ids(
-            training_data,
-            label_id_dict,
-            attribute=MESSAGE_RESPONSE_ATTRIBUTE,
-            attribute_feature_name=MESSAGE_VECTOR_FEATURE_NAMES[
-                MESSAGE_RESPONSE_ATTRIBUTE
-            ],
-        )
-
-        # check if number of negatives is less than number of label_ids
-        logger.debug(
-            "Check if num_neg {} is smaller than "
-            "number of label_ids {}, "
-            "else set num_neg to the number of label_ids - 1"
-            "".format(self.num_neg, self._encoded_all_label_ids.shape[0])
+        self._label_data = self._create_label_data(
+            training_data, label_id_dict, attribute=RESPONSE_ATTRIBUTE
         )
-        # noinspection PyAttributeOutsideInit
-        self.num_neg = min(self.num_neg, self._encoded_all_label_ids.shape[0] - 1)
 
         session_data = self._create_session_data(
-            training_data, label_id_dict, attribute=MESSAGE_RESPONSE_ATTRIBUTE
+            training_data.intent_examples,
+            label_id_dict,
+            label_attribute=RESPONSE_ATTRIBUTE,
         )
 
         self.check_input_dimension_consistency(session_data)
diff --git a/rasa/nlu/test.py b/rasa/nlu/test.py
index cbce29b25198..dd4f528e6094 100644
--- a/rasa/nlu/test.py
+++ b/rasa/nlu/test.py
@@ -23,7 +23,7 @@
 from rasa.constants import TEST_DATA_FILE, TRAIN_DATA_FILE
 from rasa.nlu.constants import (
     DEFAULT_OPEN_UTTERANCE_TYPE,
-    MESSAGE_SELECTOR_PROPERTY_NAME,
+    RESPONSE_SELECTOR_PROPERTY_NAME,
     OPEN_UTTERANCE_PREDICTION_KEY,
 )
 from rasa.model import get_model
@@ -33,7 +33,7 @@
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.model import Interpreter, Trainer, TrainingData
 from rasa.nlu.components import Component
-from rasa.nlu.tokenizers import Token
+from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.core.constants import RESPOND_PREFIX
 
 logger = logging.getLogger(__name__)
@@ -835,7 +835,7 @@ def determine_token_labels(
         entity type
     """
 
-    if len(entities) == 0:
+    if entities is None or len(entities) == 0:
         return "O"
     if not do_extractors_support_overlap(extractors) and do_entities_overlap(entities):
         raise ValueError("The possible entities should not overlap")
@@ -958,7 +958,7 @@ def get_eval_data(
 
             # including all examples here. Empty response examples are filtered at the time of metric calculation
             intent_target = example.get("intent", "")
-            selector_properties = result.get(MESSAGE_SELECTOR_PROPERTY_NAME, {})
+            selector_properties = result.get(RESPONSE_SELECTOR_PROPERTY_NAME, {})
 
             if intent_target in available_response_selector_types:
                 response_prediction_key = intent_target
diff --git a/rasa/nlu/tokenizers/__init__.py b/rasa/nlu/tokenizers/__init__.py
index 0067d51265b8..e69de29bb2d1 100644
--- a/rasa/nlu/tokenizers/__init__.py
+++ b/rasa/nlu/tokenizers/__init__.py
@@ -1,38 +0,0 @@
-import functools
-
-
-class Tokenizer:
-    pass
-
-
-@functools.total_ordering
-class Token:
-    def __init__(self, text, offset, data=None):
-        self.offset = offset
-        self.text = text
-        self.end = offset + len(text)
-        self.data = data if data else {}
-
-    def set(self, prop, info):
-        self.data[prop] = info
-
-    def get(self, prop, default=None):
-        return self.data.get(prop, default)
-
-    def __eq__(self, other):
-        if not isinstance(other, Token):
-            return NotImplemented
-        return (self.offset, self.end, self.text) == (
-            other.offset,
-            other.end,
-            other.text,
-        )
-
-    def __lt__(self, other):
-        if not isinstance(other, Token):
-            return NotImplemented
-        return (self.offset, self.end, self.text) < (
-            other.offset,
-            other.end,
-            other.text,
-        )
diff --git a/rasa/nlu/tokenizers/jieba_tokenizer.py b/rasa/nlu/tokenizers/jieba_tokenizer.py
index 0a12af7871e7..73e6e0541ab3 100644
--- a/rasa/nlu/tokenizers/jieba_tokenizer.py
+++ b/rasa/nlu/tokenizers/jieba_tokenizer.py
@@ -7,17 +7,14 @@
 
 from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.tokenizers import Token, Tokenizer
+from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
 from rasa.nlu.training_data import Message, TrainingData
 
 from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
+    INTENT_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
     MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
 )
 
 logger = logging.getLogger(__name__)
@@ -27,9 +24,9 @@
     from rasa.nlu.model import Metadata
 
 
-class JiebaTokenizer(Tokenizer, Component):
+class JiebaTokenizer(Tokenizer):
 
-    provides = [MESSAGE_TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+    provides = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
 
     language_list = ["zh"]
 
@@ -39,6 +36,8 @@ class JiebaTokenizer(Tokenizer, Component):
         "intent_tokenization_flag": False,
         # Symbol on which intent should be split
         "intent_split_symbol": "_",
+        # add __CLS__ token to the end of the list of tokens
+        "use_cls_token": False,
     }  # default don't load custom dictionary
 
     def __init__(self, component_config: Dict[Text, Any] = None) -> None:
@@ -90,30 +89,32 @@ def train(
 
                 if example.get(attribute) is not None:
                     example.set(
-                        MESSAGE_TOKENS_NAMES[attribute],
+                        TOKENS_NAMES[attribute],
                         self.tokenize(example.get(attribute), attribute),
                     )
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
         message.set(
-            MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-            self.tokenize(message.text, MESSAGE_TEXT_ATTRIBUTE),
+            TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text, TEXT_ATTRIBUTE)
         )
 
     def preprocess_text(self, text: Text, attribute: Text) -> Text:
 
-        if attribute == MESSAGE_INTENT_ATTRIBUTE and self.intent_tokenization_flag:
+        if attribute == INTENT_ATTRIBUTE and self.intent_tokenization_flag:
             return " ".join(text.split(self.intent_split_symbol))
         else:
             return text
 
-    def tokenize(self, text: Text, attribute=MESSAGE_TEXT_ATTRIBUTE) -> List[Token]:
+    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
         import jieba
 
         text = self.preprocess_text(text, attribute)
         tokenized = jieba.tokenize(text)
         tokens = [Token(word, start) for (word, start, end) in tokenized]
+
+        self.add_cls_token(tokens, attribute)
+
         return tokens
 
     @classmethod
diff --git a/rasa/nlu/tokenizers/mitie_tokenizer.py b/rasa/nlu/tokenizers/mitie_tokenizer.py
index 1032696fa4a2..d775bddc0449 100644
--- a/rasa/nlu/tokenizers/mitie_tokenizer.py
+++ b/rasa/nlu/tokenizers/mitie_tokenizer.py
@@ -1,21 +1,21 @@
 from typing import Any, List, Text
 
-from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.tokenizers import Token, Tokenizer
+from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
 from rasa.nlu.training_data import Message, TrainingData
 
-from rasa.nlu.constants import (
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-)
+from rasa.nlu.constants import TEXT_ATTRIBUTE, TOKENS_NAMES, MESSAGE_ATTRIBUTES
 from rasa.utils.io import DEFAULT_ENCODING
 
 
-class MitieTokenizer(Tokenizer, Component):
+class MitieTokenizer(Tokenizer):
 
-    provides = [MESSAGE_TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+    provides = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    defaults = {
+        # add __CLS__ token to the end of the list of tokens
+        "use_cls_token": False
+    }
 
     @classmethod
     def required_packages(cls) -> List[Text]:
@@ -31,15 +31,13 @@ def train(
 
                 if example.get(attribute) is not None:
                     example.set(
-                        MESSAGE_TOKENS_NAMES[attribute],
-                        self.tokenize(example.get(attribute)),
+                        TOKENS_NAMES[attribute],
+                        self.tokenize(example.get(attribute), attribute),
                     )
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
-        message.set(
-            MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE], self.tokenize(message.text)
-        )
+        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text))
 
     def _token_from_offset(
         self, text: bytes, offset: int, encoded_sentence: bytes
@@ -49,7 +47,7 @@ def _token_from_offset(
             self._byte_to_char_offset(encoded_sentence, offset),
         )
 
-    def tokenize(self, text: Text) -> List[Token]:
+    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
         import mitie
 
         encoded_sentence = text.encode(DEFAULT_ENCODING)
@@ -58,6 +56,9 @@ def tokenize(self, text: Text) -> List[Token]:
             self._token_from_offset(token, offset, encoded_sentence)
             for token, offset in tokenized
         ]
+
+        self.add_cls_token(tokens, attribute)
+
         return tokens
 
     @staticmethod
diff --git a/rasa/nlu/tokenizers/spacy_tokenizer.py b/rasa/nlu/tokenizers/spacy_tokenizer.py
index 56a43e48f2bc..ed14d65f8e38 100644
--- a/rasa/nlu/tokenizers/spacy_tokenizer.py
+++ b/rasa/nlu/tokenizers/spacy_tokenizer.py
@@ -1,36 +1,31 @@
 import typing
-from typing import Any
+from typing import Any, Text, List
 
-from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.tokenizers import Token, Tokenizer
+from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
 from rasa.nlu.training_data import Message, TrainingData
 
 from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
-    SPACY_FEATURIZABLE_ATTRIBUTES,
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
+    SPACY_DOCS,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
 )
 
 if typing.TYPE_CHECKING:
     from spacy.tokens.doc import Doc  # pytype: disable=import-error
 
 
-class SpacyTokenizer(Tokenizer, Component):
+class SpacyTokenizer(Tokenizer):
 
-    provides = [
-        MESSAGE_TOKENS_NAMES[attribute] for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
-    ]
+    provides = [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
 
-    requires = [
-        MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
-    ]
+    requires = [SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
+
+    defaults = {
+        # add __CLS__ token to the end of the list of tokens
+        "use_cls_token": False
+    }
 
     def train(
         self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
@@ -38,26 +33,25 @@ def train(
 
         for example in training_data.training_examples:
 
-            for attribute in SPACY_FEATURIZABLE_ATTRIBUTES:
+            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
 
                 attribute_doc = self.get_doc(example, attribute)
 
                 if attribute_doc is not None:
                     example.set(
-                        MESSAGE_TOKENS_NAMES[attribute], self.tokenize(attribute_doc)
+                        TOKENS_NAMES[attribute], self.tokenize(attribute_doc, attribute)
                     )
 
-    def get_doc(self, message, attribute) -> Any:
-
-        return message.get(MESSAGE_SPACY_FEATURES_NAMES[attribute])
+    def get_doc(self, message: Message, attribute: Text) -> "Doc":
+        return message.get(SPACY_DOCS[attribute])
 
     def process(self, message: Message, **kwargs: Any) -> None:
-
         message.set(
-            MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-            self.tokenize(self.get_doc(message, MESSAGE_TEXT_ATTRIBUTE)),
+            TOKENS_NAMES[TEXT_ATTRIBUTE],
+            self.tokenize(self.get_doc(message, TEXT_ATTRIBUTE), TEXT_ATTRIBUTE),
         )
 
-    def tokenize(self, doc: "Doc") -> typing.List[Token]:
-
-        return [Token(t.text, t.idx) for t in doc]
+    def tokenize(self, doc: "Doc", attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
+        tokens = [Token(t.text, t.idx, lemma=t.lemma_) for t in doc]
+        self.add_cls_token(tokens, attribute)
+        return tokens
diff --git a/rasa/nlu/tokenizers/tokenizer.py b/rasa/nlu/tokenizers/tokenizer.py
new file mode 100644
index 000000000000..fc63dfebc207
--- /dev/null
+++ b/rasa/nlu/tokenizers/tokenizer.py
@@ -0,0 +1,76 @@
+import logging
+
+from typing import Text, List, Optional, Dict, Any
+
+from rasa.nlu.components import Component
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE, CLS_TOKEN
+
+logger = logging.getLogger(__name__)
+
+
+class Token(object):
+    def __init__(
+        self,
+        text: Text,
+        offset: int,
+        data: Optional[Dict[Text, Any]] = None,
+        lemma: Optional[Text] = None,
+    ):
+        self.offset = offset
+        self.text = text
+        self.end = offset + len(text)
+        self.data = data if data else {}
+        self.lemma = lemma or text
+
+    def set(self, prop: Text, info: Any):
+        self.data[prop] = info
+
+    def get(self, prop: Text, default: Optional[Any] = None):
+        return self.data.get(prop, default)
+
+    def __eq__(self, other):
+        if not isinstance(other, Token):
+            return NotImplemented
+        return (self.offset, self.end, self.text, self.lemma) == (
+            other.offset,
+            other.end,
+            other.text,
+            other.lemma,
+        )
+
+    def __lt__(self, other):
+        if not isinstance(other, Token):
+            return NotImplemented
+        return (self.offset, self.end, self.text, self.lemma) < (
+            other.offset,
+            other.end,
+            other.text,
+            other.lemma,
+        )
+
+
+class Tokenizer(Component):
+    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
+        super(Tokenizer, self).__init__(component_config)
+
+        try:
+            self.use_cls_token = self.component_config["use_cls_token"]
+        except KeyError:
+            raise KeyError(
+                "No default value for 'use_cls_token' was set. Please, "
+                "add it to the default dict of the tokenizer."
+            )
+
+    def add_cls_token(
+        self, tokens: List[Token], attribute: Text = TEXT_ATTRIBUTE
+    ) -> List[Token]:
+        if (
+            attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]
+            and self.use_cls_token
+            and tokens
+        ):
+            # +1 to have a space between the last token and the __cls__ token
+            idx = tokens[-1].offset + len(tokens[-1].text) + 1
+            tokens.append(Token(CLS_TOKEN, idx))
+
+        return tokens
diff --git a/rasa/nlu/tokenizers/whitespace_tokenizer.py b/rasa/nlu/tokenizers/whitespace_tokenizer.py
index 1a5cc438ea2c..4b2f49b15e73 100644
--- a/rasa/nlu/tokenizers/whitespace_tokenizer.py
+++ b/rasa/nlu/tokenizers/whitespace_tokenizer.py
@@ -1,32 +1,30 @@
 import re
 from typing import Any, Dict, List, Text
 
-from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.tokenizers import Token, Tokenizer
+from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
 from rasa.nlu.training_data import Message, TrainingData
 from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_TOKENS_NAMES,
+    INTENT_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
     MESSAGE_ATTRIBUTES,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    MESSAGE_VECTOR_FEATURE_NAMES,
 )
 
 
-class WhitespaceTokenizer(Tokenizer, Component):
+class WhitespaceTokenizer(Tokenizer):
 
-    provides = [MESSAGE_TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+    provides = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
 
     defaults = {
         # Flag to check whether to split intents
         "intent_tokenization_flag": False,
         # Symbol on which intent should be split
         "intent_split_symbol": "_",
-        # text will be tokenized with case sensitive as default
+        # Text will be tokenized with case sensitive as default
         "case_sensitive": True,
+        # add __CLS__ token to the end of the list of tokens
+        "use_cls_token": False,
     }
 
     def __init__(self, component_config: Dict[Text, Any] = None) -> None:
@@ -48,24 +46,21 @@ def train(
             for attribute in MESSAGE_ATTRIBUTES:
                 if example.get(attribute) is not None:
                     example.set(
-                        MESSAGE_TOKENS_NAMES[attribute],
+                        TOKENS_NAMES[attribute],
                         self.tokenize(example.get(attribute), attribute),
                     )
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
-        message.set(
-            MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE], self.tokenize(message.text)
-        )
+        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text))
 
-    def tokenize(
-        self, text: Text, attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> List[Token]:
+    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
 
         if not self.case_sensitive:
             text = text.lower()
-        # remove 'not a word character' if
-        if attribute != MESSAGE_INTENT_ATTRIBUTE:
+
+        if attribute != INTENT_ATTRIBUTE:
+            # remove 'not a word character' if
             words = re.sub(
                 # there is a space or an end of a string after it
                 r"[^\w#@&]+(?=\s|$)|"
@@ -79,6 +74,9 @@ def tokenize(
                 " ",
                 text,
             ).split()
+            # if we removed everything like smiles `:)`, use the whole text as 1 token
+            if not words:
+                words = [text]
         else:
             words = (
                 text.split(self.intent_split_symbol)
@@ -88,9 +86,13 @@ def tokenize(
 
         running_offset = 0
         tokens = []
+
         for word in words:
             word_offset = text.index(word, running_offset)
             word_len = len(word)
             running_offset = word_offset + word_len
             tokens.append(Token(word, word_offset))
+
+        self.add_cls_token(tokens, attribute)
+
         return tokens
diff --git a/rasa/nlu/training_data/formats/markdown.py b/rasa/nlu/training_data/formats/markdown.py
index b3d7fec31ba1..e62f5743e7a2 100644
--- a/rasa/nlu/training_data/formats/markdown.py
+++ b/rasa/nlu/training_data/formats/markdown.py
@@ -12,8 +12,8 @@
 )
 from rasa.nlu.utils import build_entity
 from rasa.nlu.constants import (
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_RESPONSE_KEY_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    RESPONSE_KEY_ATTRIBUTE,
     RESPONSE_IDENTIFIER_DELIMITER,
 )
 
@@ -218,7 +218,7 @@ def _generate_training_examples_md(self, training_data: "TrainingData") -> Text:
 
         # Sort by intent while keeping basic intent order
         for example in [e.as_dict_nlu() for e in training_data.training_examples]:
-            intent = example[MESSAGE_INTENT_ATTRIBUTE]
+            intent = example[INTENT_ATTRIBUTE]
             training_examples.setdefault(intent, [])
             training_examples[intent].append(example)
 
diff --git a/rasa/nlu/training_data/formats/rasa.py b/rasa/nlu/training_data/formats/rasa.py
index fc73cbdecb93..3a6dd286bbb0 100644
--- a/rasa/nlu/training_data/formats/rasa.py
+++ b/rasa/nlu/training_data/formats/rasa.py
@@ -13,9 +13,9 @@
 from rasa.nlu.training_data.util import transform_entity_synonyms
 from rasa.nlu.utils import json_to_string
 from rasa.nlu.constants import (
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_RESPONSE_KEY_ATTRIBUTE,
-    MESSAGE_RESPONSE_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    RESPONSE_KEY_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
     RESPONSE_IDENTIFIER_DELIMITER,
 )
 
diff --git a/rasa/nlu/training_data/message.py b/rasa/nlu/training_data/message.py
index ab84af948938..7a661a748923 100644
--- a/rasa/nlu/training_data/message.py
+++ b/rasa/nlu/training_data/message.py
@@ -1,11 +1,11 @@
 from typing import Any, Optional, Tuple, Text
 
 from rasa.nlu.constants import (
-    MESSAGE_ENTITIES_ATTRIBUTE,
-    MESSAGE_INTENT_ATTRIBUTE,
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_RESPONSE_KEY_ATTRIBUTE,
-    MESSAGE_TEXT_ATTRIBUTE,
+    ENTITIES_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    RESPONSE_KEY_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
     RESPONSE_IDENTIFIER_DELIMITER,
 )
 from rasa.nlu.utils import ordered
@@ -30,7 +30,7 @@ def set(self, prop, info, add_to_output=False) -> None:
             self.output_properties.add(prop)
 
     def get(self, prop, default=None) -> Any:
-        if prop == MESSAGE_TEXT_ATTRIBUTE:
+        if prop == TEXT_ATTRIBUTE:
             return self.text
         return self.data.get(prop, default)
 
@@ -38,10 +38,10 @@ def as_dict_nlu(self) -> dict:
         """Get dict representation of message as it would appear in training data"""
 
         d = self.as_dict()
-        if d.get(MESSAGE_INTENT_ATTRIBUTE, None):
-            d[MESSAGE_INTENT_ATTRIBUTE] = self.get_combined_intent_response_key()
-        d.pop(MESSAGE_RESPONSE_KEY_ATTRIBUTE, None)
-        d.pop(MESSAGE_RESPONSE_ATTRIBUTE, None)
+        if d.get(INTENT_ATTRIBUTE, None):
+            d[INTENT_ATTRIBUTE] = self.get_combined_intent_response_key()
+        d.pop(RESPONSE_KEY_ATTRIBUTE, None)
+        d.pop(RESPONSE_ATTRIBUTE, None)
         return d
 
     def as_dict(self, only_output_properties=False) -> dict:
@@ -73,18 +73,18 @@ def build(cls, text, intent=None, entities=None) -> "Message":
         data = {}
         if intent:
             split_intent, response_key = cls.separate_intent_response_key(intent)
-            data[MESSAGE_INTENT_ATTRIBUTE] = split_intent
+            data[INTENT_ATTRIBUTE] = split_intent
             if response_key:
-                data[MESSAGE_RESPONSE_KEY_ATTRIBUTE] = response_key
+                data[RESPONSE_KEY_ATTRIBUTE] = response_key
         if entities:
-            data[MESSAGE_ENTITIES_ATTRIBUTE] = entities
+            data[ENTITIES_ATTRIBUTE] = entities
         return cls(text, data)
 
     def get_combined_intent_response_key(self) -> Text:
         """Get intent as it appears in training data"""
 
-        intent = self.get(MESSAGE_INTENT_ATTRIBUTE)
-        response_key = self.get(MESSAGE_RESPONSE_KEY_ATTRIBUTE)
+        intent = self.get(INTENT_ATTRIBUTE)
+        response_key = self.get(RESPONSE_KEY_ATTRIBUTE)
         response_key_suffix = (
             f"{RESPONSE_IDENTIFIER_DELIMITER}{response_key}" if response_key else ""
         )
diff --git a/rasa/nlu/training_data/training_data.py b/rasa/nlu/training_data/training_data.py
index 6ac11824f8d9..1cf1d818d346 100644
--- a/rasa/nlu/training_data/training_data.py
+++ b/rasa/nlu/training_data/training_data.py
@@ -9,10 +9,7 @@
 
 import rasa.nlu.utils
 import rasa.utils.common as rasa_utils
-from rasa.nlu.constants import (
-    MESSAGE_RESPONSE_ATTRIBUTE,
-    MESSAGE_RESPONSE_KEY_ATTRIBUTE,
-)
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE, RESPONSE_KEY_ATTRIBUTE
 from rasa.nlu.training_data.message import Message
 from rasa.nlu.training_data.util import check_duplicate_synonym
 from rasa.nlu.utils import list_to_str
@@ -180,7 +177,7 @@ def sort_regex_features(self) -> None:
     def fill_response_phrases(self) -> None:
         """Set response phrase for all examples by looking up NLG stories"""
         for example in self.training_examples:
-            response_key = example.get(MESSAGE_RESPONSE_KEY_ATTRIBUTE)
+            response_key = example.get(RESPONSE_KEY_ATTRIBUTE)
             # if response_key is None, that means the corresponding intent is not a retrieval intent
             # and hence no response text needs to be fetched.
             # If response_key is set, fetch the corresponding response text
@@ -190,7 +187,7 @@ def fill_response_phrases(self) -> None:
                 assistant_utterances = self.nlg_stories.get(story_lookup_intent, [])
                 if assistant_utterances:
                     # selecting only first assistant utterance for now
-                    example.set(MESSAGE_RESPONSE_ATTRIBUTE, assistant_utterances[0])
+                    example.set(RESPONSE_ATTRIBUTE, assistant_utterances[0])
                 else:
                     raise ValueError(
                         "No response phrases found for {}. Check training data "
@@ -387,11 +384,9 @@ def build_nlg_stories_from_examples(examples) -> Dict[Text, list]:
 
         nlg_stories = {}
         for ex in examples:
-            if ex.get(MESSAGE_RESPONSE_KEY_ATTRIBUTE) and ex.get(
-                MESSAGE_RESPONSE_ATTRIBUTE
-            ):
+            if ex.get(RESPONSE_KEY_ATTRIBUTE) and ex.get(RESPONSE_ATTRIBUTE):
                 nlg_stories[ex.get_combined_intent_response_key()] = [
-                    ex.get(MESSAGE_RESPONSE_ATTRIBUTE)
+                    ex.get(RESPONSE_ATTRIBUTE)
                 ]
         return nlg_stories
 
diff --git a/rasa/nlu/utils/spacy_utils.py b/rasa/nlu/utils/spacy_utils.py
index 2077fb1e3c96..f06ff5f7459f 100644
--- a/rasa/nlu/utils/spacy_utils.py
+++ b/rasa/nlu/utils/spacy_utils.py
@@ -14,17 +14,12 @@
     from spacy.tokens.doc import Doc  # pytype: disable=import-error
     from rasa.nlu.model import Metadata
 
-from rasa.nlu.constants import (
-    MESSAGE_TEXT_ATTRIBUTE,
-    MESSAGE_SPACY_FEATURES_NAMES,
-    SPACY_FEATURIZABLE_ATTRIBUTES,
-)
+from rasa.nlu.constants import TEXT_ATTRIBUTE, SPACY_DOCS, DENSE_FEATURIZABLE_ATTRIBUTES
 
 
 class SpacyNLP(Component):
     provides = ["spacy_nlp"] + [
-        MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES
+        SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
     ]
 
     defaults = {
@@ -191,7 +186,7 @@ def docs_for_training_data(
         self, training_data: TrainingData
     ) -> Dict[Text, List[Any]]:
         attribute_docs = {}
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES:
+        for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
             texts = [self.get_text(e, attribute) for e in training_data.intent_examples]
             # Index and freeze indices of the training samples for preserving the order
             # after processing the data.
@@ -223,23 +218,18 @@ def train(
 
         attribute_docs = self.docs_for_training_data(training_data)
 
-        for attribute in SPACY_FEATURIZABLE_ATTRIBUTES:
+        for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
 
             for idx, example in enumerate(training_data.training_examples):
                 example_attribute_doc = attribute_docs[attribute][idx]
                 if len(example_attribute_doc):
                     # If length is 0, that means the initial text feature was None and was replaced by ''
                     # in preprocess method
-                    example.set(
-                        MESSAGE_SPACY_FEATURES_NAMES[attribute], example_attribute_doc
-                    )
+                    example.set(SPACY_DOCS[attribute], example_attribute_doc)
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
-        message.set(
-            MESSAGE_SPACY_FEATURES_NAMES[MESSAGE_TEXT_ATTRIBUTE],
-            self.doc_for_text(message.text),
-        )
+        message.set(SPACY_DOCS[TEXT_ATTRIBUTE], self.doc_for_text(message.text))
 
     @classmethod
     def load(
diff --git a/rasa/utils/train_utils.py b/rasa/utils/train_utils.py
index 10b738933763..b7a8226cdff7 100644
--- a/rasa/utils/train_utils.py
+++ b/rasa/utils/train_utils.py
@@ -1,5 +1,6 @@
-from collections import namedtuple
+from collections import defaultdict
 import logging
+import scipy.sparse
 import typing
 from typing import List, Optional, Text, Dict, Tuple, Union, Generator, Callable, Any
 import numpy as np
@@ -20,103 +21,191 @@
 
 # avoid warning println on contrib import - remove for tf 2
 tf.contrib._warning = None
+
 logger = logging.getLogger(__name__)
 
-# namedtuple for all tf session related data
-SessionData = namedtuple("SessionData", ("X", "Y", "label_ids"))
+
+# type for all tf session related data
+SessionDataType = Dict[Text, List[np.ndarray]]
 
 
 def load_tf_config(config: Dict[Text, Any]) -> Optional[tf.compat.v1.ConfigProto]:
     """Prepare `tf.compat.v1.ConfigProto` for training"""
+
     if config.get("tf_config") is not None:
         return tf.compat.v1.ConfigProto(**config.pop("tf_config"))
     else:
         return None
 
 
+def create_label_ids(label_ids: "np.ndarray") -> "np.ndarray":
+    """Convert various size label_ids into single dim array.
+
+    for multi-label y, map each distinct row to a string repr
+    using join because str(row) uses an ellipsis if len(row) > 1000.
+    Idea taken from sklearn's stratify split.
+    """
+
+    if label_ids.ndim == 1:
+        return label_ids
+
+    if label_ids.ndim == 2 and label_ids.shape[-1] == 1:
+        return label_ids[:, 0]
+
+    if label_ids.ndim == 2:
+        return np.array([" ".join(row.astype("str")) for row in label_ids])
+
+    if label_ids.ndim == 3 and label_ids.shape[-1] == 1:
+        return np.array([" ".join(row.astype("str")) for row in label_ids[:, :, 0]])
+
+    raise ValueError("Unsupported label_ids dimensions")
+
+
 # noinspection PyPep8Naming
 def train_val_split(
-    session_data: "SessionData", evaluate_on_num_examples: int, random_seed: int
-) -> Tuple["SessionData", "SessionData"]:
+    session_data: SessionDataType,
+    evaluate_on_num_examples: int,
+    random_seed: int,
+    label_key: Text,
+) -> Tuple[SessionDataType, SessionDataType]:
     """Create random hold out validation set using stratified split."""
 
-    label_counts = dict(
-        zip(*np.unique(session_data.label_ids, return_counts=True, axis=0))
+    if label_key not in session_data or len(session_data[label_key]) > 1:
+        raise ValueError(f"Key '{label_key}' not in SessionData.")
+
+    label_ids = create_label_ids(session_data[label_key][0])
+
+    label_counts = dict(zip(*np.unique(label_ids, return_counts=True, axis=0)))
+
+    check_train_test_sizes(evaluate_on_num_examples, label_counts, session_data)
+
+    counts = np.array([label_counts[label] for label in label_ids])
+
+    multi_values = [v[counts > 1] for values in session_data.values() for v in values]
+
+    solo_values = [v[counts == 1] for values in session_data.values() for v in values]
+
+    output_values = train_test_split(
+        *multi_values,
+        test_size=evaluate_on_num_examples,
+        random_state=random_seed,
+        stratify=label_ids[counts > 1],
     )
 
-    if evaluate_on_num_examples >= len(session_data.X) - len(label_counts):
+    session_data_train, session_data_val = convert_train_test_split(
+        output_values, session_data, solo_values
+    )
+
+    return session_data_train, session_data_val
+
+
+def check_train_test_sizes(
+    evaluate_on_num_examples: int,
+    label_counts: Dict[Any, int],
+    session_data: SessionDataType,
+):
+    """Check whether the evaluation data set is too large or too small."""
+
+    num_examples = get_number_of_examples(session_data)
+
+    if evaluate_on_num_examples >= num_examples - len(label_counts):
         raise ValueError(
-            "Validation set of {} is too large. Remaining train set "
-            "should be at least equal to number of classes {}."
-            "".format(evaluate_on_num_examples, len(label_counts))
+            f"Validation set of {evaluate_on_num_examples} is too large. Remaining "
+            f"train set should be at least equal to number of classes "
+            f"{len(label_counts)}."
         )
     elif evaluate_on_num_examples < len(label_counts):
         raise ValueError(
-            "Validation set of {} is too small. It should be "
-            "at least equal to number of classes {}."
-            "".format(evaluate_on_num_examples, len(label_counts))
+            f"Validation set of {evaluate_on_num_examples} is too small. It should be "
+            "at least equal to number of classes {label_counts}."
         )
 
-    counts = np.array([label_counts[label] for label in session_data.label_ids])
 
-    multi_X = session_data.X[counts > 1]
-    multi_Y = session_data.Y[counts > 1]
-    multi_label_ids = session_data.label_ids[counts > 1]
+def convert_train_test_split(
+    output_values: List[Any], session_data: SessionDataType, solo_values: List[Any]
+) -> Tuple[SessionDataType, SessionDataType]:
+    """Convert the output of sklearn.model_selection.train_test_split into train and
+    eval session data."""
 
-    solo_X = session_data.X[counts == 1]
-    solo_Y = session_data.Y[counts == 1]
-    solo_label_ids = session_data.label_ids[counts == 1]
+    session_data_train = defaultdict(list)
+    session_data_val = defaultdict(list)
 
-    (X_train, X_val, Y_train, Y_val, label_ids_train, label_ids_val) = train_test_split(
-        multi_X,
-        multi_Y,
-        multi_label_ids,
-        test_size=evaluate_on_num_examples,
-        random_state=random_seed,
-        stratify=multi_label_ids,
-    )
-    X_train = np.concatenate([X_train, solo_X])
-    Y_train = np.concatenate([Y_train, solo_Y])
-    label_ids_train = np.concatenate([label_ids_train, solo_label_ids])
+    # output_values = x_train, x_val, y_train, y_val, z_train, z_val, etc.
+    # order is kept, e.g. same order as session data keys
 
-    return (
-        SessionData(X=X_train, Y=Y_train, label_ids=label_ids_train),
-        SessionData(X=X_val, Y=Y_val, label_ids=label_ids_val),
-    )
+    # train datasets have an even index
+    index = 0
+    for key, values in session_data.items():
+        for _ in range(len(values)):
+            session_data_train[key].append(
+                combine_features(output_values[index * 2], solo_values[index])
+            )
+            index += 1
 
+    # val datasets have an odd index
+    index = 0
+    for key, values in session_data.items():
+        for _ in range(len(values)):
+            session_data_val[key].append(output_values[(index * 2) + 1])
+            index += 1
 
-def shuffle_session_data(session_data: "SessionData") -> "SessionData":
+    return session_data_train, session_data_val
+
+
+def combine_features(
+    feature_1: Union[np.ndarray, scipy.sparse.spmatrix],
+    feature_2: Union[np.ndarray, scipy.sparse.spmatrix],
+) -> Union[np.ndarray, scipy.sparse.spmatrix]:
+    """Concatenate features."""
+
+    if isinstance(feature_1, scipy.sparse.spmatrix) and isinstance(
+        feature_2, scipy.sparse.spmatrix
+    ):
+        if feature_2.shape[0] == 0:
+            return feature_1
+        if feature_1.shape[0] == 0:
+            return feature_2
+        return scipy.sparse.vstack([feature_1, feature_2])
+
+    return np.concatenate([feature_1, feature_2])
+
+
+def shuffle_session_data(session_data: SessionDataType) -> SessionDataType:
     """Shuffle session data."""
 
-    ids = np.random.permutation(len(session_data.X))
-    return SessionData(
-        X=session_data.X[ids],
-        Y=session_data.Y[ids],
-        label_ids=session_data.label_ids[ids],
-    )
+    data_points = get_number_of_examples(session_data)
+    ids = np.random.permutation(data_points)
+    return session_data_for_ids(session_data, ids)
+
+
+def session_data_for_ids(session_data: SessionDataType, ids: np.ndarray):
+    """Filter session data by ids."""
 
+    new_session_data = defaultdict(list)
+    for k, values in session_data.items():
+        for v in values:
+            new_session_data[k].append(v[ids])
+    return new_session_data
 
-def split_session_data_by_label(
-    session_data: "SessionData", unique_label_ids: "np.ndarray"
-) -> List["SessionData"]:
+
+def split_session_data_by_label_ids(
+    session_data: SessionDataType,
+    label_ids: "np.ndarray",
+    unique_label_ids: "np.ndarray",
+) -> List[SessionDataType]:
     """Reorganize session data into a list of session data with the same labels."""
 
     label_data = []
     for label_id in unique_label_ids:
-        label_data.append(
-            SessionData(
-                X=session_data.X[session_data.label_ids == label_id],
-                Y=session_data.Y[session_data.label_ids == label_id],
-                label_ids=session_data.label_ids[session_data.label_ids == label_id],
-            )
-        )
+        ids = label_ids == label_id
+        label_data.append(session_data_for_ids(session_data, ids))
     return label_data
 
 
 # noinspection PyPep8Naming
 def balance_session_data(
-    session_data: "SessionData", batch_size: int, shuffle: bool
-) -> "SessionData":
+    session_data: SessionDataType, batch_size: int, shuffle: bool, label_key: Text
+) -> SessionDataType:
     """Mix session data to account for class imbalance.
 
     This batching strategy puts rare classes approximately in every other batch,
@@ -124,21 +213,28 @@ def balance_session_data(
     that more populated classes should appear more often.
     """
 
-    num_examples = len(session_data.X)
+    if label_key not in session_data or len(session_data[label_key]) > 1:
+        raise ValueError(f"Key '{label_key}' not in SessionDataType.")
+
+    label_ids = create_label_ids(session_data[label_key][0])
+
     unique_label_ids, counts_label_ids = np.unique(
-        session_data.label_ids, return_counts=True, axis=0
+        label_ids, return_counts=True, axis=0
     )
     num_label_ids = len(unique_label_ids)
 
     # need to call every time, so that the data is shuffled inside each class
-    label_data = split_session_data_by_label(session_data, unique_label_ids)
+    label_data = split_session_data_by_label_ids(
+        session_data, label_ids, unique_label_ids
+    )
 
     data_idx = [0] * num_label_ids
     num_data_cycles = [0] * num_label_ids
     skipped = [False] * num_label_ids
-    new_X = []
-    new_Y = []
-    new_label_ids = []
+
+    new_session_data = defaultdict(list)
+    num_examples = get_number_of_examples(session_data)
+
     while min(num_data_cycles) == 0:
         if shuffle:
             indices_of_labels = np.random.permutation(num_label_ids)
@@ -156,21 +252,13 @@ def balance_session_data(
                 int(counts_label_ids[index] / num_examples * batch_size) + 1
             )
 
-            new_X.append(
-                label_data[index].X[
-                    data_idx[index] : data_idx[index] + index_batch_size
-                ]
-            )
-            new_Y.append(
-                label_data[index].Y[
-                    data_idx[index] : data_idx[index] + index_batch_size
-                ]
-            )
-            new_label_ids.append(
-                label_data[index].label_ids[
-                    data_idx[index] : data_idx[index] + index_batch_size
-                ]
-            )
+            for k, values in label_data[index].items():
+                for i, v in enumerate(values):
+                    if len(new_session_data[k]) < i + 1:
+                        new_session_data[k].append([])
+                    new_session_data[k][i].append(
+                        v[data_idx[index] : data_idx[index] + index_batch_size]
+                    )
 
             data_idx[index] += index_batch_size
             if data_idx[index] >= counts_label_ids[index]:
@@ -180,91 +268,262 @@ def balance_session_data(
             if min(num_data_cycles) > 0:
                 break
 
-    return SessionData(
-        X=np.concatenate(new_X),
-        Y=np.concatenate(new_Y),
-        label_ids=np.concatenate(new_label_ids),
-    )
+    final_session_data = defaultdict(list)
+    for k, values in new_session_data.items():
+        for v in values:
+            final_session_data[k].append(np.concatenate(np.array(v)))
+
+    return final_session_data
+
+
+def get_number_of_examples(session_data: SessionDataType):
+    """Obtain number of examples in session data.
+
+    Raise a ValueError if number of examples differ for different data in session data.
+    """
+
+    example_lengths = [v.shape[0] for values in session_data.values() for v in values]
+
+    # check if number of examples is the same for all values
+    if not all(length == example_lengths[0] for length in example_lengths):
+        raise ValueError(
+            f"Number of examples differs for keys '{session_data.keys()}'. Number of "
+            f"examples should be the same for all data in session data."
+        )
+
+    return example_lengths[0]
 
 
 def gen_batch(
-    session_data: "SessionData",
+    session_data: SessionDataType,
     batch_size: int,
+    label_key: Text,
     batch_strategy: Text = "sequence",
     shuffle: bool = False,
-) -> Generator[Tuple["np.ndarray", "np.ndarray"], None, None]:
+) -> Generator[Tuple, None, None]:
     """Generate batches."""
 
     if shuffle:
         session_data = shuffle_session_data(session_data)
 
     if batch_strategy == "balanced":
-        session_data = balance_session_data(session_data, batch_size, shuffle)
+        session_data = balance_session_data(
+            session_data, batch_size, shuffle, label_key
+        )
 
-    num_batches = session_data.X.shape[0] // batch_size + int(
-        session_data.X.shape[0] % batch_size > 0
-    )
+    num_examples = get_number_of_examples(session_data)
+    num_batches = num_examples // batch_size + int(num_examples % batch_size > 0)
 
     for batch_num in range(num_batches):
-        batch_x = session_data.X[batch_num * batch_size : (batch_num + 1) * batch_size]
-        batch_y = session_data.Y[batch_num * batch_size : (batch_num + 1) * batch_size]
+        start = batch_num * batch_size
+        end = start + batch_size
 
-        yield batch_x, batch_y
+        yield prepare_batch(session_data, start, end)
+
+
+def prepare_batch(
+    session_data: SessionDataType,
+    start: Optional[int] = None,
+    end: Optional[int] = None,
+    tuple_sizes: Dict[Text, int] = None,
+) -> Tuple[Optional[np.ndarray]]:
+    """Slices session data into batch using given start and end value."""
+
+    batch_data = []
+
+    for key, values in session_data.items():
+        # add None for not present values during processing
+        if not values:
+            if tuple_sizes:
+                batch_data += [None] * tuple_sizes[key]
+            else:
+                batch_data.append(None)
+            continue
+
+        for v in values:
+            if start is not None and end is not None:
+                _data = v[start:end]
+            elif start is not None:
+                _data = v[start:]
+            elif end is not None:
+                _data = v[:end]
+            else:
+                _data = v[:]
+
+            if isinstance(_data[0], scipy.sparse.spmatrix):
+                batch_data.extend(scipy_matrix_to_values(_data))
+            else:
+                batch_data.append(pad_dense_data(_data))
+
+    # len of batch_data is equal to the number of keys in session data
+    return tuple(batch_data)
+
+
+def scipy_matrix_to_values(array_of_sparse: np.ndarray) -> List[np.ndarray]:
+    """Convert a scipy matrix into inidces, data, and shape."""
+
+    if not isinstance(array_of_sparse[0], scipy.sparse.coo_matrix):
+        array_of_sparse = [x.tocoo() for x in array_of_sparse]
+
+    max_seq_len = max([x.shape[0] for x in array_of_sparse])
+
+    indices = np.hstack(
+        [
+            np.vstack([i * np.ones_like(x.row), x.row, x.col])
+            for i, x in enumerate(array_of_sparse)
+        ]
+    ).T
+    data = np.hstack([x.data for x in array_of_sparse])
+
+    shape = np.array((len(array_of_sparse), max_seq_len, array_of_sparse[0].shape[-1]))
+
+    return [indices.astype(np.int64), data.astype(np.float32), shape.astype(np.int64)]
+
+
+def pad_dense_data(array_of_dense: np.ndarray) -> np.ndarray:
+    """Pad data of different lengths.
+
+    Sequential data is padded with zeros. Zeros are added to the end of data.
+    """
+
+    if array_of_dense[0].ndim < 2:
+        # data doesn't contain a sequence
+        return array_of_dense
+
+    data_size = len(array_of_dense)
+    max_seq_len = max([x.shape[0] for x in array_of_dense])
+
+    data_padded = np.zeros(
+        [data_size, max_seq_len, array_of_dense[0].shape[-1]],
+        dtype=array_of_dense[0].dtype,
+    )
+    for i in range(data_size):
+        data_padded[i, : array_of_dense[i].shape[0], :] = array_of_dense[i]
+
+    return data_padded.astype(np.float32)
+
+
+def batch_to_session_data(
+    batch: Union[Tuple[np.ndarray], Tuple[tf.Tensor]], session_data: SessionDataType
+) -> Tuple[Dict[Text, List[tf.Tensor]], Dict[Text, int]]:
+    """Convert input batch tensors into batch data format.
+
+    Batch contains any number of batch data. The order is equal to the
+    key-value pairs in session data. As sparse data were converted into indices, data,
+    shape before, this methods converts them into sparse tensors. Dense data is
+    kept.
+    """
+
+    batch_data = defaultdict(list)
+    # save the amount of placeholders attributed to session data keys
+    tuple_sizes = defaultdict(int)
+
+    idx = 0
+    for k, values in session_data.items():
+        tuple_sizes[k] = 0
+        for v in values:
+            if isinstance(v[0], scipy.sparse.spmatrix):
+                # explicitly substitute last dimension in shape with known static value
+                batch_data[k].append(
+                    tf.SparseTensor(
+                        batch[idx],
+                        batch[idx + 1],
+                        [batch[idx + 2][0], batch[idx + 2][1], v[0].shape[-1]],
+                    )
+                )
+                tuple_sizes[k] += 3
+                idx += 3
+            else:
+                batch_data[k].append(batch[idx])
+                tuple_sizes[k] += 1
+                idx += 1
+
+    return batch_data, tuple_sizes
 
 
-# noinspection PyPep8Naming
 def create_tf_dataset(
-    session_data: "SessionData",
+    session_data: SessionDataType,
     batch_size: Union["tf.Tensor", int],
+    label_key: Text,
     batch_strategy: Text = "sequence",
     shuffle: bool = False,
 ) -> "tf.data.Dataset":
     """Create tf dataset."""
 
-    # set batch and sequence length to None
-    if session_data.X[0].ndim == 1:
-        shape_X = (None, session_data.X[0].shape[-1])
-    else:
-        shape_X = (None, None, session_data.X[0].shape[-1])
-
-    if session_data.Y[0].ndim == 1:
-        shape_Y = (None, session_data.Y[0].shape[-1])
-    else:
-        shape_Y = (None, None, session_data.Y[0].shape[-1])
+    shapes, types = get_shapes_types(session_data)
 
     return tf.data.Dataset.from_generator(
         lambda batch_size_: gen_batch(
-            session_data, batch_size_, batch_strategy, shuffle
+            session_data, batch_size_, label_key, batch_strategy, shuffle
         ),
-        output_types=(tf.float32, tf.float32),
-        output_shapes=(shape_X, shape_Y),
+        output_types=types,
+        output_shapes=shapes,
         args=([batch_size]),
     )
 
 
+def get_shapes_types(session_data: SessionDataType) -> Tuple:
+    """Extract shapes and types from session data."""
+
+    types = []
+    shapes = []
+
+    def append_shape(v: np.ndarray):
+        if isinstance(v[0], scipy.sparse.spmatrix):
+            # scipy matrix is converted into indices, data, shape
+            shapes.append((None, v[0].ndim + 1))
+            shapes.append((None,))
+            shapes.append((v[0].ndim + 1))
+        elif v[0].ndim == 0:
+            shapes.append((None,))
+        elif v[0].ndim == 1:
+            shapes.append((None, v[0].shape[-1]))
+        else:
+            shapes.append((None, None, v[0].shape[-1]))
+
+    def append_type(v: np.ndarray):
+        if isinstance(v[0], scipy.sparse.spmatrix):
+            # scipy matrix is converted into indices, data, shape
+            types.append(tf.int64)
+            types.append(tf.float32)
+            types.append(tf.int64)
+        else:
+            types.append(tf.float32)
+
+    for values in session_data.values():
+        for v in values:
+            append_shape(v)
+            append_type(v)
+
+    return tuple(shapes), tuple(types)
+
+
 def create_iterator_init_datasets(
-    session_data: "SessionData",
-    eval_session_data: "SessionData",
+    session_data: SessionDataType,
+    eval_session_data: SessionDataType,
     batch_size: Union["tf.Tensor", int],
     batch_strategy: Text,
+    label_key: Text,
 ) -> Tuple["tf.data.Iterator", "tf.Operation", "tf.Operation"]:
     """Create iterator and init datasets."""
 
     train_dataset = create_tf_dataset(
-        session_data, batch_size, batch_strategy=batch_strategy, shuffle=True
+        session_data,
+        batch_size,
+        label_key=label_key,
+        batch_strategy=batch_strategy,
+        shuffle=True,
     )
 
     iterator = tf.data.Iterator.from_structure(
-        train_dataset.output_types,
-        train_dataset.output_shapes,
-        output_classes=train_dataset.output_classes,
+        train_dataset.output_types, train_dataset.output_shapes
     )
 
     train_init_op = iterator.make_initializer(train_dataset)
 
     if eval_session_data is not None:
         eval_init_op = iterator.make_initializer(
-            create_tf_dataset(eval_session_data, batch_size)
+            create_tf_dataset(eval_session_data, batch_size, label_key=label_key)
         )
     else:
         eval_init_op = None
@@ -272,6 +531,50 @@ def create_iterator_init_datasets(
     return iterator, train_init_op, eval_init_op
 
 
+# noinspection PyPep8Naming
+def tf_dense_layer_for_sparse(
+    inputs: tf.SparseTensor,
+    units: int,
+    name: Text,
+    C2: float,
+    activation: Optional[Callable] = tf.nn.relu,
+    use_bias: bool = True,
+) -> tf.Tensor:
+    """Dense layer for sparse input tensor"""
+
+    if not isinstance(inputs, tf.SparseTensor):
+        raise ValueError("Input tensor should be sparse.")
+
+    with tf.variable_scope("dense_layer_for_sparse_" + name, reuse=tf.AUTO_REUSE):
+        kernel_regularizer = tf.contrib.layers.l2_regularizer(C2)
+        kernel = tf.get_variable(
+            "kernel",
+            shape=[inputs.shape[-1], units],
+            dtype=inputs.dtype,
+            regularizer=kernel_regularizer,
+        )
+        bias = tf.get_variable("bias", shape=[units], dtype=inputs.dtype)
+
+        # outputs will be 2D
+        outputs = tf.sparse.matmul(
+            tf.sparse.reshape(inputs, [-1, int(inputs.shape[-1])]), kernel
+        )
+
+        if len(inputs.shape) == 3:
+            # reshape back
+            outputs = tf.reshape(
+                outputs, (tf.shape(inputs)[0], tf.shape(inputs)[1], -1)
+            )
+
+        if use_bias:
+            outputs = tf.nn.bias_add(outputs, bias)
+
+    if activation is None:
+        return outputs
+
+    return activation(outputs)
+
+
 # noinspection PyPep8Naming
 def create_tf_fnn(
     x_in: "tf.Tensor",
@@ -312,9 +615,8 @@ def tf_normalize_if_cosine(x: "tf.Tensor", similarity_type: Text) -> "tf.Tensor"
         return x
     else:
         raise ValueError(
-            "Wrong similarity type '{}', "
-            "should be 'cosine' or 'inner'"
-            "".format(similarity_type)
+            f"Wrong similarity type '{similarity_type}', "
+            f"should be 'cosine' or 'inner'"
         )
 
 
@@ -453,19 +755,21 @@ def _tf_sample_neg(
     return tf.batch_gather(tiled_all_bs, neg_ids)
 
 
-def _tf_calc_iou_mask(
+def _tf_get_bad_mask(
     pos_b: "tf.Tensor", all_bs: "tf.Tensor", neg_ids: "tf.Tensor"
 ) -> "tf.Tensor":
-    """Calculate IOU mask for given indices"""
+    """Calculate bad mask for given indices.
+
+    Checks that input features are different for positive negative samples.
+    """
 
     pos_b_in_flat = tf.expand_dims(pos_b, -2)
     neg_b_in_flat = _tf_sample_neg(tf.shape(pos_b)[0], all_bs, neg_ids)
 
-    intersection_b_in_flat = tf.minimum(neg_b_in_flat, pos_b_in_flat)
-    union_b_in_flat = tf.maximum(neg_b_in_flat, pos_b_in_flat)
-
-    iou = tf.reduce_sum(intersection_b_in_flat, -1) / tf.reduce_sum(union_b_in_flat, -1)
-    return 1.0 - tf.nn.relu(tf.sign(1.0 - iou))
+    return tf.cast(
+        tf.reduce_all(tf.equal(neg_b_in_flat, pos_b_in_flat), axis=-1),
+        pos_b_in_flat.dtype,
+    )
 
 
 def _tf_get_negs(
@@ -493,7 +797,7 @@ def _tf_get_negs(
     )
     neg_ids = shuffled_indices[:, :num_neg]
 
-    bad_negs = _tf_calc_iou_mask(raw_flat, all_raw, neg_ids)
+    bad_negs = _tf_get_bad_mask(raw_flat, all_raw, neg_ids)
     if len(raw_pos.shape) == 3:
         bad_negs = tf.reshape(bad_negs, (batch_size, seq_length, -1))
 
@@ -726,9 +1030,7 @@ def choose_loss(
         )
     else:
         raise ValueError(
-            "Wrong loss type '{}', "
-            "should be 'margin' or 'softmax'"
-            "".format(loss_type)
+            f"Wrong loss type '{loss_type}', " f"should be 'margin' or 'softmax'"
         )
 
 
@@ -867,8 +1169,8 @@ def train_tf_dataset(
 
     if evaluate_on_num_examples:
         logger.info(
-            "Validation accuracy is calculated every {} epochs"
-            "".format(evaluate_every_num_epochs)
+            f"Validation accuracy is calculated every {evaluate_every_num_epochs} "
+            f"epochs."
         )
     pbar = tqdm(range(epochs), desc="Epochs", disable=is_logging_disabled())
 
@@ -921,14 +1223,12 @@ def train_tf_dataset(
         pbar.set_postfix(postfix_dict)
 
     final_message = (
-        "Finished training embedding policy, "
-        "train loss={:.3f}, train accuracy={:.3f}"
-        "".format(train_loss, train_acc)
+        f"Finished training embedding policy, "
+        f"train loss={train_loss:.3f}, train accuracy={train_acc:.3f}"
     )
     if eval_init_op is not None:
         final_message += (
-            ", validation loss={:.3f}, validation accuracy={:.3f}"
-            "".format(val_loss, val_acc)
+            f", validation loss={val_loss:.3f}, validation accuracy={val_acc:.3f}"
         )
     logger.info(final_message)
 
@@ -947,16 +1247,31 @@ def extract_attention(attention_weights) -> Optional["tf.Tensor"]:
         return tf.concat(attention, 0)
 
 
-def persist_tensor(name: Text, tensor: "tf.Tensor", graph: "tf.Graph") -> None:
+def persist_tensor(
+    name: Text,
+    tensor: Union["tf.Tensor", Tuple["tf.Tensor"], List["tf.Tensor"]],
+    graph: "tf.Graph",
+) -> None:
     """Add tensor to collection if it is not None"""
 
     if tensor is not None:
         graph.clear_collection(name)
-        graph.add_to_collection(name, tensor)
+        if isinstance(tensor, tuple) or isinstance(tensor, list):
+            for t in tensor:
+                graph.add_to_collection(name, t)
+        else:
+            graph.add_to_collection(name, tensor)
 
 
-def load_tensor(name: Text) -> Optional["tf.Tensor"]:
+def load_tensor(name: Text) -> Optional[Union["tf.Tensor", List["tf.Tensor"]]]:
     """Load tensor or set it to None"""
 
     tensor_list = tf.get_collection(name)
-    return tensor_list[0] if tensor_list else None
+
+    if not tensor_list:
+        return None
+
+    if len(tensor_list) == 1:
+        return tensor_list[0]
+
+    return tensor_list
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 1a7a3607453c..dc221349b647 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -3,21 +3,26 @@
 from _pytest.pytester import RunResult
 
 
-@pytest.mark.repeat(3)
 def test_cli_start(run: Callable[..., RunResult]):
     """
-    Startup of cli should not take longer than n seconds
+    Measures an average startup time and checks that it
+    does not deviate more than x seconds from 5.
     """
     import time
 
-    start = time.time()
-    run("--help")
-    end = time.time()
+    durations = []
 
-    duration = end - start
+    for i in range(5):
+        start = time.time()
+        run("--help")
+        end = time.time()
+
+        durations.append(end - start)
+
+    avg_duration = sum(durations) / len(durations)
 
     # When run in parallel, it takes a little longer
-    assert duration <= 5
+    assert avg_duration - 5 <= 2
 
 
 def test_data_convert_help(run: Callable[..., RunResult]):
diff --git a/tests/core/test_policies.py b/tests/core/test_policies.py
index 1e6ee1acb946..75b5bf9daea2 100644
--- a/tests/core/test_policies.py
+++ b/tests/core/test_policies.py
@@ -1,10 +1,8 @@
-import asyncio
 from unittest.mock import patch
 
 import numpy as np
 import pytest
 
-import rasa.utils.io
 from rasa.utils import train_utils
 from rasa.core import training
 from rasa.core.actions.action import (
@@ -348,26 +346,29 @@ async def test_gen_batch(self, trained_policy, default_domain):
             training_data.X, training_data.y
         )
         batch_size = 2
-        batch_x, batch_y = next(
-            train_utils.gen_batch(session_data=session_data, batch_size=batch_size)
+        batch_x, batch_y, _ = next(
+            train_utils.gen_batch(
+                session_data=session_data, batch_size=batch_size, label_key="action_ids"
+            )
         )
         assert batch_x.shape[0] == batch_size and batch_y.shape[0] == batch_size
         assert (
-            batch_x[0].shape == session_data.X[0].shape
-            and batch_y[0].shape == session_data.Y[0].shape
+            batch_x[0].shape == session_data["dialogue_features"][0][0].shape
+            and batch_y[0].shape == session_data["bot_features"][0][0].shape
         )
-        batch_x, batch_y = next(
+        batch_x, batch_y, _ = next(
             train_utils.gen_batch(
                 session_data=session_data,
                 batch_size=batch_size,
+                label_key="action_ids",
                 batch_strategy="balanced",
                 shuffle=True,
             )
         )
         assert batch_x.shape[0] == batch_size and batch_y.shape[0] == batch_size
         assert (
-            batch_x[0].shape == session_data.X[0].shape
-            and batch_y[0].shape == session_data.Y[0].shape
+            batch_x[0].shape == session_data["dialogue_features"][0][0].shape
+            and batch_y[0].shape == session_data["bot_features"][0][0].shape
         )
 
 
diff --git a/tests/nlu/base/test_components.py b/tests/nlu/base/test_components.py
index dff1688b6021..18c23da518f2 100644
--- a/tests/nlu/base/test_components.py
+++ b/tests/nlu/base/test_components.py
@@ -55,9 +55,9 @@ def test_find_unavailable_packages():
 
 
 def test_builder_create_by_module_path(component_builder, default_config):
-    from rasa.nlu.featurizers.regex_featurizer import RegexFeaturizer
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
 
-    path = "rasa.nlu.featurizers.regex_featurizer.RegexFeaturizer"
+    path = "rasa.nlu.featurizers.sparse_featurizer.regex_featurizer.RegexFeaturizer"
     component_config = {"name": path}
     component = component_builder.create_component(component_config, default_config)
     assert type(component) == RegexFeaturizer
diff --git a/tests/nlu/base/test_evaluation.py b/tests/nlu/base/test_evaluation.py
index 51fc3dcc9f61..60971636a9f2 100644
--- a/tests/nlu/base/test_evaluation.py
+++ b/tests/nlu/base/test_evaluation.py
@@ -41,7 +41,7 @@
 from rasa.nlu.test import determine_intersection
 from rasa.nlu.test import determine_token_labels
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.tokenizers import Token
+from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu import utils
 import json
 import os
@@ -259,7 +259,7 @@ def test_run_evaluation(unpacked_trained_moodbot_path):
     data = DEFAULT_DATA_PATH
 
     result = run_evaluation(
-        data, os.path.join(unpacked_trained_moodbot_path, "nlu"), errors=None
+        data, os.path.join(unpacked_trained_moodbot_path, "nlu"), errors=False
     )
     assert result.get("intent_evaluation")
     assert result.get("entity_evaluation").get("CRFEntityExtractor")
diff --git a/tests/nlu/base/test_extractors.py b/tests/nlu/base/test_extractors.py
deleted file mode 100644
index 5b2e95606478..000000000000
--- a/tests/nlu/base/test_extractors.py
+++ /dev/null
@@ -1,531 +0,0 @@
-import responses
-
-from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.training_data import TrainingData, Message
-from tests.nlu import utilities
-
-
-def test_crf_extractor(spacy_nlp, ner_crf_pos_feature_config):
-    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
-
-    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
-    examples = [
-        Message(
-            "anywhere in the west",
-            {
-                "intent": "restaurant_search",
-                "entities": [
-                    {"start": 16, "end": 20, "value": "west", "entity": "location"}
-                ],
-                "spacy_doc": spacy_nlp("anywhere in the west"),
-            },
-        ),
-        Message(
-            "central indian restaurant",
-            {
-                "intent": "restaurant_search",
-                "entities": [
-                    {
-                        "start": 0,
-                        "end": 7,
-                        "value": "central",
-                        "entity": "location",
-                        "extractor": "random_extractor",
-                    },
-                    {
-                        "start": 8,
-                        "end": 14,
-                        "value": "indian",
-                        "entity": "cuisine",
-                        "extractor": "CRFEntityExtractor",
-                    },
-                ],
-                "spacy_doc": spacy_nlp("central indian restaurant"),
-            },
-        ),
-    ]
-
-    # uses BILOU and the default features
-    ext.train(TrainingData(training_examples=examples), RasaNLUModelConfig())
-    sentence = "anywhere in the west"
-    doc = {"spacy_doc": spacy_nlp(sentence)}
-    crf_format = ext._from_text_to_crf(Message(sentence, doc))
-    assert [word[0] for word in crf_format] == ["anywhere", "in", "the", "west"]
-    feats = ext._sentence_to_features(crf_format)
-    assert "BOS" in feats[0]
-    assert "EOS" in feats[-1]
-    assert feats[1]["0:low"] == "in"
-    sentence = "anywhere in the west"
-    ext.extract_entities(Message(sentence, {"spacy_doc": spacy_nlp(sentence)}))
-    filtered = ext.filter_trainable_entities(examples)
-    assert filtered[0].get("entities") == [
-        {"start": 16, "end": 20, "value": "west", "entity": "location"}
-    ], "Entity without extractor remains"
-    assert filtered[1].get("entities") == [
-        {
-            "start": 8,
-            "end": 14,
-            "value": "indian",
-            "entity": "cuisine",
-            "extractor": "CRFEntityExtractor",
-        }
-    ], "Only CRFEntityExtractor entity annotation remains"
-    assert examples[1].get("entities")[0] == {
-        "start": 0,
-        "end": 7,
-        "value": "central",
-        "entity": "location",
-        "extractor": "random_extractor",
-    }, "Original examples are not mutated"
-
-
-def test_crf_json_from_BILOU(spacy_nlp, ner_crf_pos_feature_config):
-    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
-
-    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
-    sentence = "I need a home cleaning close-by"
-    doc = {"spacy_doc": spacy_nlp(sentence)}
-    r = ext._from_crf_to_json(
-        Message(sentence, doc),
-        [
-            {"O": 1.0},
-            {"O": 1.0},
-            {"O": 1.0},
-            {"B-what": 1.0},
-            {"L-what": 1.0},
-            {"B-where": 1.0},
-            {"I-where": 1.0},
-            {"L-where": 1.0},
-        ],
-    )
-    assert len(r) == 2, "There should be two entities"
-
-    assert r[0]["confidence"]  # confidence should exist
-    del r[0]["confidence"]
-    assert r[0] == {"start": 9, "end": 22, "value": "home cleaning", "entity": "what"}
-
-    assert r[1]["confidence"]  # confidence should exist
-    del r[1]["confidence"]
-    assert r[1] == {"start": 23, "end": 31, "value": "close-by", "entity": "where"}
-
-
-def test_crf_json_from_non_BILOU(spacy_nlp, ner_crf_pos_feature_config):
-    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
-
-    ner_crf_pos_feature_config.update({"BILOU_flag": False})
-    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
-    sentence = "I need a home cleaning close-by"
-    doc = {"spacy_doc": spacy_nlp(sentence)}
-    rs = ext._from_crf_to_json(
-        Message(sentence, doc),
-        [
-            {"O": 1.0},
-            {"O": 1.0},
-            {"O": 1.0},
-            {"what": 1.0},
-            {"what": 1.0},
-            {"where": 1.0},
-            {"where": 1.0},
-            {"where": 1.0},
-        ],
-    )
-
-    # non BILOU will split multi-word entities - hence 5
-    assert len(rs) == 5, "There should be five entities"
-
-    for r in rs:
-        assert r["confidence"]  # confidence should exist
-        del r["confidence"]
-
-    assert rs[0] == {"start": 9, "end": 13, "value": "home", "entity": "what"}
-    assert rs[1] == {"start": 14, "end": 22, "value": "cleaning", "entity": "what"}
-    assert rs[2] == {"start": 23, "end": 28, "value": "close", "entity": "where"}
-    assert rs[3] == {"start": 28, "end": 29, "value": "-", "entity": "where"}
-    assert rs[4] == {"start": 29, "end": 31, "value": "by", "entity": "where"}
-
-
-def test_crf_create_entity_dict(spacy_nlp):
-    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-
-    crf_extractor = CRFEntityExtractor()
-    spacy_tokenizer = SpacyTokenizer()
-    white_space_tokenizer = WhitespaceTokenizer()
-
-    examples = [
-        {
-            "message": Message(
-                "where is St. Michael's Hospital?",
-                {
-                    "intent": "search_location",
-                    "entities": [
-                        {
-                            "start": 9,
-                            "end": 31,
-                            "value": "St. Michael's Hospital",
-                            "entity": "hospital",
-                            "SpacyTokenizer": {
-                                "entity_start_token_idx": 2,
-                                "entity_end_token_idx": 5,
-                            },
-                            "WhitespaceTokenizer": {
-                                "entity_start_token_idx": 2,
-                                "entity_end_token_idx": 5,
-                            },
-                        }
-                    ],
-                },
-            )
-        },
-        {
-            "message": Message(
-                "where is Children's Hospital?",
-                {
-                    "intent": "search_location",
-                    "entities": [
-                        {
-                            "start": 9,
-                            "end": 28,
-                            "value": "Children's Hospital",
-                            "entity": "hospital",
-                            "SpacyTokenizer": {
-                                "entity_start_token_idx": 2,
-                                "entity_end_token_idx": 4,
-                            },
-                            "WhitespaceTokenizer": {
-                                "entity_start_token_idx": 2,
-                                "entity_end_token_idx": 4,
-                            },
-                        }
-                    ],
-                },
-            )
-        },
-    ]
-    for ex in examples:
-        # spacy tokenizers receives a Doc as input and whitespace tokenizer receives a text
-        spacy_tokens = spacy_tokenizer.tokenize(spacy_nlp(ex["message"].text))
-        white_space_tokens = white_space_tokenizer.tokenize(ex["message"].text)
-        for tokenizer, tokens in [
-            ("SpacyTokenizer", spacy_tokens),
-            ("WhitespaceTokenizer", white_space_tokens),
-        ]:
-            for entity in ex["message"].get("entities"):
-                parsed_entities = crf_extractor._create_entity_dict(
-                    ex["message"],
-                    tokens,
-                    entity[tokenizer]["entity_start_token_idx"],
-                    entity[tokenizer]["entity_end_token_idx"],
-                    entity["entity"],
-                    0.8,
-                )
-                assert parsed_entities == {
-                    "start": entity["start"],
-                    "end": entity["end"],
-                    "value": entity["value"],
-                    "entity": entity["entity"],
-                    "confidence": 0.8,
-                }
-
-
-def test_duckling_entity_extractor(component_builder):
-    with responses.RequestsMock() as rsps:
-        rsps.add(
-            responses.POST,
-            "http://localhost:8000/parse",
-            json=[
-                {
-                    "body": "Today",
-                    "start": 0,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2018-11-13T00:00:00.000-08:00",
-                                "grain": "day",
-                                "type": "value",
-                            }
-                        ],
-                        "value": "2018-11-13T00:00:00.000-08:00",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 5,
-                    "dim": "time",
-                    "latent": False,
-                },
-                {
-                    "body": "the 5th",
-                    "start": 9,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2018-12-05T00:00:00.000-08:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                            {
-                                "value": "2019-01-05T00:00:00.000-08:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                            {
-                                "value": "2019-02-05T00:00:00.000-08:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                        ],
-                        "value": "2018-12-05T00:00:00.000-08:00",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 16,
-                    "dim": "time",
-                    "latent": False,
-                },
-                {
-                    "body": "5th of May",
-                    "start": 13,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2019-05-05T00:00:00.000-07:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                            {
-                                "value": "2020-05-05T00:00:00.000-07:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                            {
-                                "value": "2021-05-05T00:00:00.000-07:00",
-                                "grain": "day",
-                                "type": "value",
-                            },
-                        ],
-                        "value": "2019-05-05T00:00:00.000-07:00",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 23,
-                    "dim": "time",
-                    "latent": False,
-                },
-                {
-                    "body": "tomorrow",
-                    "start": 37,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2018-11-14T00:00:00.000-08:00",
-                                "grain": "day",
-                                "type": "value",
-                            }
-                        ],
-                        "value": "2018-11-14T00:00:00.000-08:00",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 45,
-                    "dim": "time",
-                    "latent": False,
-                },
-            ],
-        )
-
-        _config = RasaNLUModelConfig({"pipeline": [{"name": "DucklingHTTPExtractor"}]})
-        _config.set_component_attr(
-            0, dimensions=["time"], timezone="UTC", url="http://localhost:8000"
-        )
-        duckling = component_builder.create_component(_config.for_component(0), _config)
-        message = Message("Today is the 5th of May. Let us meet tomorrow.")
-        duckling.process(message)
-        entities = message.get("entities")
-        assert len(entities) == 4
-
-    # Test duckling with a defined date
-
-    with responses.RequestsMock() as rsps:
-        rsps.add(
-            responses.POST,
-            "http://localhost:8000/parse",
-            json=[
-                {
-                    "body": "tomorrow",
-                    "start": 12,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2013-10-13T00:00:00.000Z",
-                                "grain": "day",
-                                "type": "value",
-                            }
-                        ],
-                        "value": "2013-10-13T00:00:00.000Z",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 20,
-                    "dim": "time",
-                    "latent": False,
-                }
-            ],
-        )
-
-        # 1381536182 == 2013/10/12 02:03:02
-        message = Message("Let us meet tomorrow.", time="1381536182")
-        duckling.process(message)
-        entities = message.get("entities")
-        assert len(entities) == 1
-        assert entities[0]["text"] == "tomorrow"
-        assert entities[0]["value"] == "2013-10-13T00:00:00.000Z"
-
-        # Test dimension filtering includes only specified dimensions
-        _config = RasaNLUModelConfig({"pipeline": [{"name": "DucklingHTTPExtractor"}]})
-        _config.set_component_attr(
-            0, dimensions=["number"], url="http://localhost:8000"
-        )
-        duckling_number = component_builder.create_component(
-            _config.for_component(0), _config
-        )
-
-    with responses.RequestsMock() as rsps:
-        rsps.add(
-            responses.POST,
-            "http://localhost:8000/parse",
-            json=[
-                {
-                    "body": "Yesterday",
-                    "start": 0,
-                    "value": {
-                        "values": [
-                            {
-                                "value": "2019-02-28T00:00:00.000+01:00",
-                                "grain": "day",
-                                "type": "value",
-                            }
-                        ],
-                        "value": "2019-02-28T00:00:00.000+01:00",
-                        "grain": "day",
-                        "type": "value",
-                    },
-                    "end": 9,
-                    "dim": "time",
-                },
-                {
-                    "body": "5",
-                    "start": 21,
-                    "value": {"value": 5, "type": "value"},
-                    "end": 22,
-                    "dim": "number",
-                },
-            ],
-        )
-
-        message = Message("Yesterday there were 5 people in a room")
-        duckling_number.process(message)
-        entities = message.get("entities")
-
-        assert len(entities) == 1
-        assert entities[0]["text"] == "5"
-        assert entities[0]["value"] == 5
-
-
-def test_duckling_entity_extractor_and_synonyms(component_builder):
-    _config = RasaNLUModelConfig(
-        {
-            "pipeline": [
-                {"name": "DucklingHTTPExtractor"},
-                {"name": "EntitySynonymMapper"},
-            ]
-        }
-    )
-    _config.set_component_attr(0, dimensions=["number"])
-    duckling = component_builder.create_component(_config.for_component(0), _config)
-    synonyms = component_builder.create_component(_config.for_component(1), _config)
-    message = Message("He was 6 feet away")
-    duckling.process(message)
-    # checks that the synonym processor
-    # can handle entities that have int values
-    synonyms.process(message)
-    assert message is not None
-
-
-def test_unintentional_synonyms_capitalized(component_builder):
-    _config = utilities.base_test_conf("pretrained_embeddings_spacy")
-    ner_syn = component_builder.create_component(_config.for_component(5), _config)
-    examples = [
-        Message(
-            "Any Mexican restaurant will do",
-            {
-                "intent": "restaurant_search",
-                "entities": [
-                    {"start": 4, "end": 11, "value": "Mexican", "entity": "cuisine"}
-                ],
-            },
-        ),
-        Message(
-            "I want Tacos!",
-            {
-                "intent": "restaurant_search",
-                "entities": [
-                    {"start": 7, "end": 12, "value": "Mexican", "entity": "cuisine"}
-                ],
-            },
-        ),
-    ]
-    ner_syn.train(TrainingData(training_examples=examples), _config)
-    assert ner_syn.synonyms.get("mexican") is None
-    assert ner_syn.synonyms.get("tacos") == "Mexican"
-
-
-def test_spacy_ner_extractor(component_builder, spacy_nlp):
-    _config = RasaNLUModelConfig({"pipeline": [{"name": "SpacyEntityExtractor"}]})
-    ext = component_builder.create_component(_config.for_component(0), _config)
-    example = Message(
-        "anywhere in the U.K.",
-        {
-            "intent": "restaurant_search",
-            "entities": [],
-            "spacy_doc": spacy_nlp("anywhere in the west"),
-        },
-    )
-
-    ext.process(example, spacy_nlp=spacy_nlp)
-
-    assert len(example.get("entities", [])) == 1
-    assert example.get("entities")[0] == {
-        "start": 16,
-        "extractor": "SpacyEntityExtractor",
-        "end": 20,
-        "value": "U.K.",
-        "entity": "GPE",
-        "confidence": None,
-    }
-
-    # Test dimension filtering includes only specified dimensions
-
-    example = Message(
-        "anywhere in the West with Sebastian Thrun",
-        {
-            "intent": "example_intent",
-            "entities": [],
-            "spacy_doc": spacy_nlp("anywhere in the West with Sebastian Thrun"),
-        },
-    )
-    _config = RasaNLUModelConfig({"pipeline": [{"name": "SpacyEntityExtractor"}]})
-
-    _config.set_component_attr(0, dimensions=["PERSON"])
-    ext = component_builder.create_component(_config.for_component(0), _config)
-    ext.process(example, spacy_nlp=spacy_nlp)
-
-    assert len(example.get("entities", [])) == 1
-    assert example.get("entities")[0] == {
-        "start": 26,
-        "extractor": "SpacyEntityExtractor",
-        "end": 41,
-        "value": "Sebastian Thrun",
-        "entity": "PERSON",
-        "confidence": None,
-    }
diff --git a/tests/nlu/base/test_featurizers.py b/tests/nlu/base/test_featurizers.py
deleted file mode 100644
index eb2818d9a30c..000000000000
--- a/tests/nlu/base/test_featurizers.py
+++ /dev/null
@@ -1,605 +0,0 @@
-import numpy as np
-import pytest
-
-from rasa.nlu import training_data
-from rasa.nlu.tokenizers import Token
-from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
-from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-from rasa.nlu.training_data import Message
-from rasa.nlu.training_data import TrainingData
-from rasa.nlu.config import RasaNLUModelConfig
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        (
-            "hey how are you today",
-            [-0.19649599, 0.32493639, -0.37408298, -0.10622784, 0.062756],
-        )
-    ],
-)
-def test_spacy_featurizer(sentence, expected, spacy_nlp):
-    from rasa.nlu.featurizers import spacy_featurizer
-
-    doc = spacy_nlp(sentence)
-    vecs = spacy_featurizer.features_for_doc(doc)
-    assert np.allclose(doc.vector[:5], expected, atol=1e-5)
-    assert np.allclose(vecs, doc.vector, atol=1e-5)
-
-
-def test_spacy_training_sample_alignment(spacy_nlp_component):
-    from spacy.tokens import Doc
-
-    m1 = Message.build(text="I have a feeling", intent="feeling")
-    m2 = Message.build(text="", intent="feeling")
-    m3 = Message.build(text="I am the last message", intent="feeling")
-    td = TrainingData(training_examples=[m1, m2, m3])
-
-    attribute_docs = spacy_nlp_component.docs_for_training_data(td)
-
-    assert isinstance(attribute_docs["text"][0], Doc)
-    assert isinstance(attribute_docs["text"][1], Doc)
-    assert isinstance(attribute_docs["text"][2], Doc)
-
-    assert [t.text for t in attribute_docs["text"][0]] == ["i", "have", "a", "feeling"]
-    assert [t.text for t in attribute_docs["text"][1]] == []
-    assert [t.text for t in attribute_docs["text"][2]] == [
-        "i",
-        "am",
-        "the",
-        "last",
-        "message",
-    ]
-
-
-def test_spacy_intent_featurizer(spacy_nlp_component):
-    from rasa.nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-    spacy_nlp_component.train(td, config=None)
-    spacy_featurizer = SpacyFeaturizer()
-    spacy_featurizer.train(td, config=None)
-
-    intent_features_exist = np.array(
-        [
-            True if example.get("intent_features") is not None else False
-            for example in td.intent_examples
-        ]
-    )
-
-    # no intent features should have been set
-    assert not any(intent_features_exist)
-
-
-def test_convert_intent_featurizer():
-    from rasa.nlu.featurizers.convert_featurizer import ConveRTFeaturizer
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-
-    convert_featurizer = ConveRTFeaturizer()
-    convert_featurizer.train(td, config=None)
-
-    intent_features_exist = np.array(
-        [
-            True if example.get("intent_features") is not None else False
-            for example in td.intent_examples
-        ]
-    )
-
-    # no intent features should have been set
-    assert not any(intent_features_exist)
-
-
-def test_convert_featurizer_output_shape():
-    from rasa.nlu.featurizers.convert_featurizer import ConveRTFeaturizer
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-
-    convert_featurizer = ConveRTFeaturizer()
-    convert_featurizer.train(td, config=None)
-
-    text_features_dim = np.array(
-        [
-            example.get("text_features").shape[0]
-            for example in td.intent_examples
-            if example.get("text_features") is not None
-        ]
-    )
-
-    response_features_dim = np.array(
-        [
-            example.get("response_features").shape[0]
-            for example in td.intent_examples
-            if example.get("response_features") is not None
-        ]
-    )
-
-    assert np.all(text_features_dim == 1024)
-    assert np.all(response_features_dim == 1024)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [("hey how are you today", [-0.28451, 0.31007, -0.57039, -0.073056, -0.17322])],
-)
-def test_spacy_ner_featurizer(sentence, expected, spacy_nlp):
-    from rasa.nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-
-    doc = spacy_nlp(sentence)
-    token_vectors = [t.vector for t in doc]
-    spacy_config = {"ner_feature_vectors": True}
-    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
-    greet = {"intent": "greet", "text_features": [0.5]}
-    message = Message(sentence, greet)
-    message.set("spacy_doc", doc)
-    ftr._set_spacy_features(message)
-    ftr._set_spacy_ner_features(message)
-    vecs = message.get("ner_features")[0][:5]
-    assert np.allclose(token_vectors[0][:5], vecs, atol=1e-4)
-    assert np.allclose(vecs, expected, atol=1e-4)
-
-
-def test_spacy_ner_featurizer_config(spacy_nlp):
-    from rasa.nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-
-    sentence = "hi there friend"
-    doc = spacy_nlp(sentence)
-    spacy_config = {"ner_feature_vectors": False}
-    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
-    greet = {"intent": "greet", "text_features": [0.5]}
-    message = Message(sentence, greet)
-    message.set("spacy_doc", doc)
-    ftr._set_spacy_features(message)
-    ftr._set_spacy_ner_features(message)
-    vecs = np.array(message.get("ner_features"))
-    assert vecs.shape[0] == len(doc)
-    assert vecs.shape[1] == 0
-
-
-def test_mitie_featurizer(mitie_feature_extractor, default_config):
-    from rasa.nlu.featurizers.mitie_featurizer import MitieFeaturizer
-
-    mitie_component_config = {"name": "MitieFeaturizer"}
-    ftr = MitieFeaturizer.create(mitie_component_config, RasaNLUModelConfig())
-    sentence = "Hey how are you today"
-    tokens = MitieTokenizer().tokenize(sentence)
-    vecs = ftr.features_for_tokens(tokens, mitie_feature_extractor)
-    expected = np.array([0.0, -4.4551446, 0.26073121, -1.46632245, -1.84205751])
-    assert np.allclose(vecs[:5], expected, atol=1e-5)
-
-
-def test_ngram_featurizer(spacy_nlp):
-    from rasa.nlu.featurizers.ngram_featurizer import NGramFeaturizer
-
-    ftr = NGramFeaturizer({"max_number_of_ngrams": 10})
-
-    # ensures that during random sampling of the ngram CV we don't end up
-    # with a one-class-split
-    repetition_factor = 5
-
-    greet = {"intent": "greet", "text_features": [0.5]}
-    goodbye = {"intent": "goodbye", "text_features": [0.5]}
-    labeled_sentences = [
-        Message("heyheyheyhey", greet),
-        Message("howdyheyhowdy", greet),
-        Message("heyhey howdyheyhowdy", greet),
-        Message("howdyheyhowdy heyhey", greet),
-        Message("astalavistasista", goodbye),
-        Message("astalavistasista sistala", goodbye),
-        Message("sistala astalavistasista", goodbye),
-    ] * repetition_factor
-
-    for m in labeled_sentences:
-        m.set("spacy_doc", spacy_nlp(m.text))
-
-    ftr.min_intent_examples_for_ngram_classification = 2
-    ftr.train_on_sentences(labeled_sentences)
-    assert len(ftr.all_ngrams) > 0
-    assert ftr.best_num_ngrams > 0
-
-
-@pytest.mark.parametrize(
-    "sentence, expected, labeled_tokens",
-    [
-        ("hey how are you today", [0.0, 1.0, 0.0], [0]),
-        ("hey 456 how are you", [1.0, 1.0, 0.0], [1, 0]),
-        ("blah balh random eh", [0.0, 0.0, 0.0], []),
-        ("a 1 digit number", [1.0, 0.0, 1.0], [1, 1]),
-    ],
-)
-def test_regex_featurizer(sentence, expected, labeled_tokens, spacy_nlp):
-    from rasa.nlu.featurizers.regex_featurizer import RegexFeaturizer
-
-    patterns = [
-        {"pattern": "[0-9]+", "name": "number", "usage": "intent"},
-        {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
-        {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
-    ]
-    ftr = RegexFeaturizer(known_patterns=patterns)
-
-    # adds tokens to the message
-    tokenizer = SpacyTokenizer()
-    message = Message(sentence)
-    message.set("spacy_doc", spacy_nlp(sentence))
-    tokenizer.process(message)
-
-    result = ftr.features_for_patterns(message)
-    assert np.allclose(result, expected, atol=1e-10)
-
-    # the tokenizer should have added tokens
-    assert len(message.get("tokens", [])) > 0
-    # the number of regex matches on each token should match
-    for i, token in enumerate(message.get("tokens")):
-        token_matches = token.get("pattern").values()
-        num_matches = sum(token_matches)
-        assert num_matches == labeled_tokens.count(i)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected, labeled_tokens",
-    [
-        ("lemonade and mapo tofu", [1, 1], [0.0, 2.0, 3.0]),
-        ("a cup of tea", [1, 0], [3.0]),
-        ("Is burrito my favorite food?", [0, 1], [1.0]),
-        ("I want club?mate", [1, 0], [2.0, 3.0]),
-    ],
-)
-def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
-    from rasa.nlu.featurizers.regex_featurizer import RegexFeaturizer
-
-    lookups = [
-        {
-            "name": "drinks",
-            "elements": ["mojito", "lemonade", "sweet berry wine", "tea", "club?mate"],
-        },
-        {"name": "plates", "elements": "data/test/lookup_tables/plates.txt"},
-    ]
-    ftr = RegexFeaturizer(lookup_tables=lookups)
-
-    # adds tokens to the message
-    tokenizer = SpacyTokenizer()
-    message = Message(sentence)
-    message.set("spacy_doc", spacy_nlp(sentence))
-    tokenizer.process(message)
-
-    result = ftr.features_for_patterns(message)
-    assert np.allclose(result, expected, atol=1e-10)
-
-    # the tokenizer should have added tokens
-    assert len(message.get("tokens", [])) > 0
-    # the number of regex matches on each token should match
-    for i, token in enumerate(message.get("tokens")):
-        token_matches = token.get("pattern").values()
-        num_matches = sum(token_matches)
-        assert num_matches == labeled_tokens.count(i)
-
-
-def test_lookup_table_deprecation():
-    from rasa.nlu.featurizers.regex_featurizer import RegexFeaturizer
-
-    lookups = [
-        {
-            "name": "drinks",
-            "elements": ["mojito", "lemonade", "sweet berry wine", "tea", "club?mate"],
-        }
-    ]
-
-    with pytest.warns(FutureWarning):
-        RegexFeaturizer(lookup_tables=lookups)
-
-
-def test_spacy_featurizer_casing(spacy_nlp):
-    from rasa.nlu.featurizers import spacy_featurizer
-
-    # if this starts failing for the default model, we should think about
-    # removing the lower casing the spacy nlp component does when it
-    # retrieves vectors. For compressed spacy models (e.g. models
-    # ending in _sm) this test will most likely fail.
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-    for e in td.intent_examples:
-        doc = spacy_nlp(e.text)
-        doc_capitalized = spacy_nlp(e.text.capitalize())
-
-        vecs = spacy_featurizer.features_for_doc(doc)
-        vecs_capitalized = spacy_featurizer.features_for_doc(doc_capitalized)
-
-        assert np.allclose(
-            vecs, vecs_capitalized, atol=1e-5
-        ), "Vectors are unequal for texts '{}' and '{}'".format(
-            e.text, e.text.capitalize()
-        )
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        ("hello hello hello hello hello ", [5]),
-        ("hello goodbye hello", [1, 2]),
-        ("a b c d e f", [1, 1, 1, 1, 1, 1]),
-        ("a 1 2", [2, 1]),
-    ],
-)
-def test_count_vector_featurizer(sentence, expected):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
-    train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    test_message = Message(sentence)
-    ftr.process(test_message)
-
-    assert np.all(test_message.get("text_features") == expected)
-
-
-@pytest.mark.parametrize(
-    "sentence, intent, response, intent_features, response_features",
-    [
-        ("hello hello hello hello hello ", "greet", None, [1], None),
-        ("hello goodbye hello", "greet", None, [1], None),
-        ("a 1 2", "char", "char char", [1], [2]),
-    ],
-)
-def test_count_vector_featurizer_attribute_featurization(
-    sentence, intent, response, intent_features, response_features
-):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
-    train_message = Message(sentence)
-
-    # this is needed for a valid training example
-    train_message.set("intent", intent)
-    train_message.set("response", response)
-
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    assert train_message.get("intent_features") == intent_features
-    assert train_message.get("response_features") == response_features
-
-
-@pytest.mark.parametrize(
-    "sentence, intent, response, text_features, intent_features, response_features",
-    [
-        ("hello hello greet ", "greet", "hello", [1, 2], [1, 0], [0, 1]),
-        (
-            "I am fine",
-            "acknowledge",
-            "good",
-            [0, 1, 1, 0, 1],
-            [1, 0, 0, 0, 0],
-            [0, 0, 0, 1, 0],
-        ),
-    ],
-)
-def test_count_vector_featurizer_shared_vocab(
-    sentence, intent, response, text_features, intent_features, response_features
-):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "use_shared_vocab": True}
-    )
-    train_message = Message(sentence)
-
-    # this is needed for a valid training example
-    train_message.set("intent", intent)
-    train_message.set("response", response)
-
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    assert np.all(train_message.get("text_features") == text_features)
-    assert np.all(train_message.get("intent_features") == intent_features)
-    assert np.all(train_message.get("response_features") == response_features)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        ("hello hello hello hello hello __OOV__", [1, 5]),
-        ("hello goodbye hello __oov__", [1, 1, 2]),
-        ("a b c d e f __oov__ __OOV__ __OOV__", [3, 1, 1, 1, 1, 1, 1]),
-        ("__OOV__ a 1 2 __oov__ __OOV__", [2, 3, 1]),
-    ],
-)
-def test_count_vector_featurizer_oov_token(sentence, expected):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "OOV_token": "__oov__"}
-    )
-    train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    test_message = Message(sentence)
-    ftr.process(test_message)
-
-    assert np.all(test_message.get("text_features") == expected)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        ("hello hello hello hello hello oov_word0", [1, 5]),
-        ("hello goodbye hello oov_word0 OOV_word0", [2, 1, 2]),
-        ("a b c d e f __oov__ OOV_word0 oov_word1", [3, 1, 1, 1, 1, 1, 1]),
-        ("__OOV__ a 1 2 __oov__ OOV_word1", [2, 3, 1]),
-    ],
-)
-def test_count_vector_featurizer_oov_words(sentence, expected):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer(
-        {
-            "token_pattern": r"(?u)\b\w+\b",
-            "OOV_token": "__oov__",
-            "OOV_words": ["oov_word0", "OOV_word1"],
-        }
-    )
-    train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    test_message = Message(sentence)
-    ftr.process(test_message)
-
-    assert np.all(test_message.get("text_features") == expected)
-
-
-@pytest.mark.parametrize(
-    "tokens, expected",
-    [
-        (["hello", "hello", "hello", "hello", "hello"], [5]),
-        (["你好", "你好", "你好", "你好", "你好"], [5]),  # test for unicode chars
-        (["hello", "goodbye", "hello"], [1, 2]),
-        # Note: order has changed in Chinese version of "hello" & "goodbye"
-        (["你好", "再见", "你好"], [2, 1]),  # test for unicode chars
-        (["a", "b", "c", "d", "e", "f"], [1, 1, 1, 1, 1, 1]),
-        (["a", "1", "2"], [2, 1]),
-    ],
-)
-def test_count_vector_featurizer_using_tokens(tokens, expected):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
-
-    # using empty string instead of real text string to make sure
-    # count vector only can come from `tokens` feature.
-    # using `message.text` can not get correct result
-
-    tokens_feature = [Token(i, 0) for i in tokens]
-
-    train_message = Message("")
-    train_message.set("tokens", tokens_feature)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-
-    ftr.train(data)
-
-    test_message = Message("")
-    test_message.set("tokens", tokens_feature)
-
-    ftr.process(test_message)
-
-    assert np.all(test_message.get("text_features") == expected)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        ("ababab", [3, 3, 3, 2]),
-        ("ab ab ab", [2, 2, 3, 3, 3, 2]),
-        ("abc", [1, 1, 1, 1, 1]),
-    ],
-)
-def test_count_vector_featurizer_char(sentence, expected):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer({"min_ngram": 1, "max_ngram": 2, "analyzer": "char"})
-    train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
-
-    test_message = Message(sentence)
-    ftr.process(test_message)
-
-    assert np.all(test_message.get("text_features") == expected)
-
-
-def test_count_vector_featurizer_char_intent_featurizer():
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    ftr = CountVectorsFeaturizer({"min_ngram": 1, "max_ngram": 2, "analyzer": "char"})
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-    ftr.train(td, config=None)
-
-    intent_features_exist = np.array(
-        [
-            True if example.get("intent_features") is not None else False
-            for example in td.intent_examples
-        ]
-    )
-
-    # no intent features should have been set
-    assert not any(intent_features_exist)
-
-
-def test_count_vector_featurizer_persist_load(tmpdir):
-    from rasa.nlu.featurizers.count_vectors_featurizer import CountVectorsFeaturizer
-
-    # set non default values to config
-    config = {
-        "analyzer": "char",
-        "token_pattern": r"(?u)\b\w+\b",
-        "strip_accents": "ascii",
-        "stop_words": "stop",
-        "min_df": 2,
-        "max_df": 3,
-        "min_ngram": 2,
-        "max_ngram": 3,
-        "max_features": 10,
-        "lowercase": False,
-    }
-    train_ftr = CountVectorsFeaturizer(config)
-
-    sentence1 = "ababab 123 13xc лаомтгцу sfjv oö aà"
-    sentence2 = "abababalidcn 123123 13xcdc лаомтгцу sfjv oö aà"
-    train_message1 = Message(sentence1)
-    train_message2 = Message(sentence2)
-
-    # this is needed for a valid training example
-    train_message1.set("intent", "bla")
-    train_message2.set("intent", "bla")
-    data = TrainingData([train_message1, train_message2])
-    train_ftr.train(data)
-    # persist featurizer
-    file_dict = train_ftr.persist("ftr", tmpdir.strpath)
-    train_vect_params = {
-        attribute: vectorizer.get_params()
-        for attribute, vectorizer in train_ftr.vectorizers.items()
-    }
-    # add trained vocabulary to vectorizer params
-    for attribute, attribute_vect_params in train_vect_params.items():
-        if hasattr(train_ftr.vectorizers[attribute], "vocabulary_"):
-            train_vect_params[attribute].update(
-                {"vocabulary": train_ftr.vectorizers[attribute].vocabulary_}
-            )
-
-    # load featurizer
-    meta = train_ftr.component_config.copy()
-    meta.update(file_dict)
-    test_ftr = CountVectorsFeaturizer.load(meta, tmpdir.strpath)
-    test_vect_params = {
-        attribute: vectorizer.get_params()
-        for attribute, vectorizer in test_ftr.vectorizers.items()
-    }
-
-    assert train_vect_params == test_vect_params
-
-    test_message1 = Message(sentence1)
-    test_ftr.process(test_message1)
-    test_message2 = Message(sentence2)
-    test_ftr.process(test_message2)
-
-    # check that train features and test features after loading are the same
-    assert np.all(
-        [
-            train_message1.get("text_features") == test_message1.get("text_features"),
-            train_message2.get("text_features") == test_message2.get("text_features"),
-        ]
-    )
diff --git a/tests/nlu/classifiers/__init__.py b/tests/nlu/classifiers/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/nlu/classifiers/test_embedding_intent_classifier.py b/tests/nlu/classifiers/test_embedding_intent_classifier.py
new file mode 100644
index 000000000000..1c2c1edc237f
--- /dev/null
+++ b/tests/nlu/classifiers/test_embedding_intent_classifier.py
@@ -0,0 +1,106 @@
+import numpy as np
+import pytest
+import scipy.sparse
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
+    DENSE_FEATURE_NAMES,
+    INTENT_ATTRIBUTE,
+)
+from rasa.nlu.classifiers.embedding_intent_classifier import EmbeddingIntentClassifier
+from rasa.nlu.training_data import Message
+
+
+def test_compute_default_label_features():
+    label_features = [
+        Message("test a"),
+        Message("test b"),
+        Message("test c"),
+        Message("test d"),
+    ]
+
+    output = EmbeddingIntentClassifier._compute_default_label_features(label_features)
+
+    output = output[0]
+
+    for i, o in enumerate(output):
+        assert isinstance(o, np.ndarray)
+        assert o[0][i] == 1
+        assert o.shape == (1, len(label_features))
+
+
+def test_get_num_of_features():
+    session_data = {
+        "text_features": [
+            np.array(
+                [
+                    np.random.rand(5, 14),
+                    np.random.rand(2, 14),
+                    np.random.rand(3, 14),
+                    np.random.rand(1, 14),
+                    np.random.rand(3, 14),
+                ]
+            ),
+            np.array(
+                [
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(5, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(2, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(3, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(1, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(3, 10))),
+                ]
+            ),
+        ]
+    }
+
+    num_features = EmbeddingIntentClassifier._get_num_of_features(
+        session_data, "text_features"
+    )
+
+    assert num_features == 24
+
+
+@pytest.mark.parametrize(
+    "messages, expected",
+    [
+        (
+            [
+                Message(
+                    "test a",
+                    data={
+                        SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]: np.zeros(1),
+                        DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]: np.zeros(1),
+                    },
+                ),
+                Message(
+                    "test b",
+                    data={
+                        SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]: np.zeros(1),
+                        DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]: np.zeros(1),
+                    },
+                ),
+            ],
+            True,
+        ),
+        (
+            [
+                Message(
+                    "test a",
+                    data={
+                        SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE]: np.zeros(1),
+                        DENSE_FEATURE_NAMES[INTENT_ATTRIBUTE]: np.zeros(1),
+                    },
+                )
+            ],
+            False,
+        ),
+    ],
+)
+def test_check_labels_features_exist(messages, expected):
+    attribute = TEXT_ATTRIBUTE
+
+    assert (
+        EmbeddingIntentClassifier._check_labels_features_exist(messages, attribute)
+        == expected
+    )
diff --git a/tests/nlu/extractors/__init__.py b/tests/nlu/extractors/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/nlu/extractors/test_duckling_http_extractor.py b/tests/nlu/extractors/test_duckling_http_extractor.py
new file mode 100644
index 000000000000..a665e2108d3e
--- /dev/null
+++ b/tests/nlu/extractors/test_duckling_http_extractor.py
@@ -0,0 +1,227 @@
+import responses
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.training_data import Message
+
+
+def test_duckling_entity_extractor(component_builder):
+    with responses.RequestsMock() as rsps:
+        rsps.add(
+            responses.POST,
+            "http://localhost:8000/parse",
+            json=[
+                {
+                    "body": "Today",
+                    "start": 0,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2018-11-13T00:00:00.000-08:00",
+                                "grain": "day",
+                                "type": "value",
+                            }
+                        ],
+                        "value": "2018-11-13T00:00:00.000-08:00",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 5,
+                    "dim": "time",
+                    "latent": False,
+                },
+                {
+                    "body": "the 5th",
+                    "start": 9,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2018-12-05T00:00:00.000-08:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                            {
+                                "value": "2019-01-05T00:00:00.000-08:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                            {
+                                "value": "2019-02-05T00:00:00.000-08:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                        ],
+                        "value": "2018-12-05T00:00:00.000-08:00",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 16,
+                    "dim": "time",
+                    "latent": False,
+                },
+                {
+                    "body": "5th of May",
+                    "start": 13,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2019-05-05T00:00:00.000-07:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                            {
+                                "value": "2020-05-05T00:00:00.000-07:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                            {
+                                "value": "2021-05-05T00:00:00.000-07:00",
+                                "grain": "day",
+                                "type": "value",
+                            },
+                        ],
+                        "value": "2019-05-05T00:00:00.000-07:00",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 23,
+                    "dim": "time",
+                    "latent": False,
+                },
+                {
+                    "body": "tomorrow",
+                    "start": 37,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2018-11-14T00:00:00.000-08:00",
+                                "grain": "day",
+                                "type": "value",
+                            }
+                        ],
+                        "value": "2018-11-14T00:00:00.000-08:00",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 45,
+                    "dim": "time",
+                    "latent": False,
+                },
+            ],
+        )
+
+        _config = RasaNLUModelConfig({"pipeline": [{"name": "DucklingHTTPExtractor"}]})
+        _config.set_component_attr(
+            0, dimensions=["time"], timezone="UTC", url="http://localhost:8000"
+        )
+        duckling = component_builder.create_component(_config.for_component(0), _config)
+        message = Message("Today is the 5th of May. Let us meet tomorrow.")
+        duckling.process(message)
+        entities = message.get("entities")
+        assert len(entities) == 4
+
+    # Test duckling with a defined date
+
+    with responses.RequestsMock() as rsps:
+        rsps.add(
+            responses.POST,
+            "http://localhost:8000/parse",
+            json=[
+                {
+                    "body": "tomorrow",
+                    "start": 12,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2013-10-13T00:00:00.000Z",
+                                "grain": "day",
+                                "type": "value",
+                            }
+                        ],
+                        "value": "2013-10-13T00:00:00.000Z",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 20,
+                    "dim": "time",
+                    "latent": False,
+                }
+            ],
+        )
+
+        # 1381536182 == 2013/10/12 02:03:02
+        message = Message("Let us meet tomorrow.", time="1381536182")
+        duckling.process(message)
+        entities = message.get("entities")
+        assert len(entities) == 1
+        assert entities[0]["text"] == "tomorrow"
+        assert entities[0]["value"] == "2013-10-13T00:00:00.000Z"
+
+        # Test dimension filtering includes only specified dimensions
+        _config = RasaNLUModelConfig({"pipeline": [{"name": "DucklingHTTPExtractor"}]})
+        _config.set_component_attr(
+            0, dimensions=["number"], url="http://localhost:8000"
+        )
+        duckling_number = component_builder.create_component(
+            _config.for_component(0), _config
+        )
+
+    with responses.RequestsMock() as rsps:
+        rsps.add(
+            responses.POST,
+            "http://localhost:8000/parse",
+            json=[
+                {
+                    "body": "Yesterday",
+                    "start": 0,
+                    "value": {
+                        "values": [
+                            {
+                                "value": "2019-02-28T00:00:00.000+01:00",
+                                "grain": "day",
+                                "type": "value",
+                            }
+                        ],
+                        "value": "2019-02-28T00:00:00.000+01:00",
+                        "grain": "day",
+                        "type": "value",
+                    },
+                    "end": 9,
+                    "dim": "time",
+                },
+                {
+                    "body": "5",
+                    "start": 21,
+                    "value": {"value": 5, "type": "value"},
+                    "end": 22,
+                    "dim": "number",
+                },
+            ],
+        )
+
+        message = Message("Yesterday there were 5 people in a room")
+        duckling_number.process(message)
+        entities = message.get("entities")
+
+        assert len(entities) == 1
+        assert entities[0]["text"] == "5"
+        assert entities[0]["value"] == 5
+
+
+def test_duckling_entity_extractor_and_synonyms(component_builder):
+    _config = RasaNLUModelConfig(
+        {
+            "pipeline": [
+                {"name": "DucklingHTTPExtractor"},
+                {"name": "EntitySynonymMapper"},
+            ]
+        }
+    )
+    _config.set_component_attr(0, dimensions=["number"])
+    duckling = component_builder.create_component(_config.for_component(0), _config)
+    synonyms = component_builder.create_component(_config.for_component(1), _config)
+    message = Message("He was 6 feet away")
+    duckling.process(message)
+    # checks that the synonym processor
+    # can handle entities that have int values
+    synonyms.process(message)
+    assert message is not None
diff --git a/tests/nlu/extractors/test_entity_synonyms.py b/tests/nlu/extractors/test_entity_synonyms.py
new file mode 100644
index 000000000000..a6479b8f8b43
--- /dev/null
+++ b/tests/nlu/extractors/test_entity_synonyms.py
@@ -0,0 +1,30 @@
+from rasa.nlu.training_data import TrainingData, Message
+from tests.nlu import utilities
+
+
+def test_unintentional_synonyms_capitalized(component_builder):
+    _config = utilities.base_test_conf("pretrained_embeddings_spacy")
+    ner_syn = component_builder.create_component(_config.for_component(5), _config)
+    examples = [
+        Message(
+            "Any Mexican restaurant will do",
+            {
+                "intent": "restaurant_search",
+                "entities": [
+                    {"start": 4, "end": 11, "value": "Mexican", "entity": "cuisine"}
+                ],
+            },
+        ),
+        Message(
+            "I want Tacos!",
+            {
+                "intent": "restaurant_search",
+                "entities": [
+                    {"start": 7, "end": 12, "value": "Mexican", "entity": "cuisine"}
+                ],
+            },
+        ),
+    ]
+    ner_syn.train(TrainingData(training_examples=examples), _config)
+    assert ner_syn.synonyms.get("mexican") is None
+    assert ner_syn.synonyms.get("tacos") == "Mexican"
diff --git a/tests/nlu/extractors/test_spacy_entity_extractors.py b/tests/nlu/extractors/test_spacy_entity_extractors.py
new file mode 100644
index 000000000000..0c5e59ae5b7a
--- /dev/null
+++ b/tests/nlu/extractors/test_spacy_entity_extractors.py
@@ -0,0 +1,53 @@
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.training_data import Message
+
+
+def test_spacy_ner_extractor(component_builder, spacy_nlp):
+    _config = RasaNLUModelConfig({"pipeline": [{"name": "SpacyEntityExtractor"}]})
+    ext = component_builder.create_component(_config.for_component(0), _config)
+    example = Message(
+        "anywhere in the U.K.",
+        {
+            "intent": "restaurant_search",
+            "entities": [],
+            "spacy_doc": spacy_nlp("anywhere in the west"),
+        },
+    )
+
+    ext.process(example, spacy_nlp=spacy_nlp)
+
+    assert len(example.get("entities", [])) == 1
+    assert example.get("entities")[0] == {
+        "start": 16,
+        "extractor": "SpacyEntityExtractor",
+        "end": 20,
+        "value": "U.K.",
+        "entity": "GPE",
+        "confidence": None,
+    }
+
+    # Test dimension filtering includes only specified dimensions
+
+    example = Message(
+        "anywhere in the West with Sebastian Thrun",
+        {
+            "intent": "example_intent",
+            "entities": [],
+            "spacy_doc": spacy_nlp("anywhere in the West with Sebastian Thrun"),
+        },
+    )
+    _config = RasaNLUModelConfig({"pipeline": [{"name": "SpacyEntityExtractor"}]})
+
+    _config.set_component_attr(0, dimensions=["PERSON"])
+    ext = component_builder.create_component(_config.for_component(0), _config)
+    ext.process(example, spacy_nlp=spacy_nlp)
+
+    assert len(example.get("entities", [])) == 1
+    assert example.get("entities")[0] == {
+        "start": 26,
+        "extractor": "SpacyEntityExtractor",
+        "end": 41,
+        "value": "Sebastian Thrun",
+        "entity": "PERSON",
+        "confidence": None,
+    }
diff --git a/tests/nlu/extractors/text_crf_entity_extractor.py b/tests/nlu/extractors/text_crf_entity_extractor.py
new file mode 100644
index 000000000000..1ff19ba338de
--- /dev/null
+++ b/tests/nlu/extractors/text_crf_entity_extractor.py
@@ -0,0 +1,255 @@
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.training_data import TrainingData, Message
+
+
+def test_crf_extractor(spacy_nlp, ner_crf_pos_feature_config):
+    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
+
+    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
+    examples = [
+        Message(
+            "anywhere in the west",
+            {
+                "intent": "restaurant_search",
+                "entities": [
+                    {"start": 16, "end": 20, "value": "west", "entity": "location"}
+                ],
+                "spacy_doc": spacy_nlp("anywhere in the west"),
+            },
+        ),
+        Message(
+            "central indian restaurant",
+            {
+                "intent": "restaurant_search",
+                "entities": [
+                    {
+                        "start": 0,
+                        "end": 7,
+                        "value": "central",
+                        "entity": "location",
+                        "extractor": "random_extractor",
+                    },
+                    {
+                        "start": 8,
+                        "end": 14,
+                        "value": "indian",
+                        "entity": "cuisine",
+                        "extractor": "CRFEntityExtractor",
+                    },
+                ],
+                "spacy_doc": spacy_nlp("central indian restaurant"),
+            },
+        ),
+    ]
+
+    # uses BILOU and the default features
+    ext.train(TrainingData(training_examples=examples), RasaNLUModelConfig())
+    sentence = "anywhere in the west"
+    doc = {"spacy_doc": spacy_nlp(sentence)}
+    crf_format = ext._from_text_to_crf(Message(sentence, doc))
+    assert [word[0] for word in crf_format] == ["anywhere", "in", "the", "west"]
+    feats = ext._sentence_to_features(crf_format)
+    assert "BOS" in feats[0]
+    assert "EOS" in feats[-1]
+    assert feats[1]["0:low"] == "in"
+    sentence = "anywhere in the west"
+    ext.extract_entities(Message(sentence, {"spacy_doc": spacy_nlp(sentence)}))
+    filtered = ext.filter_trainable_entities(examples)
+    assert filtered[0].get("entities") == [
+        {"start": 16, "end": 20, "value": "west", "entity": "location"}
+    ], "Entity without extractor remains"
+    assert filtered[1].get("entities") == [
+        {
+            "start": 8,
+            "end": 14,
+            "value": "indian",
+            "entity": "cuisine",
+            "extractor": "CRFEntityExtractor",
+        }
+    ], "Only CRFEntityExtractor entity annotation remains"
+    assert examples[1].get("entities")[0] == {
+        "start": 0,
+        "end": 7,
+        "value": "central",
+        "entity": "location",
+        "extractor": "random_extractor",
+    }, "Original examples are not mutated"
+
+
+def test_crf_json_from_BILOU(spacy_nlp, ner_crf_pos_feature_config):
+    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
+
+    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
+    sentence = "I need a home cleaning close-by"
+    doc = {"spacy_doc": spacy_nlp(sentence)}
+    r = ext._from_crf_to_json(
+        Message(sentence, doc),
+        [
+            {"O": 1.0},
+            {"O": 1.0},
+            {"O": 1.0},
+            {"B-what": 1.0},
+            {"L-what": 1.0},
+            {"B-where": 1.0},
+            {"I-where": 1.0},
+            {"L-where": 1.0},
+        ],
+    )
+    assert len(r) == 2, "There should be two entities"
+
+    assert r[0]["confidence"]  # confidence should exist
+    del r[0]["confidence"]
+    assert r[0] == {"start": 9, "end": 22, "value": "home cleaning", "entity": "what"}
+
+    assert r[1]["confidence"]  # confidence should exist
+    del r[1]["confidence"]
+    assert r[1] == {"start": 23, "end": 31, "value": "close-by", "entity": "where"}
+
+
+def test_crf_json_from_non_BILOU(spacy_nlp, ner_crf_pos_feature_config):
+    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
+
+    ner_crf_pos_feature_config.update({"BILOU_flag": False})
+    ext = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
+    sentence = "I need a home cleaning close-by"
+    doc = {"spacy_doc": spacy_nlp(sentence)}
+    rs = ext._from_crf_to_json(
+        Message(sentence, doc),
+        [
+            {"O": 1.0},
+            {"O": 1.0},
+            {"O": 1.0},
+            {"what": 1.0},
+            {"what": 1.0},
+            {"where": 1.0},
+            {"where": 1.0},
+            {"where": 1.0},
+        ],
+    )
+
+    # non BILOU will split multi-word entities - hence 5
+    assert len(rs) == 5, "There should be five entities"
+
+    for r in rs:
+        assert r["confidence"]  # confidence should exist
+        del r["confidence"]
+
+    assert rs[0] == {"start": 9, "end": 13, "value": "home", "entity": "what"}
+    assert rs[1] == {"start": 14, "end": 22, "value": "cleaning", "entity": "what"}
+    assert rs[2] == {"start": 23, "end": 28, "value": "close", "entity": "where"}
+    assert rs[3] == {"start": 28, "end": 29, "value": "-", "entity": "where"}
+    assert rs[4] == {"start": 29, "end": 31, "value": "by", "entity": "where"}
+
+
+def test_crf_create_entity_dict(spacy_nlp):
+    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
+    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+
+    crf_extractor = CRFEntityExtractor()
+    spacy_tokenizer = SpacyTokenizer()
+    white_space_tokenizer = WhitespaceTokenizer()
+
+    examples = [
+        {
+            "message": Message(
+                "where is St. Michael's Hospital?",
+                {
+                    "intent": "search_location",
+                    "entities": [
+                        {
+                            "start": 9,
+                            "end": 31,
+                            "value": "St. Michael's Hospital",
+                            "entity": "hospital",
+                            "SpacyTokenizer": {
+                                "entity_start_token_idx": 2,
+                                "entity_end_token_idx": 5,
+                            },
+                            "WhitespaceTokenizer": {
+                                "entity_start_token_idx": 2,
+                                "entity_end_token_idx": 5,
+                            },
+                        }
+                    ],
+                },
+            )
+        },
+        {
+            "message": Message(
+                "where is Children's Hospital?",
+                {
+                    "intent": "search_location",
+                    "entities": [
+                        {
+                            "start": 9,
+                            "end": 28,
+                            "value": "Children's Hospital",
+                            "entity": "hospital",
+                            "SpacyTokenizer": {
+                                "entity_start_token_idx": 2,
+                                "entity_end_token_idx": 4,
+                            },
+                            "WhitespaceTokenizer": {
+                                "entity_start_token_idx": 2,
+                                "entity_end_token_idx": 4,
+                            },
+                        }
+                    ],
+                },
+            )
+        },
+    ]
+    for ex in examples:
+        # spacy tokenizers receives a Doc as input and whitespace tokenizer receives a text
+        spacy_tokens = spacy_tokenizer.tokenize(spacy_nlp(ex["message"].text))
+        white_space_tokens = white_space_tokenizer.tokenize(ex["message"].text)
+        for tokenizer, tokens in [
+            ("SpacyTokenizer", spacy_tokens),
+            ("WhitespaceTokenizer", white_space_tokens),
+        ]:
+            for entity in ex["message"].get("entities"):
+                parsed_entities = crf_extractor._create_entity_dict(
+                    ex["message"],
+                    tokens,
+                    entity[tokenizer]["entity_start_token_idx"],
+                    entity[tokenizer]["entity_end_token_idx"],
+                    entity["entity"],
+                    0.8,
+                )
+                assert parsed_entities == {
+                    "start": entity["start"],
+                    "end": entity["end"],
+                    "value": entity["value"],
+                    "entity": entity["entity"],
+                    "confidence": 0.8,
+                }
+
+
+def test_crf_use_dense_features(ner_crf_pos_feature_config, spacy_nlp):
+    from rasa.nlu.extractors.crf_entity_extractor import CRFEntityExtractor
+    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    ner_crf_pos_feature_config["features"][1].append("text_dense_features")
+    crf_extractor = CRFEntityExtractor(component_config=ner_crf_pos_feature_config)
+
+    spacy_featurizer = SpacyFeaturizer()
+    white_space_tokenizer = WhitespaceTokenizer({"use_cls_token": False})
+
+    text = "Rasa is a company in Berlin"
+    message = Message(text)
+    message.set("spacy_doc", spacy_nlp(text))
+
+    white_space_tokenizer.process(message)
+    spacy_featurizer.process(message)
+
+    text_data = crf_extractor._from_text_to_crf(message)
+    features = crf_extractor._sentence_to_features(text_data)
+
+    assert "0:text_dense_features" in features[0]
+    for i in range(0, len(message.data.get("text_dense_features")[0])):
+        assert (
+            features[0]["0:text_dense_features"]["text_dense_features"][str(i)]
+            == message.data.get("text_dense_features")[0][i]
+        )
diff --git a/tests/nlu/featurizers/__init__.py b/tests/nlu/featurizers/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/nlu/featurizers/test_convert_featurizer.py b/tests/nlu/featurizers/test_convert_featurizer.py
new file mode 100644
index 000000000000..66e25c4f9d4d
--- /dev/null
+++ b/tests/nlu/featurizers/test_convert_featurizer.py
@@ -0,0 +1,28 @@
+import numpy as np
+
+from rasa.nlu.constants import TEXT_ATTRIBUTE, DENSE_FEATURE_NAMES, TOKENS_NAMES
+from rasa.nlu.training_data import Message
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+from rasa.nlu.config import RasaNLUModelConfig
+
+
+def test_convert_featurizer(mitie_feature_extractor, default_config):
+    from rasa.nlu.featurizers.dense_featurizer.convert_featurizer import (
+        ConveRTFeaturizer,
+    )
+
+    component_config = {"name": "ConveRTFeaturizer", "return_sequence": False}
+    featurizer = ConveRTFeaturizer.create(component_config, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today ?"
+    message = Message(sentence)
+    tokens = WhitespaceTokenizer().tokenize(sentence)
+    message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens)
+
+    featurizer.process(message)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])[0]
+
+    expected = np.array([1.0251294, -0.04053932, -0.7018805, -0.82054937, -0.75054353])
+
+    assert np.allclose(vecs[:5], expected, atol=1e-5)
diff --git a/tests/nlu/featurizers/test_count_vectors_featurizer.py b/tests/nlu/featurizers/test_count_vectors_featurizer.py
new file mode 100644
index 000000000000..66d3b473b810
--- /dev/null
+++ b/tests/nlu/featurizers/test_count_vectors_featurizer.py
@@ -0,0 +1,371 @@
+import numpy as np
+import pytest
+import scipy.sparse
+
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.training_data import Message
+from rasa.nlu.training_data import TrainingData
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        ("hello hello hello hello hello ", [[1]]),
+        ("hello goodbye hello", [[0, 1]]),
+        ("a b c d e f", [[1, 0, 0, 0, 0, 0]]),
+        ("a 1 2", [[0, 1]]),
+    ],
+)
+def test_count_vector_featurizer(sentence, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
+    )
+    train_message = Message(sentence)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    test_message = Message(sentence)
+    ftr.process(test_message)
+
+    assert isinstance(test_message.get("text_sparse_features"), scipy.sparse.coo_matrix)
+
+    actual = test_message.get("text_sparse_features").toarray()
+
+    assert np.all(actual[0] == expected)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        ("hello hello hello hello hello ", [[5]]),
+        ("hello goodbye hello", [[1, 2]]),
+        ("a b c d e f", [[1, 1, 1, 1, 1, 1]]),
+        ("a 1 2", [[2, 1]]),
+    ],
+)
+def test_count_vector_featurizer_no_sequence(sentence, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": False}
+    )
+    train_message = Message(sentence)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    test_message = Message(sentence)
+    ftr.process(test_message)
+
+    assert isinstance(test_message.get("text_sparse_features"), scipy.sparse.coo_matrix)
+
+    actual = test_message.get("text_sparse_features").toarray()
+
+    assert np.all(actual == expected)
+
+
+@pytest.mark.parametrize(
+    "sentence, intent, response, intent_features, response_features",
+    [
+        ("hello hello hello hello hello ", "greet", None, [[1]], None),
+        ("hello goodbye hello", "greet", None, [[1]], None),
+        ("a 1 2", "char", "char char", [[1]], [[1]]),
+    ],
+)
+def test_count_vector_featurizer_attribute_featurization(
+    sentence, intent, response, intent_features, response_features
+):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
+    )
+    train_message = Message(sentence)
+
+    # this is needed for a valid training example
+    train_message.set("intent", intent)
+    train_message.set("response", response)
+
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    if intent_features:
+        assert (
+            train_message.get("intent_sparse_features").toarray()[0] == intent_features
+        )
+    else:
+        assert train_message.get("intent_sparse_features") is None
+
+    if response_features:
+        assert (
+            train_message.get("response_sparse_features").toarray()[0]
+            == response_features
+        )
+    else:
+        assert train_message.get("response_sparse_features") is None
+
+
+@pytest.mark.parametrize(
+    "sentence, intent, response, text_features, intent_features, response_features",
+    [
+        ("hello hello greet ", "greet", "hello", [[0, 1]], [[1, 0]], [[0, 1]]),
+        (
+            "I am fine",
+            "acknowledge",
+            "good",
+            [[0, 0, 0, 0, 1]],
+            [[1, 0, 0, 0, 0]],
+            [[0, 0, 0, 1, 0]],
+        ),
+    ],
+)
+def test_count_vector_featurizer_shared_vocab(
+    sentence, intent, response, text_features, intent_features, response_features
+):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {
+            "token_pattern": r"(?u)\b\w+\b",
+            "use_shared_vocab": True,
+            "return_sequence": True,
+        }
+    )
+    train_message = Message(sentence)
+
+    # this is needed for a valid training example
+    train_message.set("intent", intent)
+    train_message.set("response", response)
+
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    assert np.all(
+        train_message.get("text_sparse_features").toarray()[0] == text_features
+    )
+    assert np.all(
+        train_message.get("intent_sparse_features").toarray()[0] == intent_features
+    )
+    assert np.all(
+        train_message.get("response_sparse_features").toarray()[0] == response_features
+    )
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        ("hello hello hello hello hello __OOV__", [[0, 1]]),
+        ("hello goodbye hello __oov__", [[0, 0, 1]]),
+        ("a b c d e f __oov__ __OOV__ __OOV__", [[0, 1, 0, 0, 0, 0, 0]]),
+        ("__OOV__ a 1 2 __oov__ __OOV__", [[0, 1, 0]]),
+    ],
+)
+def test_count_vector_featurizer_oov_token(sentence, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {
+            "token_pattern": r"(?u)\b\w+\b",
+            "OOV_token": "__oov__",
+            "return_sequence": True,
+        }
+    )
+    train_message = Message(sentence)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    test_message = Message(sentence)
+    ftr.process(test_message)
+
+    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        ("hello hello hello hello hello oov_word0", [[0, 1]]),
+        ("hello goodbye hello oov_word0 OOV_word0", [[0, 0, 1]]),
+        ("a b c d e f __oov__ OOV_word0 oov_word1", [[0, 1, 0, 0, 0, 0, 0]]),
+        ("__OOV__ a 1 2 __oov__ OOV_word1", [[0, 1, 0]]),
+    ],
+)
+def test_count_vector_featurizer_oov_words(sentence, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {
+            "token_pattern": r"(?u)\b\w+\b",
+            "OOV_token": "__oov__",
+            "OOV_words": ["oov_word0", "OOV_word1"],
+            "return_sequence": True,
+        }
+    )
+    train_message = Message(sentence)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    test_message = Message(sentence)
+    ftr.process(test_message)
+
+    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+
+
+@pytest.mark.parametrize(
+    "tokens, expected",
+    [
+        (["hello", "hello", "hello", "hello", "hello"], [[1]]),
+        (["你好", "你好", "你好", "你好", "你好"], [[1]]),  # test for unicode chars
+        (["hello", "goodbye", "hello"], [[0, 1]]),
+        # Note: order has changed in Chinese version of "hello" & "goodbye"
+        (["你好", "再见", "你好"], [[1, 0]]),  # test for unicode chars
+        (["a", "b", "c", "d", "e", "f"], [[1, 0, 0, 0, 0, 0]]),
+        (["a", "1", "2"], [[0, 1]]),
+    ],
+)
+def test_count_vector_featurizer_using_tokens(tokens, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
+    )
+
+    # using empty string instead of real text string to make sure
+    # count vector only can come from `tokens` feature.
+    # using `message.text` can not get correct result
+
+    tokens_feature = [Token(i, 0) for i in tokens]
+
+    train_message = Message("")
+    train_message.set("tokens", tokens_feature)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+
+    ftr.train(data)
+
+    test_message = Message("")
+    test_message.set("tokens", tokens_feature)
+
+    ftr.process(test_message)
+
+    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        ("ababab", [[3, 3, 3, 2]]),
+        ("ab ab ab", [[0, 0, 1, 1, 1, 0]]),
+        ("abc", [[1, 1, 1, 1, 1]]),
+    ],
+)
+def test_count_vector_featurizer_char(sentence, expected):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    ftr = CountVectorsFeaturizer(
+        {"min_ngram": 1, "max_ngram": 2, "analyzer": "char", "return_sequence": True}
+    )
+    train_message = Message(sentence)
+    # this is needed for a valid training example
+    train_message.set("intent", "bla")
+    data = TrainingData([train_message])
+    ftr.train(data)
+
+    test_message = Message(sentence)
+    ftr.process(test_message)
+
+    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+
+
+def test_count_vector_featurizer_persist_load(tmpdir):
+    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+        CountVectorsFeaturizer,
+    )
+
+    # set non default values to config
+    config = {
+        "analyzer": "char",
+        "token_pattern": r"(?u)\b\w+\b",
+        "strip_accents": "ascii",
+        "stop_words": "stop",
+        "min_df": 2,
+        "max_df": 3,
+        "min_ngram": 2,
+        "max_ngram": 3,
+        "max_features": 10,
+        "lowercase": False,
+        "return_sequence": True,
+    }
+    train_ftr = CountVectorsFeaturizer(config)
+
+    sentence1 = "ababab 123 13xc лаомтгцу sfjv oö aà"
+    sentence2 = "abababalidcn 123123 13xcdc лаомтгцу sfjv oö aà"
+    train_message1 = Message(sentence1)
+    train_message2 = Message(sentence2)
+
+    # this is needed for a valid training example
+    train_message1.set("intent", "bla")
+    train_message2.set("intent", "bla")
+    data = TrainingData([train_message1, train_message2])
+    train_ftr.train(data)
+    # persist featurizer
+    file_dict = train_ftr.persist("ftr", tmpdir.strpath)
+    train_vect_params = {
+        attribute: vectorizer.get_params()
+        for attribute, vectorizer in train_ftr.vectorizers.items()
+    }
+    # add trained vocabulary to vectorizer params
+    for attribute, attribute_vect_params in train_vect_params.items():
+        if hasattr(train_ftr.vectorizers[attribute], "vocabulary_"):
+            train_vect_params[attribute].update(
+                {"vocabulary": train_ftr.vectorizers[attribute].vocabulary_}
+            )
+
+    # load featurizer
+    meta = train_ftr.component_config.copy()
+    meta.update(file_dict)
+    test_ftr = CountVectorsFeaturizer.load(meta, tmpdir.strpath)
+    test_vect_params = {
+        attribute: vectorizer.get_params()
+        for attribute, vectorizer in test_ftr.vectorizers.items()
+    }
+
+    assert train_vect_params == test_vect_params
+
+    test_message1 = Message(sentence1)
+    test_ftr.process(test_message1)
+    test_message2 = Message(sentence2)
+    test_ftr.process(test_message2)
+
+    # check that train features and test features after loading are the same
+    assert np.all(
+        [
+            train_message1.get("text_features") == test_message1.get("text_features"),
+            train_message2.get("text_features") == test_message2.get("text_features"),
+        ]
+    )
diff --git a/tests/nlu/featurizers/test_featurizer.py b/tests/nlu/featurizers/test_featurizer.py
new file mode 100644
index 000000000000..2d7d6436cccc
--- /dev/null
+++ b/tests/nlu/featurizers/test_featurizer.py
@@ -0,0 +1,99 @@
+import numpy as np
+import pytest
+import scipy.sparse
+
+from rasa.nlu.featurizers.featurzier import Featurizer, sequence_to_sentence_features
+from rasa.nlu.constants import DENSE_FEATURE_NAMES, SPARSE_FEATURE_NAMES, TEXT_ATTRIBUTE
+from rasa.nlu.training_data import Message
+
+
+def test_combine_with_existing_dense_features():
+
+    featurizer = Featurizer({"return_sequence": False})
+    attribute = DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+
+    existing_features = [[1, 0, 2, 3], [2, 0, 0, 1]]
+    new_features = [[1, 0], [0, 1]]
+    expected_features = [[1, 0, 2, 3, 1, 0], [2, 0, 0, 1, 0, 1]]
+
+    message = Message("This is a text.")
+    message.set(attribute, existing_features)
+
+    actual_features = featurizer._combine_with_existing_dense_features(
+        message, new_features, attribute
+    )
+
+    assert np.all(expected_features == actual_features)
+
+
+def test_combine_with_existing_dense_features_shape_mismatch():
+    featurizer = Featurizer({"return_sequence": False})
+    attribute = DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+
+    existing_features = [[1, 0, 2, 3], [2, 0, 0, 1]]
+    new_features = [[0, 1]]
+
+    message = Message("This is a text.")
+    message.set(attribute, existing_features)
+
+    with pytest.raises(ValueError):
+        featurizer._combine_with_existing_dense_features(
+            message, new_features, attribute
+        )
+
+
+def test_combine_with_existing_sparse_features():
+
+    featurizer = Featurizer({"return_sequence": False})
+    attribute = SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+
+    existing_features = scipy.sparse.csr_matrix([[1, 0, 2, 3], [2, 0, 0, 1]])
+    new_features = scipy.sparse.csr_matrix([[1, 0], [0, 1]])
+    expected_features = [[1, 0, 2, 3, 1, 0], [2, 0, 0, 1, 0, 1]]
+
+    message = Message("This is a text.")
+    message.set(attribute, existing_features)
+
+    actual_features = featurizer._combine_with_existing_sparse_features(
+        message, new_features, attribute
+    )
+    actual_features = actual_features.toarray()
+
+    assert np.all(expected_features == actual_features)
+
+
+def test_combine_with_existing_sparse_features_shape_mismatch():
+
+    featurizer = Featurizer({"return_sequence": False})
+    attribute = SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+
+    existing_features = scipy.sparse.csr_matrix([[1, 0, 2, 3], [2, 0, 0, 1]])
+    new_features = scipy.sparse.csr_matrix([[0, 1]])
+
+    message = Message("This is a text.")
+    message.set(attribute, existing_features)
+
+    with pytest.raises(ValueError):
+        featurizer._combine_with_existing_sparse_features(
+            message, new_features, attribute
+        )
+
+
+@pytest.mark.parametrize(
+    "features, expected",
+    [
+        ([[1, 0, 2, 3], [2, 0, 0, 1]], [1.5, 0, 1, 2]),
+        (
+            scipy.sparse.csr_matrix([[1, 0, 2, 3], [2, 0, 0, 1]]),
+            scipy.sparse.csr_matrix([3, 0, 2, 4]),
+        ),
+        (None, None),
+    ],
+)
+def test_sequence_to_sentence_features(features, expected):
+    actual = sequence_to_sentence_features(features)
+
+    if isinstance(expected, scipy.sparse.spmatrix):
+        assert np.all(expected.toarray() == actual.toarray())
+    else:
+        assert np.all(expected == actual)
diff --git a/tests/nlu/featurizers/test_mitie_featurizer.py b/tests/nlu/featurizers/test_mitie_featurizer.py
new file mode 100644
index 000000000000..fa9a581de7d1
--- /dev/null
+++ b/tests/nlu/featurizers/test_mitie_featurizer.py
@@ -0,0 +1,38 @@
+import numpy as np
+
+from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.constants import CLS_TOKEN
+
+
+def test_mitie_featurizer(mitie_feature_extractor, default_config):
+    from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
+
+    component_config = {"name": "MitieFeaturizer", "return_sequence": True}
+    featurizer = MitieFeaturizer.create(component_config, RasaNLUModelConfig())
+
+    sentence = f"Hey how are you today {CLS_TOKEN}"
+
+    tokens = MitieTokenizer().tokenize(sentence)
+
+    vecs = featurizer.features_for_tokens(tokens, mitie_feature_extractor)[0]
+
+    expected = np.array(
+        [0.00000000e00, -5.12735510e00, 4.39929873e-01, -5.60760403e00, -8.26445103e00]
+    )
+    assert np.allclose(vecs[:5], expected, atol=1e-5)
+
+
+def test_mitie_featurizer_no_sequence(mitie_feature_extractor, default_config):
+    from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
+
+    component_config = {"name": "MitieFeaturizer", "return_sequence": False}
+    featurizer = MitieFeaturizer.create(component_config, RasaNLUModelConfig())
+
+    sentence = f"Hey how are you today {CLS_TOKEN}"
+    tokens = MitieTokenizer().tokenize(sentence)
+
+    vecs = featurizer.features_for_tokens(tokens, mitie_feature_extractor)[0]
+
+    expected = np.array([0.0, -4.4551446, 0.26073121, -1.46632245, -1.84205751])
+    assert np.allclose(vecs[:5], expected, atol=1e-5)
diff --git a/tests/nlu/featurizers/test_regex_featurizer.py b/tests/nlu/featurizers/test_regex_featurizer.py
new file mode 100644
index 000000000000..cf252a09a49a
--- /dev/null
+++ b/tests/nlu/featurizers/test_regex_featurizer.py
@@ -0,0 +1,148 @@
+import numpy as np
+import pytest
+
+from rasa.nlu.constants import TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE, SPACY_DOCS
+from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+from rasa.nlu.training_data import Message
+
+
+@pytest.mark.parametrize(
+    "sentence, expected, labeled_tokens",
+    [
+        (
+            "hey how are you today",
+            [
+                [0.0, 1.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+            ],
+            [0],
+        ),
+        (
+            "hey 456 how are you",
+            [
+                [0.0, 1.0, 0.0],
+                [1.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+            ],
+            [1, 0],
+        ),
+        (
+            "blah balh random eh",
+            [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
+            [],
+        ),
+        (
+            "a 1 digit number",
+            [[0.0, 0.0, 0.0], [1.0, 0.0, 1.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
+            [1, 1],
+        ),
+    ],
+)
+def test_regex_featurizer(sentence, expected, labeled_tokens, spacy_nlp):
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+
+    patterns = [
+        {"pattern": "[0-9]+", "name": "number", "usage": "intent"},
+        {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
+        {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
+    ]
+    ftr = RegexFeaturizer({"return_sequence": True}, known_patterns=patterns)
+
+    # adds tokens to the message
+    tokenizer = SpacyTokenizer({"use_cls_token": False})
+    message = Message(sentence, data={RESPONSE_ATTRIBUTE: sentence})
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(sentence))
+    tokenizer.process(message)
+
+    result = ftr._features_for_patterns(message, TEXT_ATTRIBUTE)
+    assert np.allclose(result.toarray(), expected, atol=1e-10)
+
+    # the tokenizer should have added tokens
+    assert len(message.get("tokens", [])) > 0
+    # the number of regex matches on each token should match
+    for i, token in enumerate(message.get("tokens")):
+        token_matches = token.get("pattern").values()
+        num_matches = sum(token_matches)
+        assert num_matches == labeled_tokens.count(i)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected, labeled_tokens",
+    [
+        (
+            "lemonade and mapo tofu",
+            [[1.0, 0.0], [0.0, 0.0], [0.0, 1.0], [0.0, 1.0]],
+            [0.0, 2.0, 3.0],
+        ),
+        ("a cup of tea", [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [1.0, 0.0]], [3.0]),
+        (
+            "Is burrito my favorite food?",
+            [[0.0, 0.0], [0.0, 1.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0]],
+            [1.0],
+        ),
+        ("I want club?mate", [[0.0, 0.0], [0.0, 0.0], [1.0, 0.0]], [2.0, 3.0]),
+    ],
+)
+def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+
+    lookups = [
+        {
+            "name": "drinks",
+            "elements": ["mojito", "lemonade", "sweet berry wine", "tea", "club?mate"],
+        },
+        {"name": "plates", "elements": "data/test/lookup_tables/plates.txt"},
+    ]
+    ftr = RegexFeaturizer({"return_sequence": True}, lookup_tables=lookups)
+
+    # adds tokens to the message
+    component_config = {"name": "SpacyTokenizer", "use_cls_token": False}
+    tokenizer = SpacyTokenizer(component_config)
+    message = Message(sentence)
+    message.set("spacy_doc", spacy_nlp(sentence))
+    tokenizer.process(message)
+
+    result = ftr._features_for_patterns(message, TEXT_ATTRIBUTE)
+    assert np.allclose(result.toarray(), expected, atol=1e-10)
+
+    # the tokenizer should have added tokens
+    assert len(message.get("tokens", [])) > 0
+    # the number of regex matches on each token should match
+    for i, token in enumerate(message.get("tokens")):
+        token_matches = token.get("pattern").values()
+        num_matches = sum(token_matches)
+        assert num_matches == labeled_tokens.count(i)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected ",
+    [
+        ("hey how are you today", [0.0, 1.0, 0.0]),
+        ("hey 456 how are you", [1.0, 1.0, 0.0]),
+        ("blah balh random eh", [0.0, 0.0, 0.0]),
+        ("a 1 digit number", [1.0, 0.0, 1.0]),
+    ],
+)
+def test_regex_featurizer_no_sequence(sentence, expected, spacy_nlp):
+    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+
+    patterns = [
+        {"pattern": "[0-9]+", "name": "number", "usage": "intent"},
+        {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
+        {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
+    ]
+    ftr = RegexFeaturizer({"return_sequence": False}, known_patterns=patterns)
+
+    # adds tokens to the message
+    tokenizer = SpacyTokenizer()
+    message = Message(sentence)
+    message.set("spacy_doc", spacy_nlp(sentence))
+    tokenizer.process(message)
+
+    result = ftr._features_for_patterns(message, TEXT_ATTRIBUTE)
+    assert np.allclose(result.toarray()[0], expected, atol=1e-10)
diff --git a/tests/nlu/featurizers/test_spacy_featurizer.py b/tests/nlu/featurizers/test_spacy_featurizer.py
new file mode 100644
index 000000000000..536075ad1900
--- /dev/null
+++ b/tests/nlu/featurizers/test_spacy_featurizer.py
@@ -0,0 +1,144 @@
+import numpy as np
+import pytest
+
+from rasa.nlu import training_data
+from rasa.nlu.training_data import Message
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.config import RasaNLUModelConfig
+
+
+@pytest.mark.parametrize("sentence", ["hey how are you today"])
+def test_spacy_featurizer(sentence, spacy_nlp):
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    ftr = SpacyFeaturizer.create({"return_sequence": True}, RasaNLUModelConfig())
+
+    doc = spacy_nlp(sentence)
+    vecs = ftr._features_for_doc(doc)
+    expected = [t.vector for t in doc]
+    assert np.allclose(vecs, expected, atol=1e-5)
+
+
+def test_spacy_training_sample_alignment(spacy_nlp_component):
+    from spacy.tokens import Doc
+
+    m1 = Message.build(text="I have a feeling", intent="feeling")
+    m2 = Message.build(text="", intent="feeling")
+    m3 = Message.build(text="I am the last message", intent="feeling")
+    td = TrainingData(training_examples=[m1, m2, m3])
+
+    attribute_docs = spacy_nlp_component.docs_for_training_data(td)
+
+    assert isinstance(attribute_docs["text"][0], Doc)
+    assert isinstance(attribute_docs["text"][1], Doc)
+    assert isinstance(attribute_docs["text"][2], Doc)
+
+    assert [t.text for t in attribute_docs["text"][0]] == ["i", "have", "a", "feeling"]
+    assert [t.text for t in attribute_docs["text"][1]] == []
+    assert [t.text for t in attribute_docs["text"][2]] == [
+        "i",
+        "am",
+        "the",
+        "last",
+        "message",
+    ]
+
+
+def test_spacy_intent_featurizer(spacy_nlp_component):
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
+    spacy_nlp_component.train(td, config=None)
+    spacy_featurizer = SpacyFeaturizer({"return_sequence": True})
+    spacy_featurizer.train(td, config=None)
+
+    intent_features_exist = np.array(
+        [
+            True if example.get("intent_features") is not None else False
+            for example in td.intent_examples
+        ]
+    )
+
+    # no intent features should have been set
+    assert not any(intent_features_exist)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [("hey how are you today", [-0.28451, 0.31007, -0.57039, -0.073056, -0.17322])],
+)
+def test_spacy_featurizer_sequence(sentence, expected, spacy_nlp):
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    doc = spacy_nlp(sentence)
+    token_vectors = [t.vector for t in doc]
+
+    spacy_config = {"return_sequence": True}
+    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
+
+    greet = {"intent": "greet", "text_features": [0.5]}
+
+    message = Message(sentence, greet)
+    message.set("spacy_doc", doc)
+
+    ftr._set_spacy_features(message)
+
+    vecs = message.get("text_dense_features")[0][:5]
+
+    assert np.allclose(token_vectors[0][:5], vecs, atol=1e-4)
+    assert np.allclose(vecs, expected, atol=1e-4)
+
+
+@pytest.mark.parametrize(
+    "sentence, expected",
+    [
+        (
+            "hey how are you today",
+            [-0.19649599, 0.32493639, -0.37408298, -0.10622784, 0.062756],
+        )
+    ],
+)
+def test_spacy_featurizer_no_sequence(sentence, expected, spacy_nlp):
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    doc = spacy_nlp(sentence)
+
+    spacy_config = {"return_sequence": False}
+    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
+
+    greet = {"intent": "greet", "text_features": [0.5]}
+
+    message = Message(sentence, greet)
+    message.set("spacy_doc", doc)
+
+    ftr._set_spacy_features(message)
+
+    vecs = message.get("text_dense_features")[0]
+
+    assert np.allclose(doc.vector, vecs, atol=1e-4)
+    assert np.allclose(expected, vecs[:5], atol=1e-4)
+
+
+def test_spacy_featurizer_casing(spacy_nlp):
+    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+
+    # if this starts failing for the default model, we should think about
+    # removing the lower casing the spacy nlp component does when it
+    # retrieves vectors. For compressed spacy models (e.g. models
+    # ending in _sm) this test will most likely fail.
+
+    ftr = SpacyFeaturizer.create({"return_sequence": True}, RasaNLUModelConfig())
+
+    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
+    for e in td.intent_examples:
+        doc = spacy_nlp(e.text)
+        doc_capitalized = spacy_nlp(e.text.capitalize())
+
+        vecs = ftr._features_for_doc(doc)
+        vecs_capitalized = ftr._features_for_doc(doc_capitalized)
+
+        assert np.allclose(
+            vecs, vecs_capitalized, atol=1e-5
+        ), "Vectors are unequal for texts '{}' and '{}'".format(
+            e.text, e.text.capitalize()
+        )
diff --git a/tests/nlu/tokenizers/__init__.py b/tests/nlu/tokenizers/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/nlu/tokenizers/test_jieba_tokenizer.py b/tests/nlu/tokenizers/test_jieba_tokenizer.py
new file mode 100644
index 000000000000..7df57c5bfcd1
--- /dev/null
+++ b/tests/nlu/tokenizers/test_jieba_tokenizer.py
@@ -0,0 +1,53 @@
+from unittest.mock import patch
+
+from rasa.nlu.constants import CLS_TOKEN
+
+
+def test_jieba():
+    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
+
+    component_config = {"use_cls_token": False}
+
+    tk = JiebaTokenizer(component_config)
+
+    assert [t.text for t in tk.tokenize("我想去吃兰州拉面")] == ["我", "想", "去", "吃", "兰州", "拉面"]
+
+    assert [t.offset for t in tk.tokenize("我想去吃兰州拉面")] == [0, 1, 2, 3, 4, 6]
+
+    assert [t.text for t in tk.tokenize("Micheal你好吗？")] == ["Micheal", "你好", "吗", "？"]
+
+    assert [t.offset for t in tk.tokenize("Micheal你好吗？")] == [0, 7, 9, 10]
+
+
+def test_jieba_load_dictionary(tmpdir_factory):
+    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
+
+    dictionary_path = tmpdir_factory.mktemp("jieba_custom_dictionary").strpath
+
+    component_config = {"dictionary_path": dictionary_path, "use_cls_token": False}
+
+    with patch.object(
+        JiebaTokenizer, "load_custom_dictionary", return_value=None
+    ) as mock_method:
+        tk = JiebaTokenizer(component_config)
+        tk.tokenize("")
+
+    mock_method.assert_called_once_with(dictionary_path)
+
+
+def test_jieba_add_cls_token():
+    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
+
+    component_config = {"use_cls_token": True}
+
+    tk = JiebaTokenizer(component_config)
+
+    assert [t.text for t in tk.tokenize("Micheal你好吗？")] == [
+        "Micheal",
+        "你好",
+        "吗",
+        "？",
+        CLS_TOKEN,
+    ]
+
+    assert [t.offset for t in tk.tokenize("Micheal你好吗？")] == [0, 7, 9, 10, 12]
diff --git a/tests/nlu/tokenizers/test_mitie_tokenizer.py b/tests/nlu/tokenizers/test_mitie_tokenizer.py
new file mode 100644
index 000000000000..41774fb9a440
--- /dev/null
+++ b/tests/nlu/tokenizers/test_mitie_tokenizer.py
@@ -0,0 +1,41 @@
+from rasa.nlu.constants import CLS_TOKEN
+
+
+def test_mitie():
+    from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+
+    component_config = {"use_cls_token": False}
+
+    tk = MitieTokenizer(component_config)
+
+    text = "Forecast for lunch"
+    assert [t.text for t in tk.tokenize(text)] == ["Forecast", "for", "lunch"]
+    assert [t.offset for t in tk.tokenize(text)] == [0, 9, 13]
+
+    text = "hey ńöñàśçií how're you?"
+    assert [t.text for t in tk.tokenize(text)] == [
+        "hey",
+        "ńöñàśçií",
+        "how",
+        "'re",
+        "you",
+        "?",
+    ]
+    assert [t.offset for t in tk.tokenize(text)] == [0, 4, 13, 16, 20, 23]
+
+
+def test_mitie_add_cls_token():
+    from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+
+    component_config = {"use_cls_token": True}
+
+    tk = MitieTokenizer(component_config)
+
+    text = "Forecast for lunch"
+    assert [t.text for t in tk.tokenize(text)] == [
+        "Forecast",
+        "for",
+        "lunch",
+        CLS_TOKEN,
+    ]
+    assert [t.offset for t in tk.tokenize(text)] == [0, 9, 13, 19]
diff --git a/tests/nlu/tokenizers/test_spacy_tokenizer.py b/tests/nlu/tokenizers/test_spacy_tokenizer.py
new file mode 100644
index 000000000000..396fc848f249
--- /dev/null
+++ b/tests/nlu/tokenizers/test_spacy_tokenizer.py
@@ -0,0 +1,71 @@
+from rasa.nlu.constants import CLS_TOKEN
+from rasa.nlu import training_data
+
+
+def test_spacy(spacy_nlp):
+    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+
+    component_config = {"use_cls_token": False}
+
+    tk = SpacyTokenizer(component_config)
+
+    text = "Forecast for lunch"
+    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
+        "Forecast",
+        "for",
+        "lunch",
+    ]
+    assert [t.lemma for t in tk.tokenize(spacy_nlp(text))] == [
+        "forecast",
+        "for",
+        "lunch",
+    ]
+
+    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 9, 13]
+
+    text = "hey ńöñàśçií how're you?"
+    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
+        "hey",
+        "ńöñàśçií",
+        "how",
+        "'re",
+        "you",
+        "?",
+    ]
+    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 4, 13, 16, 20, 23]
+
+
+def test_spacy_add_cls_token(spacy_nlp):
+    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+
+    component_config = {"use_cls_token": True}
+
+    tk = SpacyTokenizer(component_config)
+
+    text = "Forecast for lunch"
+    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
+        "Forecast",
+        "for",
+        "lunch",
+        CLS_TOKEN,
+    ]
+    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 9, 13, 19]
+
+
+def test_spacy_intent_tokenizer(spacy_nlp_component):
+    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+
+    component_config = {"use_cls_token": False}
+
+    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
+    spacy_nlp_component.train(td, config=None)
+    spacy_tokenizer = SpacyTokenizer(component_config)
+    spacy_tokenizer.train(td, config=None)
+
+    intent_tokens_exist = [
+        True if example.get("intent_tokens") is not None else False
+        for example in td.intent_examples
+    ]
+
+    # no intent tokens should have been set
+    assert not any(intent_tokens_exist)
diff --git a/tests/nlu/tokenizers/test_tokenizer.py b/tests/nlu/tokenizers/test_tokenizer.py
new file mode 100644
index 000000000000..2b0e2d8cb32d
--- /dev/null
+++ b/tests/nlu/tokenizers/test_tokenizer.py
@@ -0,0 +1,16 @@
+import pytest
+
+
+def test_tokens_comparison():
+    from rasa.nlu.tokenizers.tokenizer import Token
+
+    x = Token("hello", 0)
+    y = Token("Hello", 0)
+
+    assert x == x
+    assert y < x
+
+    assert x != 1
+
+    with pytest.raises(TypeError):
+        assert y < "a"
diff --git a/tests/nlu/base/test_tokenizers.py b/tests/nlu/tokenizers/test_whitespace_tokenizer.py
similarity index 55%
rename from tests/nlu/base/test_tokenizers.py
rename to tests/nlu/tokenizers/test_whitespace_tokenizer.py
index 6f5a5f719fb0..a8e3b7bdabb4 100644
--- a/tests/nlu/base/test_tokenizers.py
+++ b/tests/nlu/tokenizers/test_whitespace_tokenizer.py
@@ -1,37 +1,25 @@
-from unittest.mock import patch
-
-import pytest
-
-from rasa.nlu import training_data
+from rasa.nlu.constants import CLS_TOKEN
 from rasa.nlu.training_data import TrainingData, Message
 from tests.nlu import utilities
 
 
-def test_tokens_comparison():
-    from rasa.nlu.tokenizers import Token
-
-    x = Token("hello", 0)
-    y = Token("Hello", 0)
-
-    assert x == x
-    assert y < x
-
-    assert x != 1
-
-    with pytest.raises(TypeError):
-        assert y < "a"
-
-
 def test_whitespace():
     from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 
-    tk = WhitespaceTokenizer()
+    component_config = {"use_cls_token": False}
+
+    tk = WhitespaceTokenizer(component_config)
 
     assert [t.text for t in tk.tokenize("Forecast for lunch")] == [
         "Forecast",
         "for",
         "lunch",
     ]
+    assert [t.lemma for t in tk.tokenize("Forecast for lunch")] == [
+        "Forecast",
+        "for",
+        "lunch",
+    ]
 
     assert [t.offset for t in tk.tokenize("Forecast for lunch")] == [0, 9, 13]
 
@@ -93,10 +81,30 @@ def test_whitespace():
     ] == [0, 83]
 
 
+def test_whitespace_cls_token():
+    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+
+    component_config = {"use_cls_token": True}
+
+    tk = WhitespaceTokenizer(component_config)
+
+    assert [t.text for t in tk.tokenize("Forecast for lunch")] == [
+        "Forecast",
+        "for",
+        "lunch",
+        CLS_TOKEN,
+    ]
+    assert [t.offset for t in tk.tokenize("Forecast for lunch")] == [0, 9, 13, 19]
+
+
 def test_whitespace_custom_intent_symbol():
     from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 
-    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
+    component_config = {
+        "intent_tokenization_flag": True,
+        "intent_split_symbol": "+",
+        "use_cls_token": False,
+    }
 
     tk = WhitespaceTokenizer(component_config)
 
@@ -114,23 +122,23 @@ def test_whitespace_custom_intent_symbol():
 def test_whitespace_with_case():
     from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 
-    component_config = {"case_sensitive": False}
+    component_config = {"use_cls_token": False}
     tk = WhitespaceTokenizer(component_config)
     assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
-        "forecast",
+        "Forecast",
         "for",
-        "lunch",
+        "LUNCH",
     ]
 
-    component_config = {"case_sensitive": True}
+    component_config = {"case_sensitive": False, "use_cls_token": False}
     tk = WhitespaceTokenizer(component_config)
     assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
-        "Forecast",
+        "forecast",
         "for",
-        "LUNCH",
+        "lunch",
     ]
 
-    component_config = {}
+    component_config = {"case_sensitive": True, "use_cls_token": False}
     tk = WhitespaceTokenizer(component_config)
     assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
         "Forecast",
@@ -138,14 +146,6 @@ def test_whitespace_with_case():
         "LUNCH",
     ]
 
-    component_config = {"case_sensitive": False}
-    tk = WhitespaceTokenizer(component_config)
-    message = Message("Forecast for LUNCH")
-    tk.process(message)
-    assert message.data.get("tokens")[0].text == "forecast"
-    assert message.data.get("tokens")[1].text == "for"
-    assert message.data.get("tokens")[2].text == "lunch"
-
     _config = utilities.base_test_conf("supervised_embeddings")
     examples = [
         Message(
@@ -179,96 +179,3 @@ def test_whitespace_with_case():
     assert examples[1].data.get("tokens")[0].text == "i"
     assert examples[1].data.get("tokens")[1].text == "want"
     assert examples[1].data.get("tokens")[2].text == "tacos"
-
-
-def test_spacy(spacy_nlp):
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-
-    tk = SpacyTokenizer()
-
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
-        "Forecast",
-        "for",
-        "lunch",
-    ]
-    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 9, 13]
-
-    text = "hey ńöñàśçií how're you?"
-    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
-        "hey",
-        "ńöñàśçií",
-        "how",
-        "'re",
-        "you",
-        "?",
-    ]
-    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 4, 13, 16, 20, 23]
-
-
-def test_spacy_intent_tokenizer(spacy_nlp_component):
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-    spacy_nlp_component.train(td, config=None)
-    spacy_tokenizer = SpacyTokenizer()
-    spacy_tokenizer.train(td, config=None)
-
-    intent_tokens_exist = [
-        True if example.get("intent_tokens") is not None else False
-        for example in td.intent_examples
-    ]
-
-    # no intent tokens should have been set
-    assert not any(intent_tokens_exist)
-
-
-def test_mitie():
-    from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
-
-    tk = MitieTokenizer()
-
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(text)] == ["Forecast", "for", "lunch"]
-    assert [t.offset for t in tk.tokenize(text)] == [0, 9, 13]
-
-    text = "hey ńöñàśçií how're you?"
-    assert [t.text for t in tk.tokenize(text)] == [
-        "hey",
-        "ńöñàśçií",
-        "how",
-        "'re",
-        "you",
-        "?",
-    ]
-    assert [t.offset for t in tk.tokenize(text)] == [0, 4, 13, 16, 20, 23]
-
-
-def test_jieba():
-    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
-
-    tk = JiebaTokenizer()
-
-    assert [t.text for t in tk.tokenize("我想去吃兰州拉面")] == ["我", "想", "去", "吃", "兰州", "拉面"]
-
-    assert [t.offset for t in tk.tokenize("我想去吃兰州拉面")] == [0, 1, 2, 3, 4, 6]
-
-    assert [t.text for t in tk.tokenize("Micheal你好吗？")] == ["Micheal", "你好", "吗", "？"]
-
-    assert [t.offset for t in tk.tokenize("Micheal你好吗？")] == [0, 7, 9, 10]
-
-
-def test_jieba_load_dictionary(tmpdir_factory):
-    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
-
-    dictionary_path = tmpdir_factory.mktemp("jieba_custom_dictionary").strpath
-
-    component_config = {"dictionary_path": dictionary_path}
-
-    with patch.object(
-        JiebaTokenizer, "load_custom_dictionary", return_value=None
-    ) as mock_method:
-        tk = JiebaTokenizer(component_config)
-        tk.tokenize("")
-
-    mock_method.assert_called_once_with(dictionary_path)
diff --git a/tests/nlu/training/test_train.py b/tests/nlu/training/test_train.py
index dba73ef076f3..3bd187ab3f0e 100644
--- a/tests/nlu/training/test_train.py
+++ b/tests/nlu/training/test_train.py
@@ -32,7 +32,6 @@ def pipelines_for_tests():
                 "SpacyTokenizer",
                 "MitieFeaturizer",
                 "SpacyFeaturizer",
-                "NGramFeaturizer",
                 "RegexFeaturizer",
                 "CountVectorsFeaturizer",
                 "ConveRTFeaturizer",
diff --git a/tests/utils/test_train_utils.py b/tests/utils/test_train_utils.py
new file mode 100644
index 000000000000..bdbb89cde492
--- /dev/null
+++ b/tests/utils/test_train_utils.py
@@ -0,0 +1,167 @@
+import pytest
+import scipy.sparse
+import numpy as np
+
+from rasa.utils.train_utils import (
+    SessionDataType,
+    shuffle_session_data,
+    split_session_data_by_label_ids,
+    train_val_split,
+    session_data_for_ids,
+    get_number_of_examples,
+    gen_batch,
+    balance_session_data,
+)
+
+
+@pytest.fixture
+async def session_data() -> SessionDataType:
+    return {
+        "text_features": [
+            np.array(
+                [
+                    np.random.rand(5, 14),
+                    np.random.rand(2, 14),
+                    np.random.rand(3, 14),
+                    np.random.rand(1, 14),
+                    np.random.rand(3, 14),
+                ]
+            ),
+            np.array(
+                [
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(5, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(2, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(3, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(1, 10))),
+                    scipy.sparse.csr_matrix(np.random.randint(5, size=(3, 10))),
+                ]
+            ),
+        ],
+        "intent_features": [
+            np.array(
+                [
+                    np.random.randint(2, size=(5, 10)),
+                    np.random.randint(2, size=(2, 10)),
+                    np.random.randint(2, size=(3, 10)),
+                    np.random.randint(2, size=(1, 10)),
+                    np.random.randint(2, size=(3, 10)),
+                ]
+            )
+        ],
+        "intent_ids": [np.array([0, 1, 0, 1, 1])],
+        "tag_ids": [
+            np.array(
+                [
+                    np.array([0, 1, 1, 0, 2]),
+                    np.array([2, 0]),
+                    np.array([0, 1, 1]),
+                    np.array([0, 1]),
+                    np.array([0, 0, 0]),
+                ]
+            )
+        ],
+    }
+
+
+def test_shuffle_session_data(session_data: SessionDataType):
+    shuffeled_session_data = shuffle_session_data(session_data)
+
+    assert np.array(shuffeled_session_data.values()) != np.array(session_data.values())
+
+
+def test_split_session_data_by_label(session_data: SessionDataType):
+    split_session_data = split_session_data_by_label_ids(
+        session_data, session_data["intent_ids"][0], np.array([0, 1])
+    )
+
+    assert len(split_session_data) == 2
+    for s in split_session_data:
+        assert len(set(s["intent_ids"][0])) == 1
+
+
+def test_train_val_split(session_data: SessionDataType):
+    train_session_data, val_session_data = train_val_split(
+        session_data, 2, 42, "intent_ids"
+    )
+
+    for k, values in session_data.items():
+        assert len(values) == len(train_session_data[k])
+        assert len(values) == len(val_session_data[k])
+        for i, v in enumerate(values):
+            assert v[0].dtype == train_session_data[k][i][0].dtype
+
+    for values in train_session_data.values():
+        for v in values:
+            assert v.shape[0] == 3
+
+    for values in val_session_data.values():
+        for v in values:
+            assert v.shape[0] == 2
+
+
+@pytest.mark.parametrize("size", [0, 1, 5])
+def test_train_val_split_incorrect_size(session_data: SessionDataType, size):
+    with pytest.raises(ValueError):
+        train_val_split(session_data, size, 42, "intent_ids")
+
+
+def test_session_data_for_ids(session_data: SessionDataType):
+    filtered_session_data = session_data_for_ids(session_data, np.array([0, 1]))
+
+    for values in filtered_session_data.values():
+        for v in values:
+            assert v.shape[0] == 2
+
+    k = list(session_data.keys())[0]
+
+    assert np.all(
+        np.array(filtered_session_data[k][0][0]) == np.array(session_data[k][0][0])
+    )
+    assert np.all(
+        np.array(filtered_session_data[k][0][1]) == np.array(session_data[k][0][1])
+    )
+
+
+def test_get_number_of_examples(session_data: SessionDataType):
+    num = get_number_of_examples(session_data)
+
+    assert num == 5
+
+
+def test_get_number_of_examples_raises_value_error(session_data: SessionDataType):
+    session_data["dense"] = np.random.randint(5, size=(2, 10))
+    with pytest.raises(ValueError):
+        get_number_of_examples(session_data)
+
+
+def test_gen_batch(session_data: SessionDataType):
+    iterator = gen_batch(
+        session_data, 2, "intent_ids", shuffle=True, batch_strategy="balanced"
+    )
+
+    batch = next(iterator)
+    assert len(batch) == 7
+    assert len(batch[0]) == 2
+
+    batch = next(iterator)
+    assert len(batch) == 7
+    assert len(batch[0]) == 2
+
+    batch = next(iterator)
+    assert len(batch) == 7
+    assert len(batch[0]) == 1
+
+    with pytest.raises(StopIteration):
+        next(iterator)
+
+
+def test_balance_session_data(session_data: SessionDataType):
+    balanced_session_data = balance_session_data(session_data, 2, False, "intent_ids")
+
+    for k, values in session_data.items():
+        assert k in balanced_session_data
+
+        for i, v in enumerate(values):
+            assert len(v) == len(balanced_session_data[k][i])
+
+    assert np.all(balanced_session_data["intent_ids"][0] == np.array([0, 1, 1, 0, 1]))


Code review comments:
diff_hunk: @@ -0,0 +1,131 @@
+import numpy as np
+import typing
+from typing import Any, List, Text, Dict
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    import mitie
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
+    MESSAGE_ATTRIBUTES,
+    DENSE_FEATURE_NAMES,
+    CLS_TOKEN,
+)
+
+
+class MitieFeaturizer(Featurizer):
+
+    provides = [DENSE_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    requires = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES] + [
+        "mitie_feature_extractor"
+    ]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
path: rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
text: @user1:
```suggestion
    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
```

diff_hunk: @@ -0,0 +1,88 @@
+import numpy as np
+import typing
+from typing import Any, Optional, Dict, Text
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    from spacy.tokens import Doc
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    SPACY_DOCS,
+    DENSE_FEATURE_NAMES,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
+    TOKENS_NAMES,
+    CLS_TOKEN,
+)
+
+
+class SpacyFeaturizer(Featurizer):
+
+    provides = [
+        DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ]
+
+    requires = [
+        SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ] + [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
path: rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
text: @user1:
```suggestion
    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
```

diff_hunk: @@ -0,0 +1,88 @@
+import numpy as np
+import typing
+from typing import Any, Optional, Dict, Text
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    from spacy.tokens import Doc
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    SPACY_DOCS,
+    DENSE_FEATURE_NAMES,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
+    TOKENS_NAMES,
+    CLS_TOKEN,
+)
+
+
+class SpacyFeaturizer(Featurizer):
+
+    provides = [
+        DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ]
+
+    requires = [
+        SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ] + [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    def _features_for_doc(self, doc: "Doc") -> np.ndarray:
+        """Feature vector for a single document / sentence / tokens."""
+        if self.return_sequence:
+            return np.array([t.vector for t in doc])
+        return np.array([doc.vector])
+
+    def train(
+        self,
+        training_data: TrainingData,
+        config: Optional[RasaNLUModelConfig],
+        **kwargs: Any,
+    ) -> None:
+
+        for example in training_data.intent_examples:
+            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
+                self._set_spacy_features(example, attribute)
+
+    def get_doc(self, message, attribute):
path: rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
text: @user1:
```suggestion
    def get_doc(self, message: Message, attribute: Text) -> Any:
```

diff_hunk: @@ -0,0 +1,88 @@
+import numpy as np
+import typing
+from typing import Any, Optional, Dict, Text
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    from spacy.tokens import Doc
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    SPACY_DOCS,
+    DENSE_FEATURE_NAMES,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
+    TOKENS_NAMES,
+    CLS_TOKEN,
+)
+
+
+class SpacyFeaturizer(Featurizer):
+
+    provides = [
+        DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ]
+
+    requires = [
+        SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
+    ] + [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    def _features_for_doc(self, doc: "Doc") -> np.ndarray:
+        """Feature vector for a single document / sentence / tokens."""
+        if self.return_sequence:
+            return np.array([t.vector for t in doc])
+        return np.array([doc.vector])
+
+    def train(
+        self,
+        training_data: TrainingData,
+        config: Optional[RasaNLUModelConfig],
+        **kwargs: Any,
+    ) -> None:
+
+        for example in training_data.intent_examples:
+            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
+                self._set_spacy_features(example, attribute)
+
+    def get_doc(self, message, attribute):
+
+        return message.get(SPACY_DOCS[attribute])
+
+    def process(self, message: Message, **kwargs: Any) -> None:
+
+        self._set_spacy_features(message)
+
+    def _set_spacy_features(self, message, attribute=TEXT_ATTRIBUTE):
path: rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
text: @user1:
```suggestion
    def _set_spacy_features(self, message: Message, attribute: Text=TEXT_ATTRIBUTE) -> Any:
```

diff_hunk: @@ -0,0 +1,131 @@
+import numpy as np
+import typing
+from typing import Any, List, Text, Dict
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    import mitie
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
+    MESSAGE_ATTRIBUTES,
+    DENSE_FEATURE_NAMES,
+    CLS_TOKEN,
+)
+
+
+class MitieFeaturizer(Featurizer):
+
+    provides = [DENSE_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    requires = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES] + [
+        "mitie_feature_extractor"
+    ]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    @classmethod
+    def required_packages(cls) -> List[Text]:
+        return ["mitie", "numpy"]
+
+    def ndim(self, feature_extractor: "mitie.total_word_feature_extractor"):
+
+        return feature_extractor.num_dimensions
+
+    def get_tokens_by_attribute(self, example, attribute) -> Any:
path: rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
text: @user1:
```suggestion
    def get_tokens_by_attribute(self, example: Message, attribute: Text) -> Any:
```

diff_hunk: @@ -184,12 +183,12 @@ def _check_analyzer(self) -> None:
                 )
 
     @staticmethod
-    def _attributes(analyzer):
+    def _attributes_for(analyzer):
path: rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
text: @user1:
```suggestion
    def _attributes_for(analyzer: Text) -> List[Text]:
```

diff_hunk: @@ -211,184 +210,178 @@ def __init__(
         self._check_analyzer()
 
         # set which attributes to featurize
-        self._attributes = self._attributes(self.analyzer)
+        self._attributes = self._attributes_for(self.analyzer)
 
         # declare class instance for CountVectorizer
         self.vectorizers = vectorizers
 
-    def _get_message_text_by_attribute(
-        self, message: "Message", attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
-        """Get processed text of attribute of a message"""
-
-        if message.get(attribute) is None:
-            # return empty string since sklearn countvectorizer does not like None object while training and predicting
-            return ""
-
-        tokens = self._get_message_tokens_by_attribute(message, attribute)
-
-        text = self._process_text(tokens, attribute)
-
-        text = self._replace_with_oov_token(text, attribute)
+    @staticmethod
+    def _get_message_tokens_by_attribute(
+        message: "Message", attribute: Text
+    ) -> List[Text]:
+        """Get text tokens of an attribute of a message"""
+        if message.get(TOKENS_NAMES[attribute]):
+            return [t.lemma for t in message.get(TOKENS_NAMES[attribute])]
 
-        return text
+        return message.get(attribute).split()
 
-    def _process_text(
-        self, tokens: List[Text], attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
+    def _process_tokens(
+        self, tokens: List[Text], attribute: Text = TEXT_ATTRIBUTE
+    ) -> List[Text]:
         """Apply processing and cleaning steps to text"""
 
-        text = " ".join(tokens)
-
-        if attribute == MESSAGE_INTENT_ATTRIBUTE:
+        if attribute == INTENT_ATTRIBUTE:
             # Don't do any processing for intent attribute. Treat them as whole labels
-            return text
+            return tokens
 
         # replace all digits with NUMBER token
-        text = re.sub(r"\b[0-9]+\b", "__NUMBER__", text)
+        tokens = [re.sub(r"\b[0-9]+\b", "__NUMBER__", text) for text in tokens]
 
         # convert to lowercase if necessary
         if self.lowercase:
-            text = text.lower()
-        return text
+            tokens = [text.lower() for text in tokens]
+
+        return tokens
 
-    def _replace_with_oov_token(self, text: Text, attribute: Text) -> Text:
+    def _replace_with_oov_token(
+        self, tokens: List[Text], attribute: Text
+    ) -> List[Text]:
         """Replace OOV words with OOV token"""
 
         if self.OOV_token and self.analyzer == "word":
-            text_tokens = text.split()
-            if self._check_attribute_vocabulary(
+            vocabulary_exists = self._check_attribute_vocabulary(attribute)
+            if vocabulary_exists and self.OOV_token in self._get_attribute_vocabulary(
                 attribute
-            ) and self.OOV_token in self._get_attribute_vocabulary(attribute):
+            ):
                 # CountVectorizer is trained, process for prediction
-                text_tokens = [
+                tokens = [
                     t
                     if t in self._get_attribute_vocabulary_tokens(attribute)
                     else self.OOV_token
-                    for t in text_tokens
+                    for t in tokens
                 ]
             elif self.OOV_words:
                 # CountVectorizer is not trained, process for train
-                text_tokens = [
-                    self.OOV_token if t in self.OOV_words else t for t in text_tokens
-                ]
-            text = " ".join(text_tokens)
-        return text
+                tokens = [self.OOV_token if t in self.OOV_words else t for t in tokens]
 
-    @staticmethod
-    def _get_message_tokens_by_attribute(
-        message: "Message", attribute: Text
+        return tokens
+
+    def _get_processed_message_tokens_by_attribute(
+        self, message: "Message", attribute: Text = TEXT_ATTRIBUTE
     ) -> List[Text]:
-        """Get text tokens of an attribute of a message"""
+        """Get processed text of attribute of a message"""
+
+        if message.get(attribute) is None:
+            # return empty string since sklearn countvectorizer does not like None
+            # object while training and predicting
+            return [""]
+
+        tokens = self._get_message_tokens_by_attribute(message, attribute)
+        tokens = self._process_tokens(tokens, attribute)
+        tokens = self._replace_with_oov_token(tokens, attribute)
 
-        if attribute in SPACY_FEATURIZABLE_ATTRIBUTES and message.get(
-            MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        ):  # if lemmatize is possible
-            tokens = [
-                t.lemma_ for t in message.get(MESSAGE_SPACY_FEATURES_NAMES[attribute])
-            ]
-        elif message.get(
-            MESSAGE_TOKENS_NAMES[attribute]
-        ):  # if directly tokens is provided
-            tokens = [t.text for t in message.get(MESSAGE_TOKENS_NAMES[attribute])]
-        else:
-            tokens = message.get(attribute).split()
         return tokens
 
     # noinspection PyPep8Naming
-    def _check_OOV_present(self, examples) -> None:
+    def _check_OOV_present(self, all_tokens: List[List[Text]]):
path: rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
text: @user1:
```suggestion
    def _check_OOV_present(self, all_tokens: List[List[Text]]) -> None:
```

diff_hunk: @@ -397,30 +390,60 @@ def _train_with_independent_vocab(self, attribute_texts: Dict[Text, List[Text]])
                     self.vectorizers[attribute].fit(attribute_texts[attribute])
                 except ValueError:
                     logger.warning(
-                        f"Unable to train CountVectorizer for message attribute "
-                        "{attribute}. "
-                        "Leaving an untrained CountVectorizer for it"
+                        f"Unable to train CountVectorizer for message "
+                        f"attribute {attribute}. Leaving an untrained "
+                        f"CountVectorizer for it."
                     )
             else:
                 logger.debug(
-                    "No text provided for {} attribute in any messages of training data. Skipping "
-                    "training a CountVectorizer for it.".format(attribute)
+                    f"No text provided for {attribute} attribute in any messages of "
+                    f"training data. Skipping training a CountVectorizer for it."
                 )
 
+    def _create_sequence(
+        self, attribute: Text, all_tokens: List[List[Text]]
+    ) -> List[scipy.sparse.coo_matrix]:
+        X = []
+
+        for i, tokens in enumerate(all_tokens):
+            # vectorizer.transform returns a sparse matrix of size
+            # [n_samples, n_features]
+            # set input to list of tokens if sequence should be returned
+            # otherwise join all tokens to a single string and pass that as a list
+            input = tokens
+            if not self.return_sequence:
+                input = [" ".join(tokens)]
+
+            x = self.vectorizers[attribute].transform(input)
+            x.sort_indices()
+            X.append(x.tocoo())
+
+        return X
+
     def _get_featurized_attribute(
-        self, attribute: Text, attribute_texts: List[Text]
-    ) -> Optional[np.ndarray]:
+        self, attribute: Text, all_tokens: List[List[Text]]
+    ) -> Optional[List[scipy.sparse.coo_matrix]]:
         """Return features of a particular attribute for complete data"""
 
         if self._check_attribute_vocabulary(attribute):
             # count vectorizer was trained
-            featurized_attributes = (
-                self.vectorizers[attribute].transform(attribute_texts).toarray()
-            )
-            return featurized_attributes
+            return self._create_sequence(attribute, all_tokens)
         else:
             return None
 
+    def _set_attribute_features(
+        self, attribute: Text, attribute_features: List, training_data: "TrainingData"
+    ):
path: rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
text: @user1:
```suggestion
    ) -> None:
```

diff_hunk: @@ -27,15 +29,22 @@
 
 class RegexFeaturizer(Featurizer):
 
-    provides = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    provides = [SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]]
 
-    requires = [MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE]]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
 
     def __init__(
         self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        known_patterns=None,
-        lookup_tables=None,
+        component_config: Dict[Text, Any] = None,
+        known_patterns: List[Dict[Text, Text]] = None,
path: rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
text: @user1:
```suggestion
        known_patterns: Optional[List[Dict[Text, Text]]] = None,
```

diff_hunk: @@ -33,7 +33,7 @@
 CONFIG_MANDATORY_KEYS_NLU = ["language", "pipeline"]
 CONFIG_MANDATORY_KEYS = CONFIG_MANDATORY_KEYS_CORE + CONFIG_MANDATORY_KEYS_NLU
 
-MINIMUM_COMPATIBLE_VERSION = "1.3.0a2"
+MINIMUM_COMPATIBLE_VERSION = "1.6.0a1"
path: rasa/constants.py
text: @user1:
can you please increase the version of the librabry to `1.6.0a2` and set this to `a2` as well?

if someone was working with master they will aready have models trained with `1.6.0a1` but these won't be compatible (not really relevant for end-users, but for anyone of us developing with it)

diff_hunk: @@ -0,0 +1,26 @@
+import logging
+import warnings
+
+from typing import Any, Dict, Optional, Text
+
+from rasa.nlu.featurizers.featurzier import Featurizer
+
+logger = logging.getLogger(__name__)
+
+
+class NGramFeaturizer(Featurizer):
+    def __init__(self, component_config: Optional[Dict[Text, Any]] = None):
path: rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
text: @user1:
```suggestion
    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
```

diff_hunk: @@ -44,12 +52,30 @@ def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
 
         self._load_model()
 
+        self.return_sequence = self.component_config["return_sequence"]
+
+        if self.return_sequence:
+            raise NotImplementedError(
+                f"ConveRTFeaturizer always returns a feature vector of size "
+                f"(1 x feature-dimensions). It cannot return a proper sequence "
+                f"right now. ConveRTFeaturizer can only be used "
+                f"with 'return_sequence' set to False. Also, any other featurizer "
+                f"used next to ConveRTFeaturizer should have the flag "
+                f"'return_sequence' set to False."
+            )
+
+        logger.debug(
path: rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
text: @user1:
What do we need this debug statement for? I'd remove it

diff_hunk: @@ -27,15 +29,22 @@
 
 class RegexFeaturizer(Featurizer):
 
-    provides = [MESSAGE_VECTOR_FEATURE_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    provides = [SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]]
 
-    requires = [MESSAGE_TOKENS_NAMES[MESSAGE_TEXT_ATTRIBUTE]]
+    requires = [TOKENS_NAMES[TEXT_ATTRIBUTE]]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
 
     def __init__(
         self,
-        component_config: Optional[Dict[Text, Any]] = None,
-        known_patterns=None,
-        lookup_tables=None,
+        component_config: Dict[Text, Any] = None,
+        known_patterns: List[Dict[Text, Text]] = None,
+        lookup_tables: List[Dict[Text, Union[Text, List]]] = None,
path: rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
text: @user1:
```suggestion
        lookup_tables: Optional[List[Dict[Text, Union[Text, List]]]] = None,
```

diff_hunk: @@ -0,0 +1,76 @@
+import logging
+
+from typing import Text, List, Optional, Dict, Any
+
+from rasa.nlu.components import Component
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE, CLS_TOKEN
+
+logger = logging.getLogger(__name__)
+
+
+class Token(object):
+    def __init__(
+        self,
+        text: Text,
+        offset: int,
+        data: Optional[Dict[Text, Any]] = None,
+        lemma: Optional[Text] = None,
+    ):
+        self.offset = offset
+        self.text = text
+        self.end = offset + len(text)
+        self.data = data if data else {}
+        self.lemma = lemma or text
+
+    def set(self, prop: Text, info: Any):
+        self.data[prop] = info
+
+    def get(self, prop: Text, default: Optional[Any] = None):
path: rasa/nlu/tokenizers/tokenizer.py
text: @user1:
```suggestion
    def get(self, prop: Text, default: Optional[Any] = None) -> Any:
```

diff_hunk: @@ -0,0 +1,76 @@
+import logging
+
+from typing import Text, List, Optional, Dict, Any
+
+from rasa.nlu.components import Component
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE, CLS_TOKEN
+
+logger = logging.getLogger(__name__)
+
+
+class Token(object):
+    def __init__(
+        self,
+        text: Text,
+        offset: int,
+        data: Optional[Dict[Text, Any]] = None,
+        lemma: Optional[Text] = None,
+    ):
path: rasa/nlu/tokenizers/tokenizer.py
text: @user1:
```suggestion
    ) -> None:
```

diff_hunk: @@ -0,0 +1,76 @@
+import logging
+
+from typing import Text, List, Optional, Dict, Any
+
+from rasa.nlu.components import Component
+from rasa.nlu.constants import RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE, CLS_TOKEN
+
+logger = logging.getLogger(__name__)
+
+
+class Token(object):
+    def __init__(
+        self,
+        text: Text,
+        offset: int,
+        data: Optional[Dict[Text, Any]] = None,
+        lemma: Optional[Text] = None,
+    ):
+        self.offset = offset
+        self.text = text
+        self.end = offset + len(text)
+        self.data = data if data else {}
+        self.lemma = lemma or text
+
+    def set(self, prop: Text, info: Any):
path: rasa/nlu/tokenizers/tokenizer.py
text: @user1:
```suggestion
    def set(self, prop: Text, info: Any) -> None:
```

diff_hunk: @@ -180,98 +268,313 @@ def balance_session_data(
             if min(num_data_cycles) > 0:
                 break
 
-    return SessionData(
-        X=np.concatenate(new_X),
-        Y=np.concatenate(new_Y),
-        label_ids=np.concatenate(new_label_ids),
-    )
+    final_session_data = defaultdict(list)
+    for k, values in new_session_data.items():
+        for v in values:
+            final_session_data[k].append(np.concatenate(np.array(v)))
+
+    return final_session_data
+
+
+def get_number_of_examples(session_data: SessionDataType):
+    """Obtain number of examples in session data.
+
+    Raise a ValueError if number of examples differ for different data in session data.
+    """
+
+    example_lengths = [v.shape[0] for values in session_data.values() for v in values]
+
+    # check if number of examples is the same for all values
+    if not all(length == example_lengths[0] for length in example_lengths):
+        raise ValueError(
+            f"Number of examples differs for keys '{session_data.keys()}'. Number of "
+            f"examples should be the same for all data in session data."
+        )
+
+    return example_lengths[0]
 
 
 def gen_batch(
-    session_data: "SessionData",
+    session_data: SessionDataType,
     batch_size: int,
+    label_key: Text,
     batch_strategy: Text = "sequence",
     shuffle: bool = False,
-) -> Generator[Tuple["np.ndarray", "np.ndarray"], None, None]:
+) -> Generator[Tuple, None, None]:
     """Generate batches."""
 
     if shuffle:
         session_data = shuffle_session_data(session_data)
 
     if batch_strategy == "balanced":
-        session_data = balance_session_data(session_data, batch_size, shuffle)
+        session_data = balance_session_data(
+            session_data, batch_size, shuffle, label_key
+        )
 
-    num_batches = session_data.X.shape[0] // batch_size + int(
-        session_data.X.shape[0] % batch_size > 0
-    )
+    num_examples = get_number_of_examples(session_data)
+    num_batches = num_examples // batch_size + int(num_examples % batch_size > 0)
 
     for batch_num in range(num_batches):
-        batch_x = session_data.X[batch_num * batch_size : (batch_num + 1) * batch_size]
-        batch_y = session_data.Y[batch_num * batch_size : (batch_num + 1) * batch_size]
+        start = batch_num * batch_size
+        end = start + batch_size
 
-        yield batch_x, batch_y
+        yield prepare_batch(session_data, start, end)
+
+
+def prepare_batch(
+    session_data: SessionDataType,
+    start: Optional[int] = None,
+    end: Optional[int] = None,
+    tuple_sizes: Dict[Text, int] = None,
path: rasa/utils/train_utils.py
text: @user1:
```suggestion
    tuple_sizes: Optional[Dict[Text, int]] = None,
```

diff_hunk: @@ -0,0 +1,131 @@
+import numpy as np
+import typing
+from typing import Any, List, Text, Dict
+
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.featurzier import Featurizer
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.training_data import Message, TrainingData
+
+if typing.TYPE_CHECKING:
+    import mitie
+
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
+    MESSAGE_ATTRIBUTES,
+    DENSE_FEATURE_NAMES,
+    CLS_TOKEN,
+)
+
+
+class MitieFeaturizer(Featurizer):
+
+    provides = [DENSE_FEATURE_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    requires = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES] + [
+        "mitie_feature_extractor"
+    ]
+
+    defaults = {
+        # if True return a sequence of features (return vector has size
+        # token-size x feature-dimension)
+        # if False token-size will be equal to 1
+        "return_sequence": False
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None):
+
+        super().__init__(component_config)
+
+        self.return_sequence = self.component_config["return_sequence"]
+
+    @classmethod
+    def required_packages(cls) -> List[Text]:
+        return ["mitie", "numpy"]
+
+    def ndim(self, feature_extractor: "mitie.total_word_feature_extractor"):
path: rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
text: @user1:
```suggestion
    def ndim(self, feature_extractor: "mitie.total_word_feature_extractor") -> int:
```

diff_hunk: @@ -180,98 +268,313 @@ def balance_session_data(
             if min(num_data_cycles) > 0:
                 break
 
-    return SessionData(
-        X=np.concatenate(new_X),
-        Y=np.concatenate(new_Y),
-        label_ids=np.concatenate(new_label_ids),
-    )
+    final_session_data = defaultdict(list)
+    for k, values in new_session_data.items():
+        for v in values:
+            final_session_data[k].append(np.concatenate(np.array(v)))
+
+    return final_session_data
+
+
+def get_number_of_examples(session_data: SessionDataType):
path: rasa/utils/train_utils.py
text: @user1:
```suggestion
def get_number_of_examples(session_data: SessionDataType) -> int:
```

diff_hunk: @@ -211,184 +210,178 @@ def __init__(
         self._check_analyzer()
 
         # set which attributes to featurize
-        self._attributes = self._attributes(self.analyzer)
+        self._attributes = self._attributes_for(self.analyzer)
 
         # declare class instance for CountVectorizer
         self.vectorizers = vectorizers
 
-    def _get_message_text_by_attribute(
-        self, message: "Message", attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
-        """Get processed text of attribute of a message"""
-
-        if message.get(attribute) is None:
-            # return empty string since sklearn countvectorizer does not like None object while training and predicting
-            return ""
-
-        tokens = self._get_message_tokens_by_attribute(message, attribute)
-
-        text = self._process_text(tokens, attribute)
-
-        text = self._replace_with_oov_token(text, attribute)
+    @staticmethod
+    def _get_message_tokens_by_attribute(
+        message: "Message", attribute: Text
+    ) -> List[Text]:
+        """Get text tokens of an attribute of a message"""
+        if message.get(TOKENS_NAMES[attribute]):
+            return [t.lemma for t in message.get(TOKENS_NAMES[attribute])]
 
-        return text
+        return message.get(attribute).split()
 
-    def _process_text(
-        self, tokens: List[Text], attribute: Text = MESSAGE_TEXT_ATTRIBUTE
-    ) -> Text:
+    def _process_tokens(
+        self, tokens: List[Text], attribute: Text = TEXT_ATTRIBUTE
+    ) -> List[Text]:
         """Apply processing and cleaning steps to text"""
 
-        text = " ".join(tokens)
-
-        if attribute == MESSAGE_INTENT_ATTRIBUTE:
+        if attribute == INTENT_ATTRIBUTE:
             # Don't do any processing for intent attribute. Treat them as whole labels
-            return text
+            return tokens
 
         # replace all digits with NUMBER token
-        text = re.sub(r"\b[0-9]+\b", "__NUMBER__", text)
+        tokens = [re.sub(r"\b[0-9]+\b", "__NUMBER__", text) for text in tokens]
 
         # convert to lowercase if necessary
         if self.lowercase:
-            text = text.lower()
-        return text
+            tokens = [text.lower() for text in tokens]
+
+        return tokens
 
-    def _replace_with_oov_token(self, text: Text, attribute: Text) -> Text:
+    def _replace_with_oov_token(
+        self, tokens: List[Text], attribute: Text
+    ) -> List[Text]:
         """Replace OOV words with OOV token"""
 
         if self.OOV_token and self.analyzer == "word":
-            text_tokens = text.split()
-            if self._check_attribute_vocabulary(
+            vocabulary_exists = self._check_attribute_vocabulary(attribute)
+            if vocabulary_exists and self.OOV_token in self._get_attribute_vocabulary(
                 attribute
-            ) and self.OOV_token in self._get_attribute_vocabulary(attribute):
+            ):
                 # CountVectorizer is trained, process for prediction
-                text_tokens = [
+                tokens = [
                     t
                     if t in self._get_attribute_vocabulary_tokens(attribute)
                     else self.OOV_token
-                    for t in text_tokens
+                    for t in tokens
                 ]
             elif self.OOV_words:
                 # CountVectorizer is not trained, process for train
-                text_tokens = [
-                    self.OOV_token if t in self.OOV_words else t for t in text_tokens
-                ]
-            text = " ".join(text_tokens)
-        return text
+                tokens = [self.OOV_token if t in self.OOV_words else t for t in tokens]
 
-    @staticmethod
-    def _get_message_tokens_by_attribute(
-        message: "Message", attribute: Text
+        return tokens
+
+    def _get_processed_message_tokens_by_attribute(
+        self, message: "Message", attribute: Text = TEXT_ATTRIBUTE
     ) -> List[Text]:
-        """Get text tokens of an attribute of a message"""
+        """Get processed text of attribute of a message"""
+
+        if message.get(attribute) is None:
+            # return empty string since sklearn countvectorizer does not like None
+            # object while training and predicting
+            return [""]
+
+        tokens = self._get_message_tokens_by_attribute(message, attribute)
+        tokens = self._process_tokens(tokens, attribute)
+        tokens = self._replace_with_oov_token(tokens, attribute)
 
-        if attribute in SPACY_FEATURIZABLE_ATTRIBUTES and message.get(
-            MESSAGE_SPACY_FEATURES_NAMES[attribute]
-        ):  # if lemmatize is possible
-            tokens = [
-                t.lemma_ for t in message.get(MESSAGE_SPACY_FEATURES_NAMES[attribute])
-            ]
-        elif message.get(
-            MESSAGE_TOKENS_NAMES[attribute]
-        ):  # if directly tokens is provided
-            tokens = [t.text for t in message.get(MESSAGE_TOKENS_NAMES[attribute])]
-        else:
-            tokens = message.get(attribute).split()
         return tokens
 
     # noinspection PyPep8Naming
-    def _check_OOV_present(self, examples) -> None:
+    def _check_OOV_present(self, all_tokens: List[List[Text]]):
         """Check if an OOV word is present"""
-        if self.OOV_token and not self.OOV_words:
-            for t in examples:
-                if (
-                    t is None
-                    or self.OOV_token in t
-                    or (self.lowercase and self.OOV_token in t.lower())
-                ):
-                    return
-
-            warnings.warn(
-                f"OOV_token='{self.OOV_token}' was given, but it is not present "
-                "in the training data. All unseen words "
-                "will be ignored during prediction."
-            )
-
-    def _set_attribute_features(
-        self,
-        attribute: Text,
-        attribute_features: np.ndarray,
-        training_data: "TrainingData",
-    ):
-        """Set computed features of the attribute to corresponding message objects"""
-        for i, example in enumerate(training_data.intent_examples):
-            # create bag for each example
-            example.set(
-                MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                self._combine_with_existing_features(
-                    example,
-                    attribute_features[i],
-                    MESSAGE_VECTOR_FEATURE_NAMES[attribute],
-                ),
-            )
+        if self.OOV_token and not self.OOV_words and all_tokens:
path: rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
text: @user1:
I'd us an early exit here (to decrease indentation and make this easier to understand):
```
        if not self.OOV_token or self.OOV_words or not all_tokens:
            return

```