Repository: RasaHQ/rasa
Pull request: ConvertTokenizer, CLS token & features as sequence 
**Proposed changes**:
- Add ConveRT Tokenizer
- Remove option `use_cls_token` from all tokenizers. Tokenizers add CLS token by default now.
- Remove option `return_sequence` from featurizers. Featurizers return sequence by default. The feature vector of the CLS token contains the feature for the complete utterance. 
- Implement `train` and `process` of tokenizers in `Tokenizer` class. Subclasses just need to implement `tokenize`.

closes https://github.com/RasaHQ/rasa/issues/4978

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the documentation
- [x] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
Tokenizer for ConveRT
Implement a tokenizer for ConveRT that allows us to use embedding for ConveRT in a sequence-fashion, for example, for the `CRFEntityExtractor`. 

Problem: ConveRT tokenizes words into subwords and adds special characters. Thus, the token start and end does not match the entities. We need to work on an alignment so that the tokens from ConveRT match the entities.

Hints from issue discussion:

Commit message: refactor intent tokenization

Patch to review:
diff --git a/changelog/4978.feature.rst b/changelog/4978.feature.rst
new file mode 100644
index 000000000000..2bfa51326f8a
--- /dev/null
+++ b/changelog/4978.feature.rst
@@ -0,0 +1,3 @@
+Add ``ConveRTTokenizer``.
+
+The tokenizer should be used whenever the ``ConveRTFeaturizer`` is used.
\ No newline at end of file
diff --git a/changelog/4978.removal.rst b/changelog/4978.removal.rst
new file mode 100644
index 000000000000..609a78429f2f
--- /dev/null
+++ b/changelog/4978.removal.rst
@@ -0,0 +1,9 @@
+Remove option ``use_cls_token`` from tokenizers and option ``return_sequence`` from featurizers.
+
+By default all tokenizer add a special token (``__CLS__``) to the end of the list of tokens.
+This token will be used to capture the features of the whole utterance.
+
+The featurizers will return a matrix of size (token-length x feature-dimension) by default.
+This allows to train sequence models.
+However, the feature vector of the ``__CLS__`` token can be used to train non-sequence models.
+The corresponding classifier can decide what kind of features to use.
\ No newline at end of file
diff --git a/docs/api/custom-nlu-components.rst b/docs/api/custom-nlu-components.rst
index 947327619e78..679253d9eed3 100644
--- a/docs/api/custom-nlu-components.rst
+++ b/docs/api/custom-nlu-components.rst
@@ -36,25 +36,15 @@ methods that you should implement:
     :linenos:
 
 .. note::
-    If you create a custom tokenizer you should have the following entry in your default dict.
-
-    .. code-block:: none
-
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False,
-
-    To add the __CLS__ token you can use the function `add_cls_token()
-    <https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/tokenizers/tokenizer.py#L64/>`_.
+    If you create a custom tokenizer you should implement the methods of ``rasa.nlu.tokenizers.tokenizer.Tokenizer``.
+    The ``train`` and ``process`` methods are already implemented and you simply need to overwrite the ``tokenize``
+    method. ``train`` and ``process`` will automatically add a special token ``__CLS__`` to the end of list of tokens,
+    which is needed further down the pipeline.
 
 .. note::
-    If you create a custom featurizer you should have the following entry in your default dict.
-
-    .. code-block:: none
-
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False,
+    If you create a custom featurizer you should return a sequence of features.
+    E.g. your featurizer should return a matrix of size (token-size x feature-dimension).
+    The feature vector of the ``__CLS__`` token should contain features for the complete message.
 
 Component
 ^^^^^^^^^
diff --git a/docs/nlu/choosing-a-pipeline.rst b/docs/nlu/choosing-a-pipeline.rst
index 78b7fdeadd14..96932cc953fa 100644
--- a/docs/nlu/choosing-a-pipeline.rst
+++ b/docs/nlu/choosing-a-pipeline.rst
@@ -322,7 +322,7 @@ To use the components and configure them separately:
     language: "en"
 
     pipeline:
-    - name: "WhitespaceTokenizer"
+    - name: "ConveRTTokenizer"
     - name: "ConveRTFeaturizer"
     - name: "EmbeddingIntentClassifier"
 
diff --git a/docs/nlu/components.rst b/docs/nlu/components.rst
index 7ebebcd615e9..6ff0a23cfaf4 100644
--- a/docs/nlu/components.rst
+++ b/docs/nlu/components.rst
@@ -91,15 +91,12 @@ As those feature vectors would normally take up a lot of memory, we store them a
 Sparse features only store the values that are non zero and their positions in the vector.
 Thus, we save a lot of memroy and are able to train on larger datasets.
 
-By default all featurizers will return a matrix of length (1 x feature-dimension).
-All featurizer (except the ``ConveRTFeaturizer``) have the option to return a sequence instead.
-In case the flag ``"return_sequence"`` is set to ``True``, the returned matrix of the featurizer will have the size
-(token-length x feature-dimension).
-So, the returned matrix will have an entry for every token.
-Otherwise, the matrix will just have one entry for the complete utterance.
-If you want to use custom features for your ``CRFEntityExtractor``, you should set ``"return_sequence"`` to ``True``.
-For more details, take a look :ref:`entity-extraction-custom-features`.
-
+By default all featurizers will return a matrix of length (token-length x feature-dimension).
+So, the returned matrix will have a feature vector for every token.
+This allows us to train sequence models.
+However, the additional token at the end (e.g. ``__CLS__``) contains features for the complete utterance.
+This feature vector can be used in any non-sequence model.
+The corresponding classifier can therefore decide what kind of features to use.
 
 MitieFeaturizer
 ~~~~~~~~~~~~~~~
@@ -151,7 +148,7 @@ ConveRTFeaturizer
 :Outputs:
     nothing, used as an input to intent classifiers and response selectors that need intent features and response
     features respectively (e.g. ``EmbeddingIntentClassifier`` and ``ResponseSelector``)
-:Requires: nothing
+:Requires: :ref:`ConveRTTokenizer`
 :Type: Dense featurizer
 :Description:
     Creates features for intent classification and response selection.
@@ -166,12 +163,6 @@ ConveRTFeaturizer
         To use ``ConveRTFeaturizer`` you need to install additional tensorflow libraries (``tensorflow_text`` and
         ``tensorflow_hub``). You should do a pip install of Rasa with ``pip install rasa[convert]`` to install those.
 
-    .. warning::
-        If you set the option ``"return_sequence"`` to ``True``, Rasa will raise an error informing you that this
-        option is currently not supported. Do not use this featurizer in combination with any other featurizer that
-        has the option ``"return_sequence"`` set to ``True`` as training will fail. However, you can use this
-        featurizer with any other featurizer as long as ``"return_sequence"`` is set to ``False`` for all of them.  
-
 :Configuration:
 
     .. code-block:: yaml
@@ -622,6 +613,10 @@ Response Selector
 Tokenizers
 ----------
 
+    .. note:: All tokenizer add an additional token ``__CLS__`` to the end of the list of tokens when tokenizing
+              text and responses.
+
+
 WhitespaceTokenizer
 ~~~~~~~~~~~~~~~~~~~
 
@@ -695,6 +690,18 @@ SpacyTokenizer
     Creates tokens using the spacy tokenizer. Can be used to define
     tokens for the MITIE entity extractor.
 
+.. _ConveRTTokenizer:
+
+ConveRTTokenizer
+~~~~~~~~~~~~~~~~
+
+:Short: Tokenizer using ConveRT
+:Outputs: nothing
+:Requires: nothing
+:Description:
+    Creates tokens using the ConveRT tokenizer. Must be used whenever the ``ConveRTFeaturizer`` is used.
+
+
 
 Entity Extractors
 -----------------
@@ -831,8 +838,7 @@ CRFEntityExtractor
     neighbouring entity tags: the most likely set of tags is then calculated and returned.
     If POS features are used (pos or pos2), spaCy has to be installed. If you want to use
     additional features, such as pre-trained word embeddings, from any provided dense
-    featurizer, use ``"text_dense_features"``. Make sure to set ``"return_sequence"`` to
-    ``True`` in the corresponding featurizer.
+    featurizer, use ``"text_dense_features"``.
 :Configuration:
    .. code-block:: yaml
 
diff --git a/docs/nlu/entity-extraction.rst b/docs/nlu/entity-extraction.rst
index 966a333178ef..423c4bc6aa7d 100644
--- a/docs/nlu/entity-extraction.rst
+++ b/docs/nlu/entity-extraction.rst
@@ -157,8 +157,7 @@ Passing Custom Features to ``CRFEntityExtractor``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If you want to pass custom features, such as pre-trained word embeddings, to ``CRFEntityExtractor``, you can
-add any dense featurizer (except ``ConveRTFeaturizer``) to the pipeline before the ``CRFEntityExtractor``.
-Make sure to set ``"return_sequence"`` to ``True`` for the corresponding dense featurizer.
+add any dense featurizer to the pipeline before the ``CRFEntityExtractor``.
 ``CRFEntityExtractor`` automatically finds the additional dense features and checks if the dense features are an
 iterable of ``len(tokens)``, where each entry is a vector.
 A warning will be shown in case the check fails.
diff --git a/rasa/nlu/classifiers/embedding_intent_classifier.py b/rasa/nlu/classifiers/embedding_intent_classifier.py
index 29e62b422c3c..28fffbb0cd64 100644
--- a/rasa/nlu/classifiers/embedding_intent_classifier.py
+++ b/rasa/nlu/classifiers/embedding_intent_classifier.py
@@ -8,6 +8,7 @@
 from typing import Any, Dict, List, Optional, Text, Tuple, Union
 import warnings
 
+from rasa.nlu.featurizers.featurizer import sequence_to_sentence_features
 from rasa.nlu.classifiers import LABEL_RANKING_LENGTH
 from rasa.nlu.components import Component, any_of
 from rasa.utils import train_utils
@@ -308,6 +309,11 @@ def _extract_and_add_features(
                     f"don't coincide in '{message.text}' for attribute '{attribute}'."
                 )
 
+        if attribute != INTENT_ATTRIBUTE:
+            # Use only the CLS token vector as features
+            sparse_features = sequence_to_sentence_features(sparse_features)
+            dense_features = sequence_to_sentence_features(dense_features)
+
         return sparse_features, dense_features
 
     def _extract_labels_precomputed_features(
@@ -531,8 +537,8 @@ def _combine_sparse_dense_features(
                 dense_features.append(f)
 
         output = tf.concat(dense_features, axis=-1) * mask
-        # apply mean to convert sequence to sentence features
-        output = tf.reduce_sum(output, axis=1) / tf.reduce_sum(mask, axis=1)
+        output = tf.squeeze(output, axis=1)
+
         return output
 
     def _build_tf_train_graph(
diff --git a/rasa/nlu/classifiers/keyword_intent_classifier.py b/rasa/nlu/classifiers/keyword_intent_classifier.py
index 6d1053a04703..309ccca45378 100644
--- a/rasa/nlu/classifiers/keyword_intent_classifier.py
+++ b/rasa/nlu/classifiers/keyword_intent_classifier.py
@@ -8,6 +8,7 @@
 from rasa.nlu import utils
 from rasa.nlu.components import Component
 from rasa.nlu.training_data import Message
+from rasa.nlu.constants import INTENT_ATTRIBUTE
 
 logger = logging.getLogger(__name__)
 
@@ -27,7 +28,7 @@ class KeywordIntentClassifier(Component):
 
     """
 
-    provides = ["intent"]
+    provides = [INTENT_ATTRIBUTE]
 
     defaults = {"case_sensitive": True}
 
@@ -53,17 +54,18 @@ def train(
         for ex in training_data.training_examples:
             if (
                 ex.text in self.intent_keyword_map.keys()
-                and ex.get("intent") != self.intent_keyword_map[ex.text]
+                and ex.get(INTENT_ATTRIBUTE) != self.intent_keyword_map[ex.text]
             ):
                 duplicate_examples.add(ex.text)
                 warnings.warn(
-                    f"Keyword '{ex.text}' is a keyword of intent '{self.intent_keyword_map[ex.text]}' and of "
-                    f"intent '{ex.get('intent')}', it will be removed from the list of "
-                    f"keywords.\n"
+                    f"Keyword '{ex.text}' is a keyword of intent "
+                    f"'{self.intent_keyword_map[ex.text]}' and of "
+                    f"intent '{ex.get(INTENT_ATTRIBUTE)}', it will be removed from "
+                    f"the list of keywords.\n"
                     f"Remove (one of) the duplicates from the training data."
                 )
             else:
-                self.intent_keyword_map[ex.text] = ex.get("intent")
+                self.intent_keyword_map[ex.text] = ex.get(INTENT_ATTRIBUTE)
         for keyword in duplicate_examples:
             self.intent_keyword_map.pop(keyword)
             logger.debug(
@@ -101,13 +103,16 @@ def _validate_keyword_map(self) -> None:
 
     def process(self, message: Message, **kwargs: Any) -> None:
         intent_name = self._map_keyword_to_intent(message.text)
+
         confidence = 0.0 if intent_name is None else 1.0
         intent = {"name": intent_name, "confidence": confidence}
-        if message.get("intent") is None or intent is not None:
-            message.set("intent", intent, add_to_output=True)
+
+        if message.get(INTENT_ATTRIBUTE) is None or intent is not None:
+            message.set(INTENT_ATTRIBUTE, intent, add_to_output=True)
 
     def _map_keyword_to_intent(self, text: Text) -> Optional[Text]:
         re_flag = 0 if self.case_sensitive else re.IGNORECASE
+
         for keyword, intent in self.intent_keyword_map.items():
             if re.search(r"\b" + keyword + r"\b", text, flags=re_flag):
                 logger.debug(
@@ -115,6 +120,7 @@ def _map_keyword_to_intent(self, text: Text) -> Optional[Text]:
                     f" intent '{intent}'."
                 )
                 return intent
+
         logger.debug("KeywordClassifier did not find any keywords in the message.")
         return None
 
diff --git a/rasa/nlu/classifiers/mitie_intent_classifier.py b/rasa/nlu/classifiers/mitie_intent_classifier.py
index 7a95e8e26f8c..cf156980b30c 100644
--- a/rasa/nlu/classifiers/mitie_intent_classifier.py
+++ b/rasa/nlu/classifiers/mitie_intent_classifier.py
@@ -5,7 +5,7 @@
 from rasa.nlu.components import Component
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.model import Metadata
-from rasa.nlu.constants import TOKENS_NAMES, TEXT_ATTRIBUTE
+from rasa.nlu.constants import TOKENS_NAMES, TEXT_ATTRIBUTE, INTENT_ATTRIBUTE
 from rasa.nlu.training_data import Message, TrainingData
 
 if typing.TYPE_CHECKING:
@@ -14,7 +14,7 @@
 
 class MitieIntentClassifier(Component):
 
-    provides = ["intent"]
+    provides = [INTENT_ATTRIBUTE]
 
     requires = [TOKENS_NAMES[TEXT_ATTRIBUTE], "mitie_feature_extractor", "mitie_file"]
 
@@ -49,7 +49,7 @@ def train(
 
         for example in training_data.intent_examples:
             tokens = self._tokens_of_message(example)
-            trainer.add_labeled_text(tokens, example.get("intent"))
+            trainer.add_labeled_text(tokens, example.get(INTENT_ATTRIBUTE))
 
         if training_data.intent_examples:
             # we can not call train if there are no examples!
@@ -79,7 +79,9 @@ def process(self, message: Message, **kwargs: Any) -> None:
 
     @staticmethod
     def _tokens_of_message(message) -> List[Text]:
-        return [token.text for token in message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])]
+        tokens = [token.text for token in message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])]
+        # return tokens without CLS token
+        return tokens[:-1]
 
     @classmethod
     def load(
diff --git a/rasa/nlu/classifiers/sklearn_intent_classifier.py b/rasa/nlu/classifiers/sklearn_intent_classifier.py
index 04372bdde903..d1a332fa2134 100644
--- a/rasa/nlu/classifiers/sklearn_intent_classifier.py
+++ b/rasa/nlu/classifiers/sklearn_intent_classifier.py
@@ -12,7 +12,7 @@
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.model import Metadata
 from rasa.nlu.training_data import Message, TrainingData
-from rasa.nlu.constants import DENSE_FEATURE_NAMES, SPARSE_FEATURE_NAMES, TEXT_ATTRIBUTE
+from rasa.nlu.constants import DENSE_FEATURE_NAMES, TEXT_ATTRIBUTE
 
 logger = logging.getLogger(__name__)
 
@@ -103,6 +103,8 @@ def train(
                     for example in training_data.intent_examples
                 ]
             )
+            # reduce dimensionality
+            X = np.reshape(X, (len(X), -1))
 
             self.clf = self._create_classifier(num_threads, y)
 
diff --git a/rasa/nlu/extractors/__init__.py b/rasa/nlu/extractors/__init__.py
index 537705e481a5..d4c0e716b138 100644
--- a/rasa/nlu/extractors/__init__.py
+++ b/rasa/nlu/extractors/__init__.py
@@ -35,7 +35,7 @@ def filter_irrelevant_entities(extracted: list, requested_dimensions: set) -> li
 
     @staticmethod
     def find_entity(ent, text, tokens) -> Tuple[int, int]:
-        offsets = [token.offset for token in tokens]
+        offsets = [token.start for token in tokens]
         ends = [token.end for token in tokens]
 
         if ent["start"] not in offsets:
diff --git a/rasa/nlu/extractors/crf_entity_extractor.py b/rasa/nlu/extractors/crf_entity_extractor.py
index 99854c01d3da..d291ffa18f35 100644
--- a/rasa/nlu/extractors/crf_entity_extractor.py
+++ b/rasa/nlu/extractors/crf_entity_extractor.py
@@ -172,7 +172,7 @@ def _create_dataset(self, examples: List[Message]) -> List[List[CRFToken]]:
 
         return dataset
 
-    def _check_spacy_doc(self, message) -> None:
+    def _check_spacy_doc(self, message: Message) -> None:
         if self.pos_features and message.get(SPACY_DOCS[TEXT_ATTRIBUTE]) is None:
             raise InvalidConfigError(
                 "Could not find `spacy_doc` attribute for "
@@ -212,7 +212,7 @@ def extract_entities(self, message: Message) -> List[Dict[Text, Any]]:
         else:
             return []
 
-    def most_likely_entity(self, idx, entities) -> Tuple[Text, Any]:
+    def most_likely_entity(self, idx: int, entities: List[Any]) -> Tuple[Text, Any]:
         if len(entities) > idx:
             entity_probs = entities[idx]
         else:
@@ -243,12 +243,12 @@ def _create_entity_dict(
         confidence: float,
     ) -> Dict[Text, Any]:
         if isinstance(tokens, list):  # tokens is a list of Token
-            _start = tokens[start].offset
+            _start = tokens[start].start
             _end = tokens[end].end
             value = tokens[start].text
             value += "".join(
                 [
-                    message.text[tokens[i - 1].end : tokens[i].offset] + tokens[i].text
+                    message.text[tokens[i - 1].end : tokens[i].start] + tokens[i].text
                     for i in range(start + 1, end + 1)
                 ]
             )
@@ -275,6 +275,11 @@ def _bilou_from_label(label) -> Optional[Text]:
             return label[0].upper()
         return None
 
+    @staticmethod
+    def _tokens_without_cls(message: Message) -> List[Token]:
+        # [:-1] to remove the CLS token from the list of tokens
+        return message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])[:-1]
+
     def _find_bilou_end(self, word_idx, entities) -> Any:
         ent_word_idx = word_idx + 1
         finished = False
@@ -314,7 +319,9 @@ def _find_bilou_end(self, word_idx, entities) -> Any:
                 )
         return ent_word_idx, confidence
 
-    def _handle_bilou_label(self, word_idx: int, entities) -> Tuple[Any, Any, Any]:
+    def _handle_bilou_label(
+        self, word_idx: int, entities: List[Any]
+    ) -> Tuple[Any, Any, Any]:
         label, confidence = self.most_likely_entity(word_idx, entities)
         entity_label = self._entity_from_label(label)
 
@@ -336,7 +343,7 @@ def _from_crf_to_json(
         if self.pos_features:
             tokens = message.get(SPACY_DOCS[TEXT_ATTRIBUTE])
         else:
-            tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+            tokens = self._tokens_without_cls(message)
 
         if len(tokens) != len(entities):
             raise Exception(
@@ -373,7 +380,7 @@ def _convert_bilou_tagging_to_entity_result(
         return json_ents
 
     def _convert_simple_tagging_to_entity_result(
-        self, tokens, entities
+        self, tokens: List[Union[Token, Any]], entities: List[Any]
     ) -> List[Dict[Text, Any]]:
         json_ents = []
 
@@ -381,11 +388,11 @@ def _convert_simple_tagging_to_entity_result(
             entity_label, confidence = self.most_likely_entity(word_idx, entities)
             word = tokens[word_idx]
             if entity_label != "O":
-                if self.pos_features:
+                if self.pos_features and not isinstance(word, Token):
                     start = word.idx
                     end = word.idx + len(word)
                 else:
-                    start = word.offset
+                    start = word.start
                     end = word.end
                 ent = {
                     "start": start,
@@ -501,7 +508,7 @@ def _from_json_to_crf(
             gold = GoldParse(doc_or_tokens, entities=entity_offsets)
             ents = [l[5] for l in gold.orig_annot]
         else:
-            doc_or_tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+            doc_or_tokens = self._tokens_without_cls(message)
             ents = self._bilou_tags_from_offsets(doc_or_tokens, entity_offsets)
 
         # collect badly annotated examples
@@ -533,7 +540,7 @@ def _from_json_to_crf(
     @staticmethod
     def _bilou_tags_from_offsets(tokens, entities, missing: Text = "O") -> List[Text]:
         # From spacy.spacy.GoldParse, under MIT License
-        starts = {token.offset: i for i, token in enumerate(tokens)}
+        starts = {token.start: i for i, token in enumerate(tokens)}
         ends = {token.end: i for i, token in enumerate(tokens)}
         bilou = ["-" for _ in tokens]
         # Handle entity cases
@@ -555,7 +562,7 @@ def _bilou_tags_from_offsets(tokens, entities, missing: Text = "O") -> List[Text
             for i in range(start_char, end_char):
                 entity_chars.add(i)
         for n, token in enumerate(tokens):
-            for i in range(token.offset, token.end):
+            for i in range(token.start, token.end):
                 if i in entity_chars:
                     break
             else:
@@ -564,14 +571,14 @@ def _bilou_tags_from_offsets(tokens, entities, missing: Text = "O") -> List[Text
         return bilou
 
     @staticmethod
-    def __pattern_of_token(message, i):
+    def __pattern_of_token(message: Message, i: int) -> Dict:
         if message.get(TOKENS_NAMES[TEXT_ATTRIBUTE]) is not None:
             return message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])[i].get("pattern", {})
         else:
             return {}
 
     @staticmethod
-    def __tag_of_token(token):
+    def __tag_of_token(token: Any) -> Text:
         if spacy.about.__version__ > "2" and token._.has("tag"):
             return token._.get("tag")
         else:
@@ -615,7 +622,7 @@ def _from_text_to_crf(
         if self.pos_features:
             tokens = message.get(SPACY_DOCS[TEXT_ATTRIBUTE])
         else:
-            tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+            tokens = self._tokens_without_cls(message)
 
         text_dense_features = self.__get_dense_features(message)
 
diff --git a/rasa/nlu/extractors/mitie_entity_extractor.py b/rasa/nlu/extractors/mitie_entity_extractor.py
index 671b312e499f..7bd00b0fb799 100644
--- a/rasa/nlu/extractors/mitie_entity_extractor.py
+++ b/rasa/nlu/extractors/mitie_entity_extractor.py
@@ -33,6 +33,11 @@ def __init__(self, component_config: Optional[Dict[Text, Any]] = None, ner=None)
     def required_packages(cls) -> List[Text]:
         return ["mitie"]
 
+    @staticmethod
+    def _tokens_without_cls(message: Message) -> List[Token]:
+        # [:-1] to remove the CLS token from the list of tokens
+        return message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])[:-1]
+
     def extract_entities(
         self, text: Text, tokens: List[Token], feature_extractor
     ) -> List[Dict[Text, Any]]:
@@ -42,7 +47,7 @@ def extract_entities(
             entities = self.ner.extract_entities(tokens_strs, feature_extractor)
             for e in entities:
                 if len(e[0]):
-                    start = tokens[e[0][0]].offset
+                    start = tokens[e[0][0]].start
                     end = tokens[e[0][-1]].end
 
                     ents.append(
@@ -89,11 +94,11 @@ def train(
         if found_one_entity:
             self.ner = trainer.train()
 
-    def _prepare_mitie_sample(self, training_example) -> Any:
+    def _prepare_mitie_sample(self, training_example: Message) -> Any:
         import mitie
 
         text = training_example.text
-        tokens = training_example.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+        tokens = self._tokens_without_cls(training_example)
         sample = mitie.ner_training_instance([t.text for t in tokens])
         for ent in training_example.get(ENTITIES_ATTRIBUTE, []):
             try:
@@ -125,9 +130,7 @@ def process(self, message: Message, **kwargs: Any) -> None:
             )
 
         ents = self.extract_entities(
-            message.text,
-            message.get(TOKENS_NAMES[TEXT_ATTRIBUTE]),
-            mitie_feature_extractor,
+            message.text, self._tokens_without_cls(message), mitie_feature_extractor
         )
         extracted = self.add_extractor_name(ents)
         message.set(
diff --git a/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
index 2cdb66dbef5e..1675d99098b1 100644
--- a/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
+++ b/rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py
@@ -1,11 +1,14 @@
 import logging
 import warnings
+from typing import Any, Dict, List, Optional, Text
+
+from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu.featurizers.featurizer import Featurizer
-from typing import Any, Dict, List, Optional, Text, Tuple
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.training_data import Message, TrainingData
 from rasa.nlu.constants import (
     TEXT_ATTRIBUTE,
+    TOKENS_NAMES,
     DENSE_FEATURE_NAMES,
     DENSE_FEATURIZABLE_ATTRIBUTES,
 )
@@ -21,12 +24,7 @@ class ConveRTFeaturizer(Featurizer):
         DENSE_FEATURE_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
     ]
 
-    defaults = {
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False
-    }
+    requires = [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
 
     def _load_model(self) -> None:
 
@@ -42,7 +40,10 @@ def _load_model(self) -> None:
             self.module = tfhub.Module(model_url)
 
             self.text_placeholder = tf.placeholder(dtype=tf.string, shape=[None])
-            self.encoding_tensor = self.module(self.text_placeholder)
+            self.sentence_encoding_tensor = self.module(self.text_placeholder)
+            self.sequence_encoding_tensor = self.module(
+                self.text_placeholder, signature="encode_sequence", as_dict=True
+            )
             self.session.run(tf.tables_initializer())
             self.session.run(tf.global_variables_initializer())
 
@@ -52,18 +53,6 @@ def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
 
         self._load_model()
 
-        self.return_sequence = self.component_config["return_sequence"]
-
-        if self.return_sequence:
-            raise NotImplementedError(
-                f"ConveRTFeaturizer always returns a feature vector of size "
-                f"(1 x feature-dimensions). It cannot return a proper sequence "
-                f"right now. ConveRTFeaturizer can only be used "
-                f"with 'return_sequence' set to False. Also, any other featurizer "
-                f"used next to ConveRTFeaturizer should have the flag "
-                f"'return_sequence' set to False."
-            )
-
     @classmethod
     def required_packages(cls) -> List[Text]:
         return ["tensorflow_text", "tensorflow_hub"]
@@ -72,19 +61,93 @@ def _compute_features(
         self, batch_examples: List[Message], attribute: Text = TEXT_ATTRIBUTE
     ) -> np.ndarray:
 
+        sentence_encodings = self._compute_sentence_encodings(batch_examples, attribute)
+
+        return self._compute_sequence_encodings(
+            batch_examples, sentence_encodings, attribute
+        )
+
+    def _compute_sentence_encodings(
+        self, batch_examples: List[Message], attribute: Text = TEXT_ATTRIBUTE
+    ) -> np.ndarray:
         # Get text for attribute of each example
         batch_attribute_text = [ex.get(attribute) for ex in batch_examples]
+        sentence_encodings = self._sentence_encoding_of_text(batch_attribute_text)
 
-        batch_features = self._run_model_on_text(batch_attribute_text)
+        # convert them to a sequence of 1
+        return np.reshape(sentence_encodings, (len(batch_examples), 1, -1))
 
-        return batch_features
+    def _compute_sequence_encodings(
+        self,
+        batch_examples: List[Message],
+        sentence_encodings: np.ndarray,
+        attribute: Text = TEXT_ATTRIBUTE,
+    ) -> np.ndarray:
+        list_of_tokens = [
+            example.get(TOKENS_NAMES[attribute]) for example in batch_examples
+        ]
+
+        # remove CLS token from list of tokens
+        list_of_tokens = [sent_tokens[:-1] for sent_tokens in list_of_tokens]
+
+        number_of_tokens_in_sentence = [
+            len(sent_tokens) for sent_tokens in list_of_tokens
+        ]
+
+        # join the tokens to get a clean text to ensure the sequence length of
+        # the returned embeddings from ConveRT matches the length of the tokens
+        tokenized_texts = self._tokens_to_text(list_of_tokens)
+        sequence_encodings = self._sequence_encoding_of_text(tokenized_texts)
+
+        final_embeddings = []
+        for index in range(len(batch_examples)):
+            sequence_length = number_of_tokens_in_sentence[index]
+            sequence_encoding = sequence_encodings[index][:sequence_length]
+            sentence_encoding = sentence_encodings[index]
+
+            # tile sequence encoding to duplicate as sentence encodings have size
+            # 1024 and sequence encodings only have a dimensionality of 512
+            sequence_encoding = np.tile(sequence_encoding, (1, 2))
+            # add sentence encoding to the end (position of cls token)
+            sequence_encoding = np.concatenate(
+                [sequence_encoding, sentence_encoding], axis=0
+            )
+
+            final_embeddings.append(sequence_encoding)
+
+        return np.array(final_embeddings)
+
+    @staticmethod
+    def _tokens_to_text(list_of_tokens: List[List[Token]]) -> List[Text]:
+        """Convert list of tokens to text.
 
-    def _run_model_on_text(self, batch: List[Text]) -> np.ndarray:
+        Add a whitespace between two tokens if the end value of the first tokens is
+        not the same as the end value of the second token."""
 
+        texts = []
+        for tokens in list_of_tokens:
+            text = ""
+            offset = 0
+            for token in tokens:
+                if offset != token.start:
+                    text += " "
+                text += token.text
+
+                offset = token.end
+            texts.append(text)
+
+        return texts
+
+    def _sentence_encoding_of_text(self, batch: List[Text]) -> np.ndarray:
         return self.session.run(
-            self.encoding_tensor, feed_dict={self.text_placeholder: batch}
+            self.sentence_encoding_tensor, feed_dict={self.text_placeholder: batch}
         )
 
+    def _sequence_encoding_of_text(self, batch: List[Text]) -> np.ndarray:
+        return self.session.run(
+            self.sequence_encoding_tensor, feed_dict={self.text_placeholder: batch}
+        )["sequence_encoding"]
+
     def train(
         self,
         training_data: TrainingData,
@@ -126,9 +189,7 @@ def train(
                     ex.set(
                         DENSE_FEATURE_NAMES[attribute],
                         self._combine_with_existing_dense_features(
-                            ex,
-                            np.expand_dims(batch_features[index], axis=0),
-                            DENSE_FEATURE_NAMES[attribute],
+                            ex, batch_features[index], DENSE_FEATURE_NAMES[attribute]
                         ),
                     )
 
@@ -136,12 +197,10 @@ def train(
 
     def process(self, message: Message, **kwargs: Any) -> None:
 
-        feats = self._compute_features([message])[0]
+        features = self._compute_features([message])[0]
         message.set(
             DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
             self._combine_with_existing_dense_features(
-                message,
-                np.expand_dims(feats, axis=0),
-                DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE],
+                message, features, DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
             ),
         )
diff --git a/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
index daefc82863c4..8fd4ad68a9db 100644
--- a/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
+++ b/rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py
@@ -1,6 +1,6 @@
 import numpy as np
 import typing
-from typing import Any, List, Text, Dict, Optional
+from typing import Any, List, Text
 
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.featurizers.featurizer import Featurizer
@@ -15,7 +15,7 @@
     TOKENS_NAMES,
     MESSAGE_ATTRIBUTES,
     DENSE_FEATURE_NAMES,
-    CLS_TOKEN,
+    DENSE_FEATURIZABLE_ATTRIBUTES,
 )
 
 
@@ -27,19 +27,6 @@ class MitieFeaturizer(Featurizer):
         "mitie_feature_extractor"
     ]
 
-    defaults = {
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False
-    }
-
-    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
-
-        super().__init__(component_config)
-
-        self.return_sequence = self.component_config["return_sequence"]
-
     @classmethod
     def required_packages(cls) -> List[Text]:
         return ["mitie", "numpy"]
@@ -57,7 +44,7 @@ def train(
 
         mitie_feature_extractor = self._mitie_feature_extractor(**kwargs)
         for example in training_data.intent_examples:
-            for attribute in MESSAGE_ATTRIBUTES:
+            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
                 self.process_training_example(
                     example, attribute, mitie_feature_extractor
                 )
@@ -107,11 +94,9 @@ def features_for_tokens(
         tokens: List[Token],
         feature_extractor: "mitie.total_word_feature_extractor",
     ) -> np.ndarray:
-        cls_token_used = tokens[-1].text == CLS_TOKEN if tokens else False
 
-        tokens_without_cls = tokens
-        if cls_token_used:
-            tokens_without_cls = tokens[:-1]
+        # remove CLS token from tokens
+        tokens_without_cls = tokens[:-1]
 
         # calculate features
         features = []
@@ -119,12 +104,7 @@ def features_for_tokens(
             features.append(feature_extractor.get_feature_vector(token.text))
         features = np.array(features)
 
-        if cls_token_used and self.return_sequence:
-            # cls token is used, need to append a vector
-            cls_token_vec = np.mean(features, axis=0, keepdims=True)
-            features = np.concatenate([features, cls_token_vec])
-
-        if not self.return_sequence:
-            features = np.mean(features, axis=0, keepdims=True)
+        cls_token_vec = np.mean(features, axis=0, keepdims=True)
+        features = np.concatenate([features, cls_token_vec])
 
         return features
diff --git a/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py b/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
index f6a6845da080..07ba118bd65f 100644
--- a/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
+++ b/rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py
@@ -1,6 +1,6 @@
 import numpy as np
 import typing
-from typing import Any, Optional, Dict, Text
+from typing import Any, Optional, Text
 
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.featurizers.featurizer import Featurizer
@@ -15,7 +15,6 @@
     DENSE_FEATURE_NAMES,
     DENSE_FEATURIZABLE_ATTRIBUTES,
     TOKENS_NAMES,
-    CLS_TOKEN,
 )
 
 
@@ -29,24 +28,9 @@ class SpacyFeaturizer(Featurizer):
         SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES
     ] + [TOKENS_NAMES[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
 
-    defaults = {
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False
-    }
-
-    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
-
-        super().__init__(component_config)
-
-        self.return_sequence = self.component_config["return_sequence"]
-
     def _features_for_doc(self, doc: "Doc") -> np.ndarray:
         """Feature vector for a single document / sentence / tokens."""
-        if self.return_sequence:
-            return np.array([t.vector for t in doc])
-        return np.array([doc.vector])
+        return np.array([t.vector for t in doc])
 
     def train(
         self,
@@ -71,16 +55,12 @@ def _set_spacy_features(self, message: Message, attribute: Text = TEXT_ATTRIBUTE
         """Adds the spacy word vectors to the messages features."""
 
         message_attribute_doc = self.get_doc(message, attribute)
-        tokens = message.get(TOKENS_NAMES[attribute])
-        cls_token_used = tokens[-1].text == CLS_TOKEN if tokens else False
 
         if message_attribute_doc is not None:
             features = self._features_for_doc(message_attribute_doc)
 
-            if cls_token_used and self.return_sequence:
-                # cls token is used, need to append a vector
-                cls_token_vec = np.mean(features, axis=0, keepdims=True)
-                features = np.concatenate([features, cls_token_vec])
+            cls_token_vec = np.mean(features, axis=0, keepdims=True)
+            features = np.concatenate([features, cls_token_vec])
 
             features = self._combine_with_existing_dense_features(
                 message, features, DENSE_FEATURE_NAMES[attribute]
diff --git a/rasa/nlu/featurizers/featurizer.py b/rasa/nlu/featurizers/featurizer.py
index d28b392b1624..63f28465dc58 100644
--- a/rasa/nlu/featurizers/featurizer.py
+++ b/rasa/nlu/featurizers/featurizer.py
@@ -10,28 +10,21 @@
 def sequence_to_sentence_features(
     features: Union[np.ndarray, scipy.sparse.spmatrix]
 ) -> Optional[Union[np.ndarray, scipy.sparse.spmatrix]]:
+    """Extract the CLS token vector as sentence features.
+
+    Features is a sequence. The last token is the CLS token. The feature vector of
+    this token contains the sentence features."""
+
     if features is None:
         return None
 
     if isinstance(features, scipy.sparse.spmatrix):
-        return scipy.sparse.coo_matrix(features.sum(axis=0))
+        return scipy.sparse.coo_matrix(features.tocsr()[-1])
 
-    return np.mean(features, axis=0)
+    return np.expand_dims(features[-1], axis=0)
 
 
 class Featurizer(Component):
-    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
-        super(Featurizer, self).__init__(component_config)
-
-        try:
-            self.return_sequence = self.component_config["return_sequence"]
-        except KeyError:
-            warnings.warn(
-                "No default value for 'return_sequence' was set. Please, "
-                "add it to the default dict of the featurizer and set it to 'False'."
-            )
-            self.return_sequence = False
-
     @staticmethod
     def _combine_with_existing_dense_features(
         message: Message,
diff --git a/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
index e5f58acf00cd..1d27146be482 100644
--- a/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py
@@ -17,6 +17,7 @@
     SPARSE_FEATURE_NAMES,
     INTENT_ATTRIBUTE,
     DENSE_FEATURIZABLE_ATTRIBUTES,
+    RESPONSE_ATTRIBUTE,
 )
 
 logger = logging.getLogger(__name__)
@@ -72,10 +73,6 @@ class CountVectorsFeaturizer(Featurizer):
         # will be converted to lowercase if lowercase is True
         "OOV_token": None,  # string or None
         "OOV_words": [],  # string or list of strings
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False,
     }
 
     @classmethod
@@ -116,9 +113,6 @@ def _load_count_vect_params(self) -> None:
         # if convert all characters to lowercase
         self.lowercase = self.component_config["lowercase"]
 
-        # whether to return a sequence or not
-        self.return_sequence = self.component_config["return_sequence"]
-
     # noinspection PyPep8Naming
     def _load_OOV_params(self) -> None:
         self.OOV_token = self.component_config["OOV_token"]
@@ -328,9 +322,11 @@ def _convert_attribute_tokens_to_texts(
         attribute_texts = {}
 
         for attribute in attribute_tokens.keys():
-            attribute_texts[attribute] = [
-                " ".join(tokens) for tokens in attribute_tokens[attribute]
-            ]
+            list_of_tokens = attribute_tokens[attribute]
+            if attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]:
+                # vocabulary should not contain CLS token
+                list_of_tokens = [tokens[:-1] for tokens in list_of_tokens]
+            attribute_texts[attribute] = [" ".join(tokens) for tokens in list_of_tokens]
 
         return attribute_texts
 
@@ -412,12 +408,22 @@ def _create_sequence(
             # [n_samples, n_features]
             # set input to list of tokens if sequence should be returned
             # otherwise join all tokens to a single string and pass that as a list
-            input = tokens
-            if not self.return_sequence:
-                input = [" ".join(tokens)]
+            tokens_without_cls = tokens
+            if attribute in [TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE]:
+                tokens_without_cls = tokens[:-1]
+
+            seq_vec = self.vectorizers[attribute].transform(tokens_without_cls)
+            seq_vec.sort_indices()
+
+            if attribute in [TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE]:
+                tokens_text = [" ".join(tokens_without_cls)]
+                cls_vec = self.vectorizers[attribute].transform(tokens_text)
+                cls_vec.sort_indices()
+
+                x = scipy.sparse.vstack([seq_vec, cls_vec])
+            else:
+                x = seq_vec
 
-            x = self.vectorizers[attribute].transform(input)
-            x.sort_indices()
             X.append(x.tocoo())
 
         return X
diff --git a/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
index d6eadfc90a0b..60badfb41dc1 100644
--- a/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/ngram_featurizer.py
@@ -1,5 +1,4 @@
 import logging
-import warnings
 
 from typing import Any, Dict, Optional, Text
 
diff --git a/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
index 087e15e1bf58..d9205cac056f 100644
--- a/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
+++ b/rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py
@@ -18,6 +18,7 @@
     TEXT_ATTRIBUTE,
     RESPONSE_ATTRIBUTE,
     SPARSE_FEATURE_NAMES,
+    CLS_TOKEN,
 )
 from rasa.constants import DOCS_BASE_URL
 
@@ -33,13 +34,6 @@ class RegexFeaturizer(Featurizer):
 
     requires = [TOKENS_NAMES[TEXT_ATTRIBUTE]]
 
-    defaults = {
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False
-    }
-
     def __init__(
         self,
         component_config: Optional[Dict[Text, Any]] = None,
@@ -53,8 +47,6 @@ def __init__(
         lookup_tables = lookup_tables or []
         self._add_lookup_table_regexes(lookup_tables)
 
-        self.return_sequence = self.component_config["return_sequence"]
-
     def train(
         self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
     ) -> None:
@@ -96,11 +88,7 @@ def _features_for_patterns(
         message is tokenized, the function will mark all tokens with a dict
         relating the name of the regex to whether it was matched."""
         tokens = message.get(TOKENS_NAMES[attribute], [])
-
-        if self.return_sequence:
-            seq_length = len(tokens)
-        else:
-            seq_length = 1
+        seq_length = len(tokens)
 
         vec = np.zeros([seq_length, len(self.known_patterns)])
 
@@ -112,15 +100,19 @@ def _features_for_patterns(
                 patterns = t.get("pattern", default={})
                 patterns[pattern["name"]] = False
 
-                if self.return_sequence:
-                    seq_index = token_index
-                else:
-                    seq_index = 0
+                if t.text == CLS_TOKEN:
+                    # make sure to set all patterns for the CLS token to False
+                    # the attribute patterns is needed later on and in the tests
+                    t.set("pattern", patterns)
+                    continue
 
                 for match in matches:
-                    if t.offset < match.end() and t.end > match.start():
+                    if t.start < match.end() and t.end > match.start():
                         patterns[pattern["name"]] = True
-                        vec[seq_index][pattern_index] = 1.0
+                        vec[token_index][pattern_index] = 1.0
+                        if attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]:
+                            # CLS token vector should contain all patterns
+                            vec[-1][pattern_index] = 1.0
 
                 t.set("pattern", patterns)
 
diff --git a/rasa/nlu/registry.py b/rasa/nlu/registry.py
index 34a17a930bcc..04c7e88df940 100644
--- a/rasa/nlu/registry.py
+++ b/rasa/nlu/registry.py
@@ -29,6 +29,7 @@
 from rasa.nlu.model import Metadata
 from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
 from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa.nlu.tokenizers.convert_tokenizer import ConveRTTokenizer
 from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
 from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 from rasa.nlu.utils.mitie_utils import MitieNLP
@@ -53,6 +54,7 @@
     MitieTokenizer,
     SpacyTokenizer,
     WhitespaceTokenizer,
+    ConveRTTokenizer,
     JiebaTokenizer,
     # extractors
     SpacyEntityExtractor,
@@ -131,7 +133,7 @@
         {"name": "EmbeddingIntentClassifier"},
     ],
     "pretrained_embeddings_convert": [
-        {"name": "WhitespaceTokenizer"},
+        {"name": "ConveRTTokenizer"},
         {"name": "ConveRTFeaturizer"},
         {"name": "EmbeddingIntentClassifier"},
     ],
diff --git a/rasa/nlu/selectors/embedding_response_selector.py b/rasa/nlu/selectors/embedding_response_selector.py
index 98733bd60ed8..5c34b697e44d 100644
--- a/rasa/nlu/selectors/embedding_response_selector.py
+++ b/rasa/nlu/selectors/embedding_response_selector.py
@@ -2,13 +2,15 @@
 import typing
 from typing import Any, Dict, Text
 
+from rasa.nlu.components import any_of
 from rasa.nlu.classifiers.embedding_intent_classifier import EmbeddingIntentClassifier
 from rasa.nlu.constants import (
     RESPONSE_ATTRIBUTE,
-    TEXT_ATTRIBUTE,
-    SPARSE_FEATURE_NAMES,
     RESPONSE_SELECTOR_PROPERTY_NAME,
     DEFAULT_OPEN_UTTERANCE_TYPE,
+    DENSE_FEATURE_NAMES,
+    TEXT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
 )
 
 logger = logging.getLogger(__name__)
@@ -41,9 +43,17 @@ class ResponseSelector(EmbeddingIntentClassifier):
     and additional hidden layers are added together with dropout.
     """
 
-    provides = ["response", "response_ranking"]
-
-    requires = []
+    provides = [RESPONSE_ATTRIBUTE, "response_ranking"]
+
+    requires = [
+        any_of(
+            DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE], SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]
+        ),
+        any_of(
+            DENSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE],
+            SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE],
+        ),
+    ]
 
     # default properties (DOC MARKER - don't remove)
     defaults = {
diff --git a/rasa/nlu/test.py b/rasa/nlu/test.py
index a6cfee1fcb30..5dc402aa6bbc 100644
--- a/rasa/nlu/test.py
+++ b/rasa/nlu/test.py
@@ -761,7 +761,7 @@ def does_token_cross_borders(token: Token, entity: Dict) -> bool:
 def determine_intersection(token: Token, entity: Dict) -> int:
     """Calculates how many characters a given token and entity share."""
 
-    pos_token = set(range(token.offset, token.end))
+    pos_token = set(range(token.start, token.end))
     pos_entity = set(range(entity["start"], entity["end"]))
     return len(pos_token.intersection(pos_entity))
 
@@ -803,7 +803,7 @@ def find_intersecting_entites(token: Token, entities: List[Dict]) -> List[Dict]:
             logger.debug(
                 "Token boundary error for token {}({}, {}) "
                 "and entity {}"
-                "".format(token.text, token.offset, token.end, e)
+                "".format(token.text, token.start, token.end, e)
             )
     return candidates
 
diff --git a/rasa/nlu/tokenizers/convert_tokenizer.py b/rasa/nlu/tokenizers/convert_tokenizer.py
new file mode 100644
index 000000000000..8b61624c1c6a
--- /dev/null
+++ b/rasa/nlu/tokenizers/convert_tokenizer.py
@@ -0,0 +1,124 @@
+from typing import Any, Dict, List, Text
+
+from rasa.nlu.tokenizers.tokenizer import Token
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+from rasa.nlu.training_data import Message
+from rasa.nlu.constants import MESSAGE_ATTRIBUTES, TOKENS_NAMES
+import tensorflow as tf
+
+
+class ConveRTTokenizer(WhitespaceTokenizer):
+
+    provides = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
+
+    defaults = {
+        # Flag to check whether to split intents
+        "intent_tokenization_flag": False,
+        # Symbol on which intent should be split
+        "intent_split_symbol": "_",
+        # Text will be tokenized with case sensitive as default
+        "case_sensitive": True,
+    }
+
+    def __init__(self, component_config: Dict[Text, Any] = None) -> None:
+        """Construct a new tokenizer using the WhitespaceTokenizer framework."""
+
+        super().__init__(component_config)
+
+        self._load_tokenizer_params()
+
+    def _load_tokenizer_params(self):
+
+        # needed to load the ConveRT model
+        import tensorflow_text
+        import tensorflow_hub as tfhub
+
+        self.graph = tf.Graph()
+        model_url = "http://models.poly-ai.com/convert/v1/model.tar.gz"
+
+        with self.graph.as_default():
+            self.session = tf.Session()
+            self.module = tfhub.Module(model_url)
+
+            self.text_placeholder = tf.placeholder(dtype=tf.string, shape=[None])
+            self.tokenized = self.module(self.text_placeholder, signature="tokenize")
+
+            self.session.run(tf.tables_initializer())
+            self.session.run(tf.global_variables_initializer())
+
+    def _tokenize(self, sentence: Text) -> Any:
+        return self.session.run(
+            self.tokenized, feed_dict={self.text_placeholder: [sentence]}
+        )
+
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        """Tokenize the text using the ConveRT model.
+
+        ConveRT adds a special char in front of (some) words and splits words into
+        sub-words. To ensure the entity start and end values matches the token values,
+        tokenize the text first using the whitespace tokenizer. If individual tokens
+        are split up into multiple tokens, we make sure that the start end end value
+        of the first and last respective tokens stay the same.
+        """
+
+        # perform whitespace tokenization
+        tokens_in = super().tokenize(message, attribute)
+
+        tokens_out = []
+
+        for token in tokens_in:
+            token_start, token_end, token_text = token.start, token.end, token.text
+
+            # use ConveRT model to tokenize the text
+            split_token_strings = self._tokenize(token_text)[0]
+
+            # clean tokens (remove special chars and empty tokens)
+            split_token_strings = self._clean_tokens(split_token_strings)
+
+            _aligned_tokens = self._align_tokens(
+                split_token_strings, token_end, token_start
+            )
+            tokens_out += _aligned_tokens
+
+        return tokens_out
+
+    def _clean_tokens(self, tokens: List[bytes]):
+        """Encode tokens and remove special char added by ConveRT."""
+
+        tokens = [string.decode("utf-8").replace("﹏", "") for string in tokens]
+        return [string for string in tokens if string]
+
+    def _align_tokens(self, tokens_in: List[Text], token_end: int, token_start: int):
+        """Align sub-tokens of ConveRT with tokens return by the WhitespaceTokenizer.
+
+        As ConveRT might split a single word into multiple tokens, we need to make
+        sure that the start and end value of first and last sub-token matches the
+        start and end value of the token return by the WhitespaceTokenizer as the
+        entities are using those start and end values.
+        """
+
+        tokens_out = []
+
+        current_token_offset = token_start
+
+        for index, string in enumerate(tokens_in):
+            if index == 0:
+                if index == len(tokens_in) - 1:
+                    s_token_end = token_end
+                else:
+                    s_token_end = current_token_offset + len(string)
+                tokens_out.append(Token(string, token_start, end=s_token_end))
+            elif index == len(tokens_in) - 1:
+                tokens_out.append(Token(string, current_token_offset, end=token_end))
+            else:
+                tokens_out.append(
+                    Token(
+                        string,
+                        current_token_offset,
+                        end=current_token_offset + len(string),
+                    )
+                )
+
+            current_token_offset += len(string)
+
+        return tokens_out
diff --git a/rasa/nlu/tokenizers/jieba_tokenizer.py b/rasa/nlu/tokenizers/jieba_tokenizer.py
index 73e6e0541ab3..6cf2af11f45e 100644
--- a/rasa/nlu/tokenizers/jieba_tokenizer.py
+++ b/rasa/nlu/tokenizers/jieba_tokenizer.py
@@ -6,16 +6,10 @@
 from typing import Any, Dict, List, Optional, Text
 
 from rasa.nlu.components import Component
-from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
-from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.training_data import Message
 
-from rasa.nlu.constants import (
-    INTENT_ATTRIBUTE,
-    TEXT_ATTRIBUTE,
-    TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-)
+from rasa.nlu.constants import TOKENS_NAMES, MESSAGE_ATTRIBUTES
 
 logger = logging.getLogger(__name__)
 
@@ -36,8 +30,6 @@ class JiebaTokenizer(Tokenizer):
         "intent_tokenization_flag": False,
         # Symbol on which intent should be split
         "intent_split_symbol": "_",
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False,
     }  # default don't load custom dictionary
 
     def __init__(self, component_config: Dict[Text, Any] = None) -> None:
@@ -48,14 +40,6 @@ def __init__(self, component_config: Dict[Text, Any] = None) -> None:
         # path to dictionary file or None
         self.dictionary_path = self.component_config.get("dictionary_path")
 
-        # flag to check whether to split intents
-        self.intent_tokenization_flag = self.component_config.get(
-            "intent_tokenization_flag"
-        )
-
-        # symbol to split intents on
-        self.intent_split_symbol = self.component_config.get("intent_split_symbol")
-
         # load dictionary
         if self.dictionary_path is not None:
             self.load_custom_dictionary(self.dictionary_path)
@@ -79,42 +63,14 @@ def load_custom_dictionary(path: Text) -> None:
             logger.info(f"Loading Jieba User Dictionary at {jieba_userdict}")
             jieba.load_userdict(jieba_userdict)
 
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-
-        for example in training_data.training_examples:
-
-            for attribute in MESSAGE_ATTRIBUTES:
-
-                if example.get(attribute) is not None:
-                    example.set(
-                        TOKENS_NAMES[attribute],
-                        self.tokenize(example.get(attribute), attribute),
-                    )
-
-    def process(self, message: Message, **kwargs: Any) -> None:
-
-        message.set(
-            TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text, TEXT_ATTRIBUTE)
-        )
-
-    def preprocess_text(self, text: Text, attribute: Text) -> Text:
-
-        if attribute == INTENT_ATTRIBUTE and self.intent_tokenization_flag:
-            return " ".join(text.split(self.intent_split_symbol))
-        else:
-            return text
-
-    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
         import jieba
 
-        text = self.preprocess_text(text, attribute)
+        text = message.get(attribute)
+
         tokenized = jieba.tokenize(text)
         tokens = [Token(word, start) for (word, start, end) in tokenized]
 
-        self.add_cls_token(tokens, attribute)
-
         return tokens
 
     @classmethod
diff --git a/rasa/nlu/tokenizers/mitie_tokenizer.py b/rasa/nlu/tokenizers/mitie_tokenizer.py
index d775bddc0449..1a9e4d34c980 100644
--- a/rasa/nlu/tokenizers/mitie_tokenizer.py
+++ b/rasa/nlu/tokenizers/mitie_tokenizer.py
@@ -1,10 +1,9 @@
-from typing import Any, List, Text
+from typing import List, Text
 
-from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
-from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.training_data import Message
 
-from rasa.nlu.constants import TEXT_ATTRIBUTE, TOKENS_NAMES, MESSAGE_ATTRIBUTES
+from rasa.nlu.constants import TOKENS_NAMES, MESSAGE_ATTRIBUTES
 from rasa.utils.io import DEFAULT_ENCODING
 
 
@@ -13,31 +12,29 @@ class MitieTokenizer(Tokenizer):
     provides = [TOKENS_NAMES[attribute] for attribute in MESSAGE_ATTRIBUTES]
 
     defaults = {
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False
+        # Flag to check whether to split intents
+        "intent_tokenization_flag": False,
+        # Symbol on which intent should be split
+        "intent_split_symbol": "_",
     }
 
     @classmethod
     def required_packages(cls) -> List[Text]:
         return ["mitie"]
 
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-
-        for example in training_data.training_examples:
-
-            for attribute in MESSAGE_ATTRIBUTES:
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        import mitie
 
-                if example.get(attribute) is not None:
-                    example.set(
-                        TOKENS_NAMES[attribute],
-                        self.tokenize(example.get(attribute), attribute),
-                    )
+        text = message.get(attribute)
 
-    def process(self, message: Message, **kwargs: Any) -> None:
+        encoded_sentence = text.encode(DEFAULT_ENCODING)
+        tokenized = mitie.tokenize_with_offsets(encoded_sentence)
+        tokens = [
+            self._token_from_offset(token, offset, encoded_sentence)
+            for token, offset in tokenized
+        ]
 
-        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text))
+        return tokens
 
     def _token_from_offset(
         self, text: bytes, offset: int, encoded_sentence: bytes
@@ -47,20 +44,6 @@ def _token_from_offset(
             self._byte_to_char_offset(encoded_sentence, offset),
         )
 
-    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
-        import mitie
-
-        encoded_sentence = text.encode(DEFAULT_ENCODING)
-        tokenized = mitie.tokenize_with_offsets(encoded_sentence)
-        tokens = [
-            self._token_from_offset(token, offset, encoded_sentence)
-            for token, offset in tokenized
-        ]
-
-        self.add_cls_token(tokens, attribute)
-
-        return tokens
-
     @staticmethod
     def _byte_to_char_offset(text: bytes, byte_offset: int) -> int:
         return len(text[:byte_offset].decode(DEFAULT_ENCODING))
diff --git a/rasa/nlu/tokenizers/spacy_tokenizer.py b/rasa/nlu/tokenizers/spacy_tokenizer.py
index ed14d65f8e38..9a66cb6522e9 100644
--- a/rasa/nlu/tokenizers/spacy_tokenizer.py
+++ b/rasa/nlu/tokenizers/spacy_tokenizer.py
@@ -1,16 +1,10 @@
 import typing
-from typing import Any, Text, List
+from typing import Text, List
 
-from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
-from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.training_data import Message
 
-from rasa.nlu.constants import (
-    TEXT_ATTRIBUTE,
-    TOKENS_NAMES,
-    SPACY_DOCS,
-    DENSE_FEATURIZABLE_ATTRIBUTES,
-)
+from rasa.nlu.constants import TOKENS_NAMES, SPACY_DOCS, DENSE_FEATURIZABLE_ATTRIBUTES
 
 if typing.TYPE_CHECKING:
     from spacy.tokens.doc import Doc  # pytype: disable=import-error
@@ -23,35 +17,16 @@ class SpacyTokenizer(Tokenizer):
     requires = [SPACY_DOCS[attribute] for attribute in DENSE_FEATURIZABLE_ATTRIBUTES]
 
     defaults = {
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False
+        # Flag to check whether to split intents
+        "intent_tokenization_flag": False,
+        # Symbol on which intent should be split
+        "intent_split_symbol": "_",
     }
 
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-
-        for example in training_data.training_examples:
-
-            for attribute in DENSE_FEATURIZABLE_ATTRIBUTES:
-
-                attribute_doc = self.get_doc(example, attribute)
-
-                if attribute_doc is not None:
-                    example.set(
-                        TOKENS_NAMES[attribute], self.tokenize(attribute_doc, attribute)
-                    )
-
     def get_doc(self, message: Message, attribute: Text) -> "Doc":
         return message.get(SPACY_DOCS[attribute])
 
-    def process(self, message: Message, **kwargs: Any) -> None:
-        message.set(
-            TOKENS_NAMES[TEXT_ATTRIBUTE],
-            self.tokenize(self.get_doc(message, TEXT_ATTRIBUTE), TEXT_ATTRIBUTE),
-        )
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        doc = self.get_doc(message, attribute)
 
-    def tokenize(self, doc: "Doc", attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
-        tokens = [Token(t.text, t.idx, lemma=t.lemma_) for t in doc]
-        self.add_cls_token(tokens, attribute)
-        return tokens
+        return [Token(t.text, t.idx, lemma=t.lemma_) for t in doc]
diff --git a/rasa/nlu/tokenizers/tokenizer.py b/rasa/nlu/tokenizers/tokenizer.py
index e1b8498cf055..f179953f9a2d 100644
--- a/rasa/nlu/tokenizers/tokenizer.py
+++ b/rasa/nlu/tokenizers/tokenizer.py
@@ -3,8 +3,17 @@
 
 from typing import Text, List, Optional, Dict, Any
 
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.training_data import TrainingData, Message
 from rasa.nlu.components import Component
-from rasa.nlu.constants import RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE, CLS_TOKEN
+from rasa.nlu.constants import (
+    RESPONSE_ATTRIBUTE,
+    TEXT_ATTRIBUTE,
+    CLS_TOKEN,
+    TOKENS_NAMES,
+    MESSAGE_ATTRIBUTES,
+    INTENT_ATTRIBUTE,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -13,15 +22,17 @@ class Token(object):
     def __init__(
         self,
         text: Text,
-        offset: int,
+        start: int,
         data: Optional[Dict[Text, Any]] = None,
         lemma: Optional[Text] = None,
+        end: Optional[int] = None,
     ) -> None:
-        self.offset = offset
+        self.start = start
         self.text = text
-        self.end = offset + len(text)
+        self.end = start + len(text)
         self.data = data if data else {}
         self.lemma = lemma or text
+        self.end = end if end else start + len(text)
 
     def set(self, prop: Text, info: Any) -> None:
         self.data[prop] = info
@@ -32,8 +43,8 @@ def get(self, prop: Text, default: Optional[Any] = None) -> Any:
     def __eq__(self, other):
         if not isinstance(other, Token):
             return NotImplemented
-        return (self.offset, self.end, self.text, self.lemma) == (
-            other.offset,
+        return (self.start, self.end, self.text, self.lemma) == (
+            other.start,
             other.end,
             other.text,
             other.lemma,
@@ -42,8 +53,8 @@ def __eq__(self, other):
     def __lt__(self, other):
         if not isinstance(other, Token):
             return NotImplemented
-        return (self.offset, self.end, self.text, self.lemma) < (
-            other.offset,
+        return (self.start, self.end, self.text, self.lemma) < (
+            other.start,
             other.end,
             other.text,
             other.lemma,
@@ -51,28 +62,77 @@ def __lt__(self, other):
 
 
 class Tokenizer(Component):
-    def __init__(self, component_config: Optional[Dict[Text, Any]] = None) -> None:
-        super(Tokenizer, self).__init__(component_config)
-
-        try:
-            self.use_cls_token = self.component_config["use_cls_token"]
-        except KeyError:
-            warnings.warn(
-                "No default value for 'use_cls_token' was set. Please, "
-                "add it to the default dict of the tokenizer and set it to 'False'."
-            )
-            self.use_cls_token = False
-
-    def add_cls_token(
-        self, tokens: List[Token], attribute: Text = TEXT_ATTRIBUTE
-    ) -> List[Token]:
-        if (
-            attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]
-            and self.use_cls_token
-            and tokens
-        ):
+    def __init__(self, component_config: Dict[Text, Any] = None) -> None:
+        """Construct a new tokenizer using the WhitespaceTokenizer framework."""
+
+        super().__init__(component_config)
+
+        # flag to check whether to split intents
+        self.intent_tokenization_flag = self.component_config.get(
+            "intent_tokenization_flag", False
+        )
+        # split symbol for intents
+        self.intent_split_symbol = self.component_config.get("intent_split_symbol", "_")
+
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        """Tokenizes the text of the provided attribute of the incoming message."""
+
+        raise NotImplementedError
+
+    def train(
+        self,
+        training_data: TrainingData,
+        config: Optional[RasaNLUModelConfig] = None,
+        **kwargs: Any,
+    ) -> None:
+        """Tokenize all training data."""
+
+        for example in training_data.training_examples:
+            for attribute in MESSAGE_ATTRIBUTES:
+                if example.get(attribute) is not None:
+                    if attribute == INTENT_ATTRIBUTE:
+                        tokens = self._split_intent(example)
+                    else:
+                        tokens = self.tokenize(example, attribute)
+                        tokens = self.add_cls_token(tokens, attribute)
+                    example.set(TOKENS_NAMES[attribute], tokens)
+
+    def process(self, message: Message, **kwargs: Any) -> None:
+        """Tokenize the incoming message."""
+
+        tokens = self.tokenize(message, TEXT_ATTRIBUTE)
+        tokens = self.add_cls_token(tokens, TEXT_ATTRIBUTE)
+        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens)
+
+    def _split_intent(self, message: Message):
+        text = message.get(INTENT_ATTRIBUTE)
+
+        words = (
+            text.split(self.intent_split_symbol)
+            if self.intent_tokenization_flag
+            else [text]
+        )
+
+        return self._convert_words_to_tokens(words, text)
+
+    @staticmethod
+    def _convert_words_to_tokens(words: List[Text], text: Text) -> List[Token]:
+        running_offset = 0
+        tokens = []
+
+        for word in words:
+            word_offset = text.index(word, running_offset)
+            word_len = len(word)
+            running_offset = word_offset + word_len
+            tokens.append(Token(word, word_offset))
+
+        return tokens
+
+    @staticmethod
+    def add_cls_token(tokens: List[Token], attribute: Text) -> List[Token]:
+        if attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE] and tokens:
             # +1 to have a space between the last token and the __cls__ token
-            idx = tokens[-1].offset + len(tokens[-1].text) + 1
+            idx = tokens[-1].end + 1
             tokens.append(Token(CLS_TOKEN, idx))
 
         return tokens
diff --git a/rasa/nlu/tokenizers/whitespace_tokenizer.py b/rasa/nlu/tokenizers/whitespace_tokenizer.py
index 4b2f49b15e73..fb0fbace3131 100644
--- a/rasa/nlu/tokenizers/whitespace_tokenizer.py
+++ b/rasa/nlu/tokenizers/whitespace_tokenizer.py
@@ -1,15 +1,9 @@
 import re
 from typing import Any, Dict, List, Text
 
-from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
-from rasa.nlu.training_data import Message, TrainingData
-from rasa.nlu.constants import (
-    INTENT_ATTRIBUTE,
-    TEXT_ATTRIBUTE,
-    TOKENS_NAMES,
-    MESSAGE_ATTRIBUTES,
-)
+from rasa.nlu.training_data import Message
+from rasa.nlu.constants import TOKENS_NAMES, MESSAGE_ATTRIBUTES
 
 
 class WhitespaceTokenizer(Tokenizer):
@@ -23,76 +17,39 @@ class WhitespaceTokenizer(Tokenizer):
         "intent_split_symbol": "_",
         # Text will be tokenized with case sensitive as default
         "case_sensitive": True,
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False,
     }
 
     def __init__(self, component_config: Dict[Text, Any] = None) -> None:
         """Construct a new tokenizer using the WhitespaceTokenizer framework."""
 
         super().__init__(component_config)
-        # flag to check whether to split intents
-        self.intent_tokenization_flag = self.component_config.get(
-            "intent_tokenization_flag"
-        )
-        # split symbol for intents
-        self.intent_split_symbol = self.component_config["intent_split_symbol"]
-        self.case_sensitive = self.component_config["case_sensitive"]
-
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-        for example in training_data.training_examples:
-            for attribute in MESSAGE_ATTRIBUTES:
-                if example.get(attribute) is not None:
-                    example.set(
-                        TOKENS_NAMES[attribute],
-                        self.tokenize(example.get(attribute), attribute),
-                    )
 
-    def process(self, message: Message, **kwargs: Any) -> None:
-
-        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text))
+        self.case_sensitive = self.component_config["case_sensitive"]
 
-    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        text = message.get(attribute)
 
         if not self.case_sensitive:
             text = text.lower()
 
-        if attribute != INTENT_ATTRIBUTE:
-            # remove 'not a word character' if
-            words = re.sub(
-                # there is a space or an end of a string after it
-                r"[^\w#@&]+(?=\s|$)|"
-                # there is a space or beginning of a string before it
-                # not followed by a number
-                r"(\s|^)[^\w#@&]+(?=[^0-9\s])|"
-                # not in between numbers and not . or @ or & or - or #
-                # e.g. 10'000.00 or blabla@gmail.com
-                # and not url characters
-                r"(?<=[^0-9\s])[^\w._~:/?#\[\]()@!$&*+,;=-]+(?=[^0-9\s])",
-                " ",
-                text,
-            ).split()
-            # if we removed everything like smiles `:)`, use the whole text as 1 token
-            if not words:
-                words = [text]
-        else:
-            words = (
-                text.split(self.intent_split_symbol)
-                if self.intent_tokenization_flag
-                else [text]
-            )
-
-        running_offset = 0
-        tokens = []
-
-        for word in words:
-            word_offset = text.index(word, running_offset)
-            word_len = len(word)
-            running_offset = word_offset + word_len
-            tokens.append(Token(word, word_offset))
-
-        self.add_cls_token(tokens, attribute)
-
-        return tokens
+        # remove 'not a word character' if
+        words = re.sub(
+            # there is a space or an end of a string after it
+            r"[^\w#@&]+(?=\s|$)|"
+            # there is a space or beginning of a string before it
+            # not followed by a number
+            r"(\s|^)[^\w#@&]+(?=[^0-9\s])|"
+            # not in between numbers and not . or @ or & or - or #
+            # e.g. 10'000.00 or blabla@gmail.com
+            # and not url characters
+            r"(?<=[^0-9\s])[^\w._~:/?#\[\]()@!$&*+,;=-]+(?=[^0-9\s])",
+            " ",
+            text,
+        ).split()
+        # Fix for convert tokenizer, remove any single '&'
+        words = [w for w in words if w != "&"]
+        # if we removed everything like smiles `:)`, use the whole text as 1 token
+        if not words:
+            words = [text]
+
+        return self._convert_words_to_tokens(words, text)
diff --git a/tests/nlu/base/test_training_data.py b/tests/nlu/base/test_training_data.py
index 23e9437c5808..5ccc4c425a0a 100644
--- a/tests/nlu/base/test_training_data.py
+++ b/tests/nlu/base/test_training_data.py
@@ -2,6 +2,7 @@
 import tempfile
 from jsonschema import ValidationError
 
+from rasa.nlu.constants import TEXT_ATTRIBUTE
 from rasa.nlu import training_data
 from rasa.nlu.convert import convert_training_data
 from rasa.nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
@@ -268,7 +269,7 @@ def test_repeated_entities():
         example = td.entity_examples[0]
         entities = example.get("entities")
         assert len(entities) == 1
-        tokens = WhitespaceTokenizer().tokenize(example.text)
+        tokens = WhitespaceTokenizer().tokenize(example, attribute=TEXT_ATTRIBUTE)
         start, end = MitieEntityExtractor.find_entity(entities[0], example.text, tokens)
         assert start == 9
         assert end == 10
@@ -302,7 +303,7 @@ def test_multiword_entities():
         example = td.entity_examples[0]
         entities = example.get("entities")
         assert len(entities) == 1
-        tokens = WhitespaceTokenizer().tokenize(example.text)
+        tokens = WhitespaceTokenizer().tokenize(example, attribute=TEXT_ATTRIBUTE)
         start, end = MitieEntityExtractor.find_entity(entities[0], example.text, tokens)
         assert start == 4
         assert end == 7
diff --git a/tests/nlu/classifiers/test_embedding_intent_classifier.py b/tests/nlu/classifiers/test_embedding_intent_classifier.py
index f1607814e48b..0b3d6ffdd730 100644
--- a/tests/nlu/classifiers/test_embedding_intent_classifier.py
+++ b/tests/nlu/classifiers/test_embedding_intent_classifier.py
@@ -2,6 +2,8 @@
 import pytest
 import scipy.sparse
 
+from rasa.nlu import train
+from rasa.nlu.model import Interpreter
 from rasa.nlu.config import RasaNLUModelConfig
 from rasa.nlu.constants import (
     TEXT_ATTRIBUTE,
@@ -108,6 +110,31 @@ def test_check_labels_features_exist(messages, expected):
     )
 
 
+async def test_train(component_builder, tmpdir):
+    pipeline = [
+        {"name": "ConveRTTokenizer"},
+        {"name": "CountVectorsFeaturizer"},
+        {"name": "ConveRTFeaturizer"},
+        {"name": "EmbeddingIntentClassifier"},
+    ]
+
+    _config = RasaNLUModelConfig({"pipeline": pipeline, "language": "en"})
+
+    (trained, _, persisted_path) = await train(
+        _config,
+        path=tmpdir.strpath,
+        data=DEFAULT_DATA_PATH,
+        component_builder=component_builder,
+    )
+
+    assert trained.pipeline
+
+    loaded = Interpreter.load(persisted_path, component_builder)
+    assert loaded.pipeline
+    assert loaded.parse("hello") is not None
+    assert loaded.parse("Hello today is Monday, again!") is not None
+
+
 async def test_raise_error_on_incorrect_pipeline(component_builder, tmpdir):
     from rasa.nlu import train
 
diff --git a/tests/nlu/featurizers/test_convert_featurizer.py b/tests/nlu/featurizers/test_convert_featurizer.py
index 66e25c4f9d4d..fbf904500796 100644
--- a/tests/nlu/featurizers/test_convert_featurizer.py
+++ b/tests/nlu/featurizers/test_convert_featurizer.py
@@ -1,28 +1,92 @@
 import numpy as np
+import pytest
 
-from rasa.nlu.constants import TEXT_ATTRIBUTE, DENSE_FEATURE_NAMES, TOKENS_NAMES
+from rasa.nlu.tokenizers.tokenizer import Tokenizer
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.tokenizers.convert_tokenizer import ConveRTTokenizer
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    DENSE_FEATURE_NAMES,
+    TOKENS_NAMES,
+    RESPONSE_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+)
 from rasa.nlu.training_data import Message
-from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.dense_featurizer.convert_featurizer import ConveRTFeaturizer
 
 
-def test_convert_featurizer(mitie_feature_extractor, default_config):
-    from rasa.nlu.featurizers.dense_featurizer.convert_featurizer import (
-        ConveRTFeaturizer,
+def test_convert_featurizer_process():
+    featurizer = ConveRTFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today ?"
+    message = Message(sentence)
+    tokens = ConveRTTokenizer().tokenize(message, attribute=TEXT_ATTRIBUTE)
+    tokens = Tokenizer.add_cls_token(tokens, attribute=TEXT_ATTRIBUTE)
+    message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens)
+
+    featurizer.process(message)
+
+    expected = np.array([2.2636216, -0.26475656, -1.1358104, -0.49751878, -1.3946456])
+    expected_cls = np.array(
+        [1.0251294, -0.04053932, -0.7018805, -0.82054937, -0.75054353]
     )
 
-    component_config = {"name": "ConveRTFeaturizer", "return_sequence": False}
-    featurizer = ConveRTFeaturizer.create(component_config, RasaNLUModelConfig())
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    assert len(tokens) == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+
+def test_convert_featurizer_train():
+    featurizer = ConveRTFeaturizer.create({}, RasaNLUModelConfig())
 
     sentence = "Hey how are you today ?"
     message = Message(sentence)
-    tokens = WhitespaceTokenizer().tokenize(sentence)
+    message.set(RESPONSE_ATTRIBUTE, sentence)
+    tokens = ConveRTTokenizer().tokenize(message, attribute=TEXT_ATTRIBUTE)
+    tokens = Tokenizer.add_cls_token(tokens, attribute=TEXT_ATTRIBUTE)
     message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens)
+    message.set(TOKENS_NAMES[RESPONSE_ATTRIBUTE], tokens)
+
+    featurizer.train(TrainingData([message]), RasaNLUModelConfig())
+
+    expected = np.array([2.2636216, -0.26475656, -1.1358104, -0.49751878, -1.3946456])
+    expected_cls = np.array(
+        [1.0251294, -0.04053932, -0.7018805, -0.82054937, -0.75054353]
+    )
+
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    assert len(tokens) == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE])
+
+    assert len(tokens) == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[INTENT_ATTRIBUTE])
+
+    assert vecs is None
 
-    featurizer.process(message)
 
-    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])[0]
+@pytest.mark.parametrize(
+    "sentence, expected_text",
+    [
+        ("hello", "hello"),
+        ("you're", "you re"),
+        ("r. n. b.", "r n b"),
+        ("rock & roll", "rock roll"),
+        ("ńöñàśçií", "ńöñàśçií"),
+    ],
+)
+def test_convert_featurizer_tokens_to_text(sentence, expected_text):
+    tokens = ConveRTTokenizer().tokenize(Message(sentence), attribute=TEXT_ATTRIBUTE)
 
-    expected = np.array([1.0251294, -0.04053932, -0.7018805, -0.82054937, -0.75054353])
+    actual_text = ConveRTFeaturizer._tokens_to_text([tokens])[0]
 
-    assert np.allclose(vecs[:5], expected, atol=1e-5)
+    assert expected_text == actual_text
diff --git a/tests/nlu/featurizers/test_count_vectors_featurizer.py b/tests/nlu/featurizers/test_count_vectors_featurizer.py
index 66d3b473b810..581eefc77648 100644
--- a/tests/nlu/featurizers/test_count_vectors_featurizer.py
+++ b/tests/nlu/featurizers/test_count_vectors_featurizer.py
@@ -2,75 +2,54 @@
 import pytest
 import scipy.sparse
 
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+from rasa.nlu.constants import (
+    CLS_TOKEN,
+    TOKENS_NAMES,
+    TEXT_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
+    RESPONSE_ATTRIBUTE,
+)
 from rasa.nlu.tokenizers.tokenizer import Token
 from rasa.nlu.training_data import Message
 from rasa.nlu.training_data import TrainingData
+from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
+    CountVectorsFeaturizer,
+)
 
 
 @pytest.mark.parametrize(
-    "sentence, expected",
+    "sentence, expected, expected_cls",
     [
-        ("hello hello hello hello hello ", [[1]]),
-        ("hello goodbye hello", [[0, 1]]),
-        ("a b c d e f", [[1, 0, 0, 0, 0, 0]]),
-        ("a 1 2", [[0, 1]]),
+        ("hello hello hello hello hello", [[1]], [[5]]),
+        ("hello goodbye hello", [[0, 1]], [[1, 2]]),
+        ("a b c d e f", [[1, 0, 0, 0, 0, 0]], [[1, 1, 1, 1, 1, 1]]),
+        ("a 1 2", [[0, 1]], [[2, 1]]),
     ],
 )
-def test_count_vector_featurizer(sentence, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
+def test_count_vector_featurizer(sentence, expected, expected_cls):
+    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
 
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
-    )
     train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
-
     test_message = Message(sentence)
-    ftr.process(test_message)
 
-    assert isinstance(test_message.get("text_sparse_features"), scipy.sparse.coo_matrix)
-
-    actual = test_message.get("text_sparse_features").toarray()
-
-    assert np.all(actual[0] == expected)
-
-
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        ("hello hello hello hello hello ", [[5]]),
-        ("hello goodbye hello", [[1, 2]]),
-        ("a b c d e f", [[1, 1, 1, 1, 1, 1]]),
-        ("a 1 2", [[2, 1]]),
-    ],
-)
-def test_count_vector_featurizer_no_sequence(sentence, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
+    WhitespaceTokenizer().process(train_message)
+    WhitespaceTokenizer().process(test_message)
 
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": False}
-    )
-    train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
-    data = TrainingData([train_message])
-    ftr.train(data)
+    ftr.train(TrainingData([train_message]))
 
-    test_message = Message(sentence)
     ftr.process(test_message)
 
-    assert isinstance(test_message.get("text_sparse_features"), scipy.sparse.coo_matrix)
+    assert isinstance(
+        test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]), scipy.sparse.coo_matrix
+    )
 
-    actual = test_message.get("text_sparse_features").toarray()
+    actual = test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()
 
-    assert np.all(actual == expected)
+    assert np.all(actual[0] == expected)
+    assert np.all(actual[-1] == expected_cls)
 
 
 @pytest.mark.parametrize(
@@ -84,36 +63,34 @@ def test_count_vector_featurizer_no_sequence(sentence, expected):
 def test_count_vector_featurizer_attribute_featurization(
     sentence, intent, response, intent_features, response_features
 ):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
+    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
+    tk = WhitespaceTokenizer()
 
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
-    )
     train_message = Message(sentence)
-
     # this is needed for a valid training example
-    train_message.set("intent", intent)
-    train_message.set("response", response)
+    train_message.set(INTENT_ATTRIBUTE, intent)
+    train_message.set(RESPONSE_ATTRIBUTE, response)
 
     data = TrainingData([train_message])
+
+    tk.train(data)
     ftr.train(data)
 
     if intent_features:
         assert (
-            train_message.get("intent_sparse_features").toarray()[0] == intent_features
+            train_message.get(SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE]).toarray()[0]
+            == intent_features
         )
     else:
-        assert train_message.get("intent_sparse_features") is None
+        assert train_message.get(SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE]) is None
 
     if response_features:
         assert (
-            train_message.get("response_sparse_features").toarray()[0]
+            train_message.get(SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE]).toarray()[0]
             == response_features
         )
     else:
-        assert train_message.get("response_sparse_features") is None
+        assert train_message.get(SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE]) is None
 
 
 @pytest.mark.parametrize(
@@ -133,34 +110,31 @@ def test_count_vector_featurizer_attribute_featurization(
 def test_count_vector_featurizer_shared_vocab(
     sentence, intent, response, text_features, intent_features, response_features
 ):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
-
     ftr = CountVectorsFeaturizer(
-        {
-            "token_pattern": r"(?u)\b\w+\b",
-            "use_shared_vocab": True,
-            "return_sequence": True,
-        }
+        {"token_pattern": r"(?u)\b\w+\b", "use_shared_vocab": True}
     )
-    train_message = Message(sentence)
+    tk = WhitespaceTokenizer()
 
+    train_message = Message(sentence)
     # this is needed for a valid training example
-    train_message.set("intent", intent)
-    train_message.set("response", response)
+    train_message.set(INTENT_ATTRIBUTE, intent)
+    train_message.set(RESPONSE_ATTRIBUTE, response)
 
     data = TrainingData([train_message])
+    tk.train(data)
     ftr.train(data)
 
     assert np.all(
-        train_message.get("text_sparse_features").toarray()[0] == text_features
+        train_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()[0]
+        == text_features
     )
     assert np.all(
-        train_message.get("intent_sparse_features").toarray()[0] == intent_features
+        train_message.get(SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE]).toarray()[0]
+        == intent_features
     )
     assert np.all(
-        train_message.get("response_sparse_features").toarray()[0] == response_features
+        train_message.get(SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE]).toarray()[0]
+        == response_features
     )
 
 
@@ -174,27 +148,21 @@ def test_count_vector_featurizer_shared_vocab(
     ],
 )
 def test_count_vector_featurizer_oov_token(sentence, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
-
     ftr = CountVectorsFeaturizer(
-        {
-            "token_pattern": r"(?u)\b\w+\b",
-            "OOV_token": "__oov__",
-            "return_sequence": True,
-        }
+        {"token_pattern": r"(?u)\b\w+\b", "OOV_token": "__oov__"}
     )
     train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
+    WhitespaceTokenizer().process(train_message)
+
     data = TrainingData([train_message])
     ftr.train(data)
 
     test_message = Message(sentence)
     ftr.process(test_message)
 
-    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+    assert np.all(
+        test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()[0] == expected
+    )
 
 
 @pytest.mark.parametrize(
@@ -207,50 +175,43 @@ def test_count_vector_featurizer_oov_token(sentence, expected):
     ],
 )
 def test_count_vector_featurizer_oov_words(sentence, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
 
     ftr = CountVectorsFeaturizer(
         {
             "token_pattern": r"(?u)\b\w+\b",
             "OOV_token": "__oov__",
             "OOV_words": ["oov_word0", "OOV_word1"],
-            "return_sequence": True,
         }
     )
     train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
+    WhitespaceTokenizer().process(train_message)
+
     data = TrainingData([train_message])
     ftr.train(data)
 
     test_message = Message(sentence)
     ftr.process(test_message)
 
-    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+    assert np.all(
+        test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()[0] == expected
+    )
 
 
 @pytest.mark.parametrize(
     "tokens, expected",
     [
-        (["hello", "hello", "hello", "hello", "hello"], [[1]]),
-        (["你好", "你好", "你好", "你好", "你好"], [[1]]),  # test for unicode chars
-        (["hello", "goodbye", "hello"], [[0, 1]]),
+        (["hello", "hello", "hello", "hello", "hello", CLS_TOKEN], [[1]]),
+        (["你好", "你好", "你好", "你好", "你好", CLS_TOKEN], [[1]]),  # test for unicode chars
+        (["hello", "goodbye", "hello", CLS_TOKEN], [[0, 1]]),
         # Note: order has changed in Chinese version of "hello" & "goodbye"
-        (["你好", "再见", "你好"], [[1, 0]]),  # test for unicode chars
-        (["a", "b", "c", "d", "e", "f"], [[1, 0, 0, 0, 0, 0]]),
-        (["a", "1", "2"], [[0, 1]]),
+        (["你好", "再见", "你好", CLS_TOKEN], [[1, 0]]),  # test for unicode chars
+        (["a", "b", "c", "d", "e", "f", CLS_TOKEN], [[1, 0, 0, 0, 0, 0]]),
+        (["a", "1", "2", CLS_TOKEN], [[0, 1]]),
     ],
 )
 def test_count_vector_featurizer_using_tokens(tokens, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
 
-    ftr = CountVectorsFeaturizer(
-        {"token_pattern": r"(?u)\b\w+\b", "return_sequence": True}
-    )
+    ftr = CountVectorsFeaturizer({"token_pattern": r"(?u)\b\w+\b"})
 
     # using empty string instead of real text string to make sure
     # count vector only can come from `tokens` feature.
@@ -259,19 +220,20 @@ def test_count_vector_featurizer_using_tokens(tokens, expected):
     tokens_feature = [Token(i, 0) for i in tokens]
 
     train_message = Message("")
-    train_message.set("tokens", tokens_feature)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
+    train_message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens_feature)
+
     data = TrainingData([train_message])
 
     ftr.train(data)
 
     test_message = Message("")
-    test_message.set("tokens", tokens_feature)
+    test_message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], tokens_feature)
 
     ftr.process(test_message)
 
-    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+    assert np.all(
+        test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()[0] == expected
+    )
 
 
 @pytest.mark.parametrize(
@@ -283,29 +245,24 @@ def test_count_vector_featurizer_using_tokens(tokens, expected):
     ],
 )
 def test_count_vector_featurizer_char(sentence, expected):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
+    ftr = CountVectorsFeaturizer({"min_ngram": 1, "max_ngram": 2, "analyzer": "char"})
 
-    ftr = CountVectorsFeaturizer(
-        {"min_ngram": 1, "max_ngram": 2, "analyzer": "char", "return_sequence": True}
-    )
     train_message = Message(sentence)
-    # this is needed for a valid training example
-    train_message.set("intent", "bla")
+    WhitespaceTokenizer().process(train_message)
+
     data = TrainingData([train_message])
     ftr.train(data)
 
     test_message = Message(sentence)
+    WhitespaceTokenizer().process(test_message)
     ftr.process(test_message)
 
-    assert np.all(test_message.get("text_sparse_features").toarray()[0] == expected)
+    assert np.all(
+        test_message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()[0] == expected
+    )
 
 
 def test_count_vector_featurizer_persist_load(tmpdir):
-    from rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer import (
-        CountVectorsFeaturizer,
-    )
 
     # set non default values to config
     config = {
@@ -319,7 +276,6 @@ def test_count_vector_featurizer_persist_load(tmpdir):
         "max_ngram": 3,
         "max_features": 10,
         "lowercase": False,
-        "return_sequence": True,
     }
     train_ftr = CountVectorsFeaturizer(config)
 
@@ -328,17 +284,16 @@ def test_count_vector_featurizer_persist_load(tmpdir):
     train_message1 = Message(sentence1)
     train_message2 = Message(sentence2)
 
-    # this is needed for a valid training example
-    train_message1.set("intent", "bla")
-    train_message2.set("intent", "bla")
     data = TrainingData([train_message1, train_message2])
     train_ftr.train(data)
+
     # persist featurizer
     file_dict = train_ftr.persist("ftr", tmpdir.strpath)
     train_vect_params = {
         attribute: vectorizer.get_params()
         for attribute, vectorizer in train_ftr.vectorizers.items()
     }
+
     # add trained vocabulary to vectorizer params
     for attribute, attribute_vect_params in train_vect_params.items():
         if hasattr(train_ftr.vectorizers[attribute], "vocabulary_"):
@@ -365,7 +320,42 @@ def test_count_vector_featurizer_persist_load(tmpdir):
     # check that train features and test features after loading are the same
     assert np.all(
         [
-            train_message1.get("text_features") == test_message1.get("text_features"),
-            train_message2.get("text_features") == test_message2.get("text_features"),
+            train_message1.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()
+            == test_message1.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray(),
+            train_message2.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray()
+            == test_message2.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE]).toarray(),
         ]
     )
+
+
+def test_count_vectors_featurizer_train():
+
+    featurizer = CountVectorsFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today ?"
+    message = Message(sentence)
+    message.set(RESPONSE_ATTRIBUTE, sentence)
+    message.set(INTENT_ATTRIBUTE, "intent")
+    WhitespaceTokenizer().train(TrainingData([message]))
+
+    featurizer.train(TrainingData([message]), RasaNLUModelConfig())
+
+    expected = np.array([0, 1, 0, 0, 0])
+    expected_cls = np.array([1, 1, 1, 1, 1])
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    assert (6, 5) == vecs.shape
+    assert np.all(vecs.toarray()[0] == expected)
+    assert np.all(vecs.toarray()[-1] == expected_cls)
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE])
+
+    assert (6, 5) == vecs.shape
+    assert np.all(vecs.toarray()[0] == expected)
+    assert np.all(vecs.toarray()[-1] == expected_cls)
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE])
+
+    assert (1, 1) == vecs.shape
+    assert np.all(vecs.toarray()[0] == np.array([1]))
diff --git a/tests/nlu/featurizers/test_featurizer.py b/tests/nlu/featurizers/test_featurizer.py
index 1d3fb775e27a..666dcdd4ba98 100644
--- a/tests/nlu/featurizers/test_featurizer.py
+++ b/tests/nlu/featurizers/test_featurizer.py
@@ -82,12 +82,16 @@ def test_combine_with_existing_sparse_features_shape_mismatch():
 @pytest.mark.parametrize(
     "features, expected",
     [
-        ([[1, 0, 2, 3], [2, 0, 0, 1]], [1.5, 0, 1, 2]),
+        (None, None),
+        ([[1, 0, 2, 3], [2, 0, 0, 1]], [[2, 0, 0, 1]]),
+        (
+            scipy.sparse.coo_matrix([[1, 0, 2, 3], [2, 0, 0, 1]]),
+            scipy.sparse.coo_matrix([2, 0, 0, 1]),
+        ),
         (
             scipy.sparse.csr_matrix([[1, 0, 2, 3], [2, 0, 0, 1]]),
-            scipy.sparse.csr_matrix([3, 0, 2, 4]),
+            scipy.sparse.csr_matrix([2, 0, 0, 1]),
         ),
-        (None, None),
     ],
 )
 def test_sequence_to_sentence_features(features, expected):
diff --git a/tests/nlu/featurizers/test_mitie_featurizer.py b/tests/nlu/featurizers/test_mitie_featurizer.py
index fa9a581de7d1..95fb1f7a83b0 100644
--- a/tests/nlu/featurizers/test_mitie_featurizer.py
+++ b/tests/nlu/featurizers/test_mitie_featurizer.py
@@ -1,38 +1,72 @@
 import numpy as np
 
+from rasa.nlu.constants import (
+    DENSE_FEATURE_NAMES,
+    TEXT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    TOKENS_NAMES,
+)
+from rasa.nlu.training_data import Message, TrainingData
 from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
 from rasa.nlu.config import RasaNLUModelConfig
-from rasa.nlu.constants import CLS_TOKEN
+from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
 
 
-def test_mitie_featurizer(mitie_feature_extractor, default_config):
-    from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
+def test_mitie_featurizer(mitie_feature_extractor):
 
-    component_config = {"name": "MitieFeaturizer", "return_sequence": True}
-    featurizer = MitieFeaturizer.create(component_config, RasaNLUModelConfig())
+    featurizer = MitieFeaturizer.create({}, RasaNLUModelConfig())
 
-    sentence = f"Hey how are you today {CLS_TOKEN}"
+    sentence = "Hey how are you today"
+    message = Message(sentence)
+    MitieTokenizer().process(message)
+    tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
 
-    tokens = MitieTokenizer().tokenize(sentence)
+    vecs = featurizer.features_for_tokens(tokens, mitie_feature_extractor)
 
-    vecs = featurizer.features_for_tokens(tokens, mitie_feature_extractor)[0]
+    expected = np.array(
+        [0.00000000e00, -5.12735510e00, 4.39929873e-01, -5.60760403e00, -8.26445103e00]
+    )
+    expected_cls = np.array([0.0, -4.4551446, 0.26073121, -1.46632245, -1.84205751])
+
+    assert 6 == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+
+def test_mitie_featurizer_train(mitie_feature_extractor):
+
+    featurizer = MitieFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today"
+    message = Message(sentence)
+    message.set(RESPONSE_ATTRIBUTE, sentence)
+    message.set(INTENT_ATTRIBUTE, "intent")
+    MitieTokenizer().train(TrainingData([message]))
+
+    featurizer.train(
+        TrainingData([message]),
+        RasaNLUModelConfig(),
+        **{"mitie_feature_extractor": mitie_feature_extractor},
+    )
 
     expected = np.array(
         [0.00000000e00, -5.12735510e00, 4.39929873e-01, -5.60760403e00, -8.26445103e00]
     )
-    assert np.allclose(vecs[:5], expected, atol=1e-5)
+    expected_cls = np.array([0.0, -4.4551446, 0.26073121, -1.46632245, -1.84205751])
 
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
 
-def test_mitie_featurizer_no_sequence(mitie_feature_extractor, default_config):
-    from rasa.nlu.featurizers.dense_featurizer.mitie_featurizer import MitieFeaturizer
+    assert len(message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])) == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
 
-    component_config = {"name": "MitieFeaturizer", "return_sequence": False}
-    featurizer = MitieFeaturizer.create(component_config, RasaNLUModelConfig())
+    vecs = message.get(DENSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE])
 
-    sentence = f"Hey how are you today {CLS_TOKEN}"
-    tokens = MitieTokenizer().tokenize(sentence)
+    assert len(message.get(TOKENS_NAMES[RESPONSE_ATTRIBUTE])) == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
 
-    vecs = featurizer.features_for_tokens(tokens, mitie_feature_extractor)[0]
+    vecs = message.get(DENSE_FEATURE_NAMES[INTENT_ATTRIBUTE])
 
-    expected = np.array([0.0, -4.4551446, 0.26073121, -1.46632245, -1.84205751])
-    assert np.allclose(vecs[:5], expected, atol=1e-5)
+    assert vecs is None
diff --git a/tests/nlu/featurizers/test_regex_featurizer.py b/tests/nlu/featurizers/test_regex_featurizer.py
index cf252a09a49a..dcc9b80e107d 100644
--- a/tests/nlu/featurizers/test_regex_featurizer.py
+++ b/tests/nlu/featurizers/test_regex_featurizer.py
@@ -1,7 +1,18 @@
 import numpy as np
 import pytest
 
-from rasa.nlu.constants import TEXT_ATTRIBUTE, RESPONSE_ATTRIBUTE, SPACY_DOCS
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+from rasa.nlu.constants import (
+    TEXT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    SPACY_DOCS,
+    TOKENS_NAMES,
+    INTENT_ATTRIBUTE,
+    SPARSE_FEATURE_NAMES,
+)
 from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
 from rasa.nlu.training_data import Message
 
@@ -17,6 +28,7 @@
                 [0.0, 0.0, 0.0],
                 [0.0, 0.0, 0.0],
                 [0.0, 0.0, 0.0],
+                [0.0, 1.0, 0.0],
             ],
             [0],
         ),
@@ -28,17 +40,30 @@
                 [0.0, 0.0, 0.0],
                 [0.0, 0.0, 0.0],
                 [0.0, 0.0, 0.0],
+                [1.0, 1.0, 0.0],
             ],
             [1, 0],
         ),
         (
             "blah balh random eh",
-            [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
+            [
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+            ],
             [],
         ),
         (
             "a 1 digit number",
-            [[0.0, 0.0, 0.0], [1.0, 0.0, 1.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
+            [
+                [0.0, 0.0, 0.0],
+                [1.0, 0.0, 1.0],
+                [0.0, 0.0, 0.0],
+                [0.0, 0.0, 0.0],
+                [1.0, 0.0, 1.0],
+            ],
             [1, 1],
         ),
     ],
@@ -51,10 +76,10 @@ def test_regex_featurizer(sentence, expected, labeled_tokens, spacy_nlp):
         {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
         {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
     ]
-    ftr = RegexFeaturizer({"return_sequence": True}, known_patterns=patterns)
+    ftr = RegexFeaturizer({}, known_patterns=patterns)
 
     # adds tokens to the message
-    tokenizer = SpacyTokenizer({"use_cls_token": False})
+    tokenizer = SpacyTokenizer({})
     message = Message(sentence, data={RESPONSE_ATTRIBUTE: sentence})
     message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(sentence))
     tokenizer.process(message)
@@ -63,9 +88,9 @@ def test_regex_featurizer(sentence, expected, labeled_tokens, spacy_nlp):
     assert np.allclose(result.toarray(), expected, atol=1e-10)
 
     # the tokenizer should have added tokens
-    assert len(message.get("tokens", [])) > 0
+    assert len(message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])) > 0
     # the number of regex matches on each token should match
-    for i, token in enumerate(message.get("tokens")):
+    for i, token in enumerate(message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])):
         token_matches = token.get("pattern").values()
         num_matches = sum(token_matches)
         assert num_matches == labeled_tokens.count(i)
@@ -76,16 +101,28 @@ def test_regex_featurizer(sentence, expected, labeled_tokens, spacy_nlp):
     [
         (
             "lemonade and mapo tofu",
-            [[1.0, 0.0], [0.0, 0.0], [0.0, 1.0], [0.0, 1.0]],
+            [[1.0, 0.0], [0.0, 0.0], [0.0, 1.0], [0.0, 1.0], [1.0, 1.0]],
             [0.0, 2.0, 3.0],
         ),
-        ("a cup of tea", [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [1.0, 0.0]], [3.0]),
+        (
+            "a cup of tea",
+            [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [1.0, 0.0], [1.0, 0.0]],
+            [3.0],
+        ),
         (
             "Is burrito my favorite food?",
-            [[0.0, 0.0], [0.0, 1.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0]],
+            [
+                [0.0, 0.0],
+                [0.0, 1.0],
+                [0.0, 0.0],
+                [0.0, 0.0],
+                [0.0, 0.0],
+                [0.0, 0.0],
+                [0.0, 1.0],
+            ],
             [1.0],
         ),
-        ("I want club?mate", [[0.0, 0.0], [0.0, 0.0], [1.0, 0.0]], [2.0, 3.0]),
+        ("I want club?mate", [[0.0, 0.0], [0.0, 0.0], [1.0, 0.0], [1.0, 0.0]], [2.0]),
     ],
 )
 def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
@@ -98,10 +135,10 @@ def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
         },
         {"name": "plates", "elements": "data/test/lookup_tables/plates.txt"},
     ]
-    ftr = RegexFeaturizer({"return_sequence": True}, lookup_tables=lookups)
+    ftr = RegexFeaturizer({}, lookup_tables=lookups)
 
     # adds tokens to the message
-    component_config = {"name": "SpacyTokenizer", "use_cls_token": False}
+    component_config = {"name": "SpacyTokenizer"}
     tokenizer = SpacyTokenizer(component_config)
     message = Message(sentence)
     message.set("spacy_doc", spacy_nlp(sentence))
@@ -111,38 +148,78 @@ def test_lookup_tables(sentence, expected, labeled_tokens, spacy_nlp):
     assert np.allclose(result.toarray(), expected, atol=1e-10)
 
     # the tokenizer should have added tokens
-    assert len(message.get("tokens", [])) > 0
+    assert len(message.get(TOKENS_NAMES[TEXT_ATTRIBUTE], [])) > 0
     # the number of regex matches on each token should match
-    for i, token in enumerate(message.get("tokens")):
+    for i, token in enumerate(message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])):
         token_matches = token.get("pattern").values()
         num_matches = sum(token_matches)
         assert num_matches == labeled_tokens.count(i)
 
 
 @pytest.mark.parametrize(
-    "sentence, expected ",
+    "sentence, expected, expected_cls",
     [
-        ("hey how are you today", [0.0, 1.0, 0.0]),
-        ("hey 456 how are you", [1.0, 1.0, 0.0]),
-        ("blah balh random eh", [0.0, 0.0, 0.0]),
-        ("a 1 digit number", [1.0, 0.0, 1.0]),
+        ("hey how are you today", [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]),
+        ("hey 456 how are you", [0.0, 1.0, 0.0], [1.0, 1.0, 0.0]),
+        ("blah balh random eh", [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]),
+        ("a 1 digit number", [0.0, 0.0, 0.0], [1.0, 0.0, 1.0]),
     ],
 )
-def test_regex_featurizer_no_sequence(sentence, expected, spacy_nlp):
-    from rasa.nlu.featurizers.sparse_featurizer.regex_featurizer import RegexFeaturizer
+def test_regex_featurizer_no_sequence(sentence, expected, expected_cls, spacy_nlp):
 
     patterns = [
         {"pattern": "[0-9]+", "name": "number", "usage": "intent"},
         {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
         {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
     ]
-    ftr = RegexFeaturizer({"return_sequence": False}, known_patterns=patterns)
+    ftr = RegexFeaturizer({}, known_patterns=patterns)
 
     # adds tokens to the message
     tokenizer = SpacyTokenizer()
     message = Message(sentence)
-    message.set("spacy_doc", spacy_nlp(sentence))
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(sentence))
     tokenizer.process(message)
 
     result = ftr._features_for_patterns(message, TEXT_ATTRIBUTE)
     assert np.allclose(result.toarray()[0], expected, atol=1e-10)
+    assert np.allclose(result.toarray()[-1], expected_cls, atol=1e-10)
+
+
+def test_regex_featurizer_train():
+
+    patterns = [
+        {"pattern": "[0-9]+", "name": "number", "usage": "intent"},
+        {"pattern": "\\bhey*", "name": "hello", "usage": "intent"},
+        {"pattern": "[0-1]+", "name": "binary", "usage": "intent"},
+    ]
+
+    featurizer = RegexFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "hey how are you today 19.12.2019 ?"
+    message = Message(sentence)
+    message.set(RESPONSE_ATTRIBUTE, sentence)
+    message.set(INTENT_ATTRIBUTE, "intent")
+    WhitespaceTokenizer().train(TrainingData([message]))
+
+    featurizer.train(
+        TrainingData([message], regex_features=patterns), RasaNLUModelConfig()
+    )
+
+    expected = np.array([0, 1, 0])
+    expected_cls = np.array([1, 1, 1])
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    assert (7, 3) == vecs.shape
+    assert np.all(vecs.toarray()[0] == expected)
+    assert np.all(vecs.toarray()[-1] == expected_cls)
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE])
+
+    assert (7, 3) == vecs.shape
+    assert np.all(vecs.toarray()[0] == expected)
+    assert np.all(vecs.toarray()[-1] == expected_cls)
+
+    vecs = message.get(SPARSE_FEATURE_NAMES[INTENT_ATTRIBUTE])
+
+    assert vecs is None
diff --git a/tests/nlu/featurizers/test_spacy_featurizer.py b/tests/nlu/featurizers/test_spacy_featurizer.py
index 536075ad1900..e13acd4a0312 100644
--- a/tests/nlu/featurizers/test_spacy_featurizer.py
+++ b/tests/nlu/featurizers/test_spacy_featurizer.py
@@ -5,17 +5,44 @@
 from rasa.nlu.training_data import Message
 from rasa.nlu.training_data import TrainingData
 from rasa.nlu.config import RasaNLUModelConfig
+from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
+from rasa.nlu.constants import (
+    SPACY_DOCS,
+    TEXT_ATTRIBUTE,
+    DENSE_FEATURE_NAMES,
+    RESPONSE_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+)
+
+
+def test_spacy_featurizer_cls_vector(spacy_nlp):
+    featurizer = SpacyFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today"
+    message = Message(sentence)
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(sentence))
+
+    featurizer._set_spacy_features(message)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    expected = np.array([-0.28451, 0.31007, -0.57039, -0.073056, -0.17322])
+    expected_cls = np.array([-0.196496, 0.3249364, -0.37408298, -0.10622784, 0.062756])
+
+    assert 6 == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
 
 
 @pytest.mark.parametrize("sentence", ["hey how are you today"])
 def test_spacy_featurizer(sentence, spacy_nlp):
-    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
 
-    ftr = SpacyFeaturizer.create({"return_sequence": True}, RasaNLUModelConfig())
+    ftr = SpacyFeaturizer.create({}, RasaNLUModelConfig())
 
     doc = spacy_nlp(sentence)
     vecs = ftr._features_for_doc(doc)
     expected = [t.vector for t in doc]
+
     assert np.allclose(vecs, expected, atol=1e-5)
 
 
@@ -49,7 +76,7 @@ def test_spacy_intent_featurizer(spacy_nlp_component):
 
     td = training_data.load_data("data/examples/rasa/demo-rasa.json")
     spacy_nlp_component.train(td, config=None)
-    spacy_featurizer = SpacyFeaturizer({"return_sequence": True})
+    spacy_featurizer = SpacyFeaturizer()
     spacy_featurizer.train(td, config=None)
 
     intent_features_exist = np.array(
@@ -73,8 +100,7 @@ def test_spacy_featurizer_sequence(sentence, expected, spacy_nlp):
     doc = spacy_nlp(sentence)
     token_vectors = [t.vector for t in doc]
 
-    spacy_config = {"return_sequence": True}
-    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
+    ftr = SpacyFeaturizer.create({}, RasaNLUModelConfig())
 
     greet = {"intent": "greet", "text_features": [0.5]}
 
@@ -89,36 +115,6 @@ def test_spacy_featurizer_sequence(sentence, expected, spacy_nlp):
     assert np.allclose(vecs, expected, atol=1e-4)
 
 
-@pytest.mark.parametrize(
-    "sentence, expected",
-    [
-        (
-            "hey how are you today",
-            [-0.19649599, 0.32493639, -0.37408298, -0.10622784, 0.062756],
-        )
-    ],
-)
-def test_spacy_featurizer_no_sequence(sentence, expected, spacy_nlp):
-    from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
-
-    doc = spacy_nlp(sentence)
-
-    spacy_config = {"return_sequence": False}
-    ftr = SpacyFeaturizer.create(spacy_config, RasaNLUModelConfig())
-
-    greet = {"intent": "greet", "text_features": [0.5]}
-
-    message = Message(sentence, greet)
-    message.set("spacy_doc", doc)
-
-    ftr._set_spacy_features(message)
-
-    vecs = message.get("text_dense_features")[0]
-
-    assert np.allclose(doc.vector, vecs, atol=1e-4)
-    assert np.allclose(expected, vecs[:5], atol=1e-4)
-
-
 def test_spacy_featurizer_casing(spacy_nlp):
     from rasa.nlu.featurizers.dense_featurizer.spacy_featurizer import SpacyFeaturizer
 
@@ -127,7 +123,7 @@ def test_spacy_featurizer_casing(spacy_nlp):
     # retrieves vectors. For compressed spacy models (e.g. models
     # ending in _sm) this test will most likely fail.
 
-    ftr = SpacyFeaturizer.create({"return_sequence": True}, RasaNLUModelConfig())
+    ftr = SpacyFeaturizer.create({}, RasaNLUModelConfig())
 
     td = training_data.load_data("data/examples/rasa/demo-rasa.json")
     for e in td.intent_examples:
@@ -142,3 +138,36 @@ def test_spacy_featurizer_casing(spacy_nlp):
         ), "Vectors are unequal for texts '{}' and '{}'".format(
             e.text, e.text.capitalize()
         )
+
+
+def test_spacy_featurizer_train(spacy_nlp):
+
+    featurizer = SpacyFeaturizer.create({}, RasaNLUModelConfig())
+
+    sentence = "Hey how are you today"
+    message = Message(sentence)
+    message.set(RESPONSE_ATTRIBUTE, sentence)
+    message.set(INTENT_ATTRIBUTE, "intent")
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(sentence))
+    message.set(SPACY_DOCS[RESPONSE_ATTRIBUTE], spacy_nlp(sentence))
+
+    featurizer.train(TrainingData([message]), RasaNLUModelConfig())
+
+    expected = np.array([-0.28451, 0.31007, -0.57039, -0.073056, -0.17322])
+    expected_cls = np.array([-0.196496, 0.3249364, -0.37408298, -0.10622784, 0.062756])
+
+    vecs = message.get(DENSE_FEATURE_NAMES[TEXT_ATTRIBUTE])
+
+    assert 6 == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[RESPONSE_ATTRIBUTE])
+
+    assert 6 == len(vecs)
+    assert np.allclose(vecs[0][:5], expected, atol=1e-5)
+    assert np.allclose(vecs[-1][:5], expected_cls, atol=1e-5)
+
+    vecs = message.get(DENSE_FEATURE_NAMES[INTENT_ATTRIBUTE])
+
+    assert vecs is None
diff --git a/tests/nlu/tokenizers/test_convert_tokenizer.py b/tests/nlu/tokenizers/test_convert_tokenizer.py
new file mode 100644
index 000000000000..cc11642b1727
--- /dev/null
+++ b/tests/nlu/tokenizers/test_convert_tokenizer.py
@@ -0,0 +1,56 @@
+import pytest
+
+from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.constants import TEXT_ATTRIBUTE, INTENT_ATTRIBUTE, TOKENS_NAMES
+from rasa.nlu.tokenizers.convert_tokenizer import ConveRTTokenizer
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["forecast", "for", "lunch"],
+            [(0, 8), (9, 12), (13, 18)],
+        ),
+        ("hello", ["hello"], [(0, 5)]),
+        ("you're", ["you", "re"], [(0, 3), (4, 6)]),
+        ("r. n. b.", ["r", "n", "b"], [(0, 1), (3, 4), (6, 7)]),
+        ("rock & roll", ["rock", "roll"], [(0, 4), (7, 11)]),
+        (
+            "ńöñàśçií",
+            ["ń", "ö", "ñ", "à", "ś", "ç", "i", "í"],
+            [(0, 1), (1, 2), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8)],
+        ),
+    ],
+)
+def test_convert_tokenizer_edge_cases(text, expected_tokens, expected_indices):
+    tk = ConveRTTokenizer()
+
+    tokens = tk.tokenize(Message(text), attribute=TEXT_ATTRIBUTE)
+
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_custom_intent_symbol(text, expected_tokens):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
+
+    tk = ConveRTTokenizer(component_config)
+
+    message = Message(text)
+    message.set(INTENT_ATTRIBUTE, text)
+
+    tk.train(TrainingData([message]))
+
+    assert [
+        t.text for t in message.get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+    ] == expected_tokens
diff --git a/tests/nlu/tokenizers/test_jieba_tokenizer.py b/tests/nlu/tokenizers/test_jieba_tokenizer.py
index 7df57c5bfcd1..428e4b7cd350 100644
--- a/tests/nlu/tokenizers/test_jieba_tokenizer.py
+++ b/tests/nlu/tokenizers/test_jieba_tokenizer.py
@@ -1,53 +1,69 @@
 from unittest.mock import patch
 
-from rasa.nlu.constants import CLS_TOKEN
+from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
 
+import pytest
 
-def test_jieba():
-    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
+from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.constants import TEXT_ATTRIBUTE, INTENT_ATTRIBUTE, TOKENS_NAMES
 
-    component_config = {"use_cls_token": False}
 
-    tk = JiebaTokenizer(component_config)
-
-    assert [t.text for t in tk.tokenize("我想去吃兰州拉面")] == ["我", "想", "去", "吃", "兰州", "拉面"]
-
-    assert [t.offset for t in tk.tokenize("我想去吃兰州拉面")] == [0, 1, 2, 3, 4, 6]
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "我想去吃兰州拉面",
+            ["我", "想", "去", "吃", "兰州", "拉面"],
+            [(0, 1), (1, 2), (2, 3), (3, 4), (4, 6), (6, 8)],
+        ),
+        (
+            "Micheal你好吗？",
+            ["Micheal", "你好", "吗", "？"],
+            [(0, 7), (7, 9), (9, 10), (10, 11)],
+        ),
+    ],
+)
+def test_jieba(text, expected_tokens, expected_indices):
+    tk = JiebaTokenizer()
 
-    assert [t.text for t in tk.tokenize("Micheal你好吗？")] == ["Micheal", "你好", "吗", "？"]
+    tokens = tk.tokenize(Message(text), attribute=TEXT_ATTRIBUTE)
 
-    assert [t.offset for t in tk.tokenize("Micheal你好吗？")] == [0, 7, 9, 10]
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
 
 
 def test_jieba_load_dictionary(tmpdir_factory):
-    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
-
     dictionary_path = tmpdir_factory.mktemp("jieba_custom_dictionary").strpath
 
-    component_config = {"dictionary_path": dictionary_path, "use_cls_token": False}
+    component_config = {"dictionary_path": dictionary_path}
 
     with patch.object(
         JiebaTokenizer, "load_custom_dictionary", return_value=None
     ) as mock_method:
         tk = JiebaTokenizer(component_config)
-        tk.tokenize("")
+        tk.tokenize(Message(""), attribute=TEXT_ATTRIBUTE)
 
     mock_method.assert_called_once_with(dictionary_path)
 
 
-def test_jieba_add_cls_token():
-    from rasa.nlu.tokenizers.jieba_tokenizer import JiebaTokenizer
-
-    component_config = {"use_cls_token": True}
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_custom_intent_symbol(text, expected_tokens):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
 
     tk = JiebaTokenizer(component_config)
 
-    assert [t.text for t in tk.tokenize("Micheal你好吗？")] == [
-        "Micheal",
-        "你好",
-        "吗",
-        "？",
-        CLS_TOKEN,
-    ]
+    message = Message(text)
+    message.set(INTENT_ATTRIBUTE, text)
+
+    tk.train(TrainingData([message]))
 
-    assert [t.offset for t in tk.tokenize("Micheal你好吗？")] == [0, 7, 9, 10, 12]
+    assert [
+        t.text for t in message.get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+    ] == expected_tokens
diff --git a/tests/nlu/tokenizers/test_mitie_tokenizer.py b/tests/nlu/tokenizers/test_mitie_tokenizer.py
index 41774fb9a440..647290990b48 100644
--- a/tests/nlu/tokenizers/test_mitie_tokenizer.py
+++ b/tests/nlu/tokenizers/test_mitie_tokenizer.py
@@ -1,41 +1,52 @@
-from rasa.nlu.constants import CLS_TOKEN
-
-
-def test_mitie():
-    from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
-
-    component_config = {"use_cls_token": False}
+import pytest
+
+from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.constants import TEXT_ATTRIBUTE, INTENT_ATTRIBUTE, TOKENS_NAMES
+from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch"],
+            [(0, 8), (9, 12), (13, 18)],
+        ),
+        (
+            "hey ńöñàśçií how're you?",
+            ["hey", "ńöñàśçií", "how", "'re", "you", "?"],
+            [(0, 3), (4, 12), (13, 16), (16, 19), (20, 23), (23, 24)],
+        ),
+    ],
+)
+def test_mitie(text, expected_tokens, expected_indices):
+    tk = MitieTokenizer()
+
+    tokens = tk.tokenize(Message(text), attribute=TEXT_ATTRIBUTE)
+
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_custom_intent_symbol(text, expected_tokens):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
 
     tk = MitieTokenizer(component_config)
 
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(text)] == ["Forecast", "for", "lunch"]
-    assert [t.offset for t in tk.tokenize(text)] == [0, 9, 13]
-
-    text = "hey ńöñàśçií how're you?"
-    assert [t.text for t in tk.tokenize(text)] == [
-        "hey",
-        "ńöñàśçií",
-        "how",
-        "'re",
-        "you",
-        "?",
-    ]
-    assert [t.offset for t in tk.tokenize(text)] == [0, 4, 13, 16, 20, 23]
+    message = Message(text)
+    message.set(INTENT_ATTRIBUTE, text)
 
+    tk.train(TrainingData([message]))
 
-def test_mitie_add_cls_token():
-    from rasa.nlu.tokenizers.mitie_tokenizer import MitieTokenizer
-
-    component_config = {"use_cls_token": True}
-
-    tk = MitieTokenizer(component_config)
-
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(text)] == [
-        "Forecast",
-        "for",
-        "lunch",
-        CLS_TOKEN,
-    ]
-    assert [t.offset for t in tk.tokenize(text)] == [0, 9, 13, 19]
+    assert [
+        t.text for t in message.get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+    ] == expected_tokens
diff --git a/tests/nlu/tokenizers/test_spacy_tokenizer.py b/tests/nlu/tokenizers/test_spacy_tokenizer.py
index 396fc848f249..66e8a2b80919 100644
--- a/tests/nlu/tokenizers/test_spacy_tokenizer.py
+++ b/tests/nlu/tokenizers/test_spacy_tokenizer.py
@@ -1,71 +1,95 @@
-from rasa.nlu.constants import CLS_TOKEN
-from rasa.nlu import training_data
-
-
-def test_spacy(spacy_nlp):
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-
-    component_config = {"use_cls_token": False}
-
-    tk = SpacyTokenizer(component_config)
-
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
-        "Forecast",
-        "for",
-        "lunch",
-    ]
-    assert [t.lemma for t in tk.tokenize(spacy_nlp(text))] == [
-        "forecast",
-        "for",
-        "lunch",
-    ]
-
-    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 9, 13]
-
-    text = "hey ńöñàśçií how're you?"
-    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
-        "hey",
-        "ńöñàśçií",
-        "how",
-        "'re",
-        "you",
-        "?",
-    ]
-    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 4, 13, 16, 20, 23]
-
-
-def test_spacy_add_cls_token(spacy_nlp):
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-
-    component_config = {"use_cls_token": True}
+import pytest
+
+from rasa.nlu.training_data import TrainingData
+from rasa.nlu.training_data import Message
+from rasa.nlu.constants import (
+    CLS_TOKEN,
+    TEXT_ATTRIBUTE,
+    SPACY_DOCS,
+    INTENT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    TOKENS_NAMES,
+)
+from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch"],
+            [(0, 8), (9, 12), (13, 18)],
+        ),
+        (
+            "hey ńöñàśçií how're you?",
+            ["hey", "ńöñàśçií", "how", "'re", "you", "?"],
+            [(0, 3), (4, 12), (13, 16), (16, 19), (20, 23), (23, 24)],
+        ),
+    ],
+)
+def test_spacy(text, expected_tokens, expected_indices, spacy_nlp):
+    tk = SpacyTokenizer()
+
+    message = Message(text)
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(text))
+
+    tokens = tk.tokenize(message, attribute=TEXT_ATTRIBUTE)
+
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch", CLS_TOKEN],
+            [(0, 8), (9, 12), (13, 18), (19, 26)],
+        )
+    ],
+)
+def test_train_tokenizer(text, expected_tokens, expected_indices, spacy_nlp):
+    tk = SpacyTokenizer()
+
+    message = Message(text)
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(text))
+    message.set(RESPONSE_ATTRIBUTE, text)
+    message.set(SPACY_DOCS[RESPONSE_ATTRIBUTE], spacy_nlp(text))
+
+    training_data = TrainingData()
+    training_data.training_examples = [message]
+
+    tk.train(training_data)
+
+    for attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]:
+        tokens = training_data.training_examples[0].get(TOKENS_NAMES[attribute])
+
+        assert [t.text for t in tokens] == expected_tokens
+        assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+        assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_custom_intent_symbol(text, expected_tokens, spacy_nlp):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
 
     tk = SpacyTokenizer(component_config)
 
-    text = "Forecast for lunch"
-    assert [t.text for t in tk.tokenize(spacy_nlp(text))] == [
-        "Forecast",
-        "for",
-        "lunch",
-        CLS_TOKEN,
-    ]
-    assert [t.offset for t in tk.tokenize(spacy_nlp(text))] == [0, 9, 13, 19]
-
-
-def test_spacy_intent_tokenizer(spacy_nlp_component):
-    from rasa.nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-
-    component_config = {"use_cls_token": False}
-
-    td = training_data.load_data("data/examples/rasa/demo-rasa.json")
-    spacy_nlp_component.train(td, config=None)
-    spacy_tokenizer = SpacyTokenizer(component_config)
-    spacy_tokenizer.train(td, config=None)
+    message = Message(text)
+    message.set(SPACY_DOCS[TEXT_ATTRIBUTE], spacy_nlp(text))
+    message.set(INTENT_ATTRIBUTE, text)
 
-    intent_tokens_exist = [
-        True if example.get("intent_tokens") is not None else False
-        for example in td.intent_examples
-    ]
+    tk.train(TrainingData([message]))
 
-    # no intent tokens should have been set
-    assert not any(intent_tokens_exist)
+    assert [
+        t.text for t in message.get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+    ] == expected_tokens
diff --git a/tests/nlu/tokenizers/test_tokenizer.py b/tests/nlu/tokenizers/test_tokenizer.py
index 2b0e2d8cb32d..b69bc6af7adf 100644
--- a/tests/nlu/tokenizers/test_tokenizer.py
+++ b/tests/nlu/tokenizers/test_tokenizer.py
@@ -1,5 +1,15 @@
 import pytest
 
+from rasa.nlu.constants import (
+    CLS_TOKEN,
+    TEXT_ATTRIBUTE,
+    INTENT_ATTRIBUTE,
+    RESPONSE_ATTRIBUTE,
+    TOKENS_NAMES,
+)
+from rasa.nlu.training_data import Message, TrainingData
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+
 
 def test_tokens_comparison():
     from rasa.nlu.tokenizers.tokenizer import Token
@@ -14,3 +24,80 @@ def test_tokens_comparison():
 
     with pytest.raises(TypeError):
         assert y < "a"
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch", CLS_TOKEN],
+            [(0, 8), (9, 12), (13, 18), (19, 26)],
+        )
+    ],
+)
+def test_train_tokenizer(text, expected_tokens, expected_indices):
+    tk = WhitespaceTokenizer()
+
+    message = Message(text)
+    message.set(RESPONSE_ATTRIBUTE, text)
+    message.set(INTENT_ATTRIBUTE, text)
+
+    training_data = TrainingData()
+    training_data.training_examples = [message]
+
+    tk.train(training_data)
+
+    for attribute in [RESPONSE_ATTRIBUTE, TEXT_ATTRIBUTE]:
+        tokens = training_data.training_examples[0].get(TOKENS_NAMES[attribute])
+
+        assert [t.text for t in tokens] == expected_tokens
+        assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+        assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+    # check intent attribute
+    tokens = training_data.training_examples[0].get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+
+    assert [t.text for t in tokens] == [text]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch", CLS_TOKEN],
+            [(0, 8), (9, 12), (13, 18), (19, 26)],
+        )
+    ],
+)
+def test_process_tokenizer(text, expected_tokens, expected_indices):
+    tk = WhitespaceTokenizer()
+
+    message = Message(text)
+
+    tk.process(message)
+
+    tokens = message.get(TOKENS_NAMES[TEXT_ATTRIBUTE])
+
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
+
+
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_split_intent(text, expected_tokens):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
+
+    tk = WhitespaceTokenizer(component_config)
+
+    message = Message(text)
+    message.set(INTENT_ATTRIBUTE, text)
+
+    assert [t.text for t in tk._split_intent(message)] == expected_tokens
diff --git a/tests/nlu/tokenizers/test_whitespace_tokenizer.py b/tests/nlu/tokenizers/test_whitespace_tokenizer.py
index a8e3b7bdabb4..5a7cff88ebaa 100644
--- a/tests/nlu/tokenizers/test_whitespace_tokenizer.py
+++ b/tests/nlu/tokenizers/test_whitespace_tokenizer.py
@@ -1,152 +1,90 @@
-from rasa.nlu.constants import CLS_TOKEN
+import pytest
+
+from rasa.nlu.constants import TOKENS_NAMES, TEXT_ATTRIBUTE, INTENT_ATTRIBUTE
 from rasa.nlu.training_data import TrainingData, Message
 from tests.nlu import utilities
+from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
 
 
-def test_whitespace():
-    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-
-    component_config = {"use_cls_token": False}
+@pytest.mark.parametrize(
+    "text, expected_tokens, expected_indices",
+    [
+        (
+            "Forecast for lunch",
+            ["Forecast", "for", "lunch"],
+            [(0, 8), (9, 12), (13, 18)],
+        ),
+        (
+            "hey ńöñàśçií how're you?",
+            ["hey", "ńöñàśçií", "how", "re", "you"],
+            [(0, 3), (4, 12), (13, 16), (17, 19), (20, 23)],
+        ),
+        (
+            "https://www.google.com/search?client=safari&rls=en&q=i+like+rasa&ie=UTF-8&oe=UTF-8 https://rasa.com/docs/nlu/components/#tokenizer-whitespace",
+            [
+                "https://www.google.com/search?"
+                "client=safari&rls=en&q=i+like+rasa&ie=UTF-8&oe=UTF-8",
+                "https://rasa.com/docs/nlu/components/#tokenizer-whitespace",
+            ],
+            [(0, 82), (83, 141)],
+        ),
+    ],
+)
+def test_whitespace(text, expected_tokens, expected_indices):
 
-    tk = WhitespaceTokenizer(component_config)
+    tk = WhitespaceTokenizer()
 
-    assert [t.text for t in tk.tokenize("Forecast for lunch")] == [
-        "Forecast",
-        "for",
-        "lunch",
-    ]
-    assert [t.lemma for t in tk.tokenize("Forecast for lunch")] == [
-        "Forecast",
-        "for",
-        "lunch",
-    ]
+    tokens = tk.tokenize(Message(text), attribute=TEXT_ATTRIBUTE)
 
-    assert [t.offset for t in tk.tokenize("Forecast for lunch")] == [0, 9, 13]
+    assert [t.text for t in tokens] == expected_tokens
+    assert [t.start for t in tokens] == [i[0] for i in expected_indices]
+    assert [t.end for t in tokens] == [i[1] for i in expected_indices]
 
-    # we ignore .,!?
-    assert [t.text for t in tk.tokenize("hey ńöñàśçií how're you?")] == [
-        "hey",
-        "ńöñàśçií",
-        "how",
-        "re",
-        "you",
-    ]
 
-    assert [t.offset for t in tk.tokenize("hey ńöñàśçií how're you?")] == [
-        0,
-        4,
-        13,
-        17,
-        20,
-    ]
+@pytest.mark.parametrize(
+    "text, expected_tokens",
+    [
+        ("Forecast_for_LUNCH", ["Forecast_for_LUNCH"]),
+        ("Forecast for LUNCH", ["Forecast for LUNCH"]),
+    ],
+)
+def test_custom_intent_symbol(text, expected_tokens):
+    component_config = {"intent_tokenization_flag": True, "intent_split_symbol": "+"}
 
-    assert [t.text for t in tk.tokenize("привет! 10.000, ńöñàśçií. (how're you?)")] == [
-        "привет",
-        "10.000",
-        "ńöñàśçií",
-        "how",
-        "re",
-        "you",
-    ]
+    tk = WhitespaceTokenizer(component_config)
 
-    assert [
-        t.offset for t in tk.tokenize("привет! 10.000, ńöñàśçií. (how're you?)")
-    ] == [0, 8, 16, 27, 31, 34]
+    message = Message(text)
+    message.set(INTENT_ATTRIBUTE, text)
 
-    # urls are single token
-    assert [
-        t.text
-        for t in tk.tokenize(
-            "https://www.google.com/search?client="
-            "safari&rls=en&q="
-            "i+like+rasa&ie=UTF-8&oe=UTF-8 "
-            "https://rasa.com/docs/nlu/"
-            "components/#tokenizer-whitespace"
-        )
-    ] == [
-        "https://www.google.com/search?"
-        "client=safari&rls=en&q=i+like+rasa&ie=UTF-8&oe=UTF-8",
-        "https://rasa.com/docs/nlu/components/#tokenizer-whitespace",
-    ]
+    tk.train(TrainingData([message]))
 
     assert [
-        t.offset
-        for t in tk.tokenize(
-            "https://www.google.com/search?client="
-            "safari&rls=en&q="
-            "i+like+rasa&ie=UTF-8&oe=UTF-8 "
-            "https://rasa.com/docs/nlu/"
-            "components/#tokenizer-whitespace"
-        )
-    ] == [0, 83]
+        t.text for t in message.get(TOKENS_NAMES[INTENT_ATTRIBUTE])
+    ] == expected_tokens
 
 
-def test_whitespace_cls_token():
-    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-
-    component_config = {"use_cls_token": True}
+@pytest.mark.parametrize(
+    "text, component_config, expected_tokens",
+    [
+        ("Forecast for LUNCH", {}, ["Forecast", "for", "LUNCH"]),
+        ("Forecast for LUNCH", {"case_sensitive": False}, ["forecast", "for", "lunch"]),
+        ("Forecast for LUNCH", {"case_sensitive": True}, ["Forecast", "for", "LUNCH"]),
+    ],
+)
+def test_whitespace_with_case(text, component_config, expected_tokens):
 
     tk = WhitespaceTokenizer(component_config)
 
-    assert [t.text for t in tk.tokenize("Forecast for lunch")] == [
-        "Forecast",
-        "for",
-        "lunch",
-        CLS_TOKEN,
-    ]
-    assert [t.offset for t in tk.tokenize("Forecast for lunch")] == [0, 9, 13, 19]
-
+    message = Message(text)
 
-def test_whitespace_custom_intent_symbol():
-    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+    tokens = tk.tokenize(message, attribute=TEXT_ATTRIBUTE)
 
-    component_config = {
-        "intent_tokenization_flag": True,
-        "intent_split_symbol": "+",
-        "use_cls_token": False,
-    }
+    assert [t.text for t in tokens] == expected_tokens
 
-    tk = WhitespaceTokenizer(component_config)
-
-    assert [t.text for t in tk.tokenize("Forecast_for_LUNCH", attribute="intent")] == [
-        "Forecast_for_LUNCH"
-    ]
-
-    assert [t.text for t in tk.tokenize("Forecast+for+LUNCH", attribute="intent")] == [
-        "Forecast",
-        "for",
-        "LUNCH",
-    ]
-
-
-def test_whitespace_with_case():
-    from rasa.nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-
-    component_config = {"use_cls_token": False}
-    tk = WhitespaceTokenizer(component_config)
-    assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
-        "Forecast",
-        "for",
-        "LUNCH",
-    ]
-
-    component_config = {"case_sensitive": False, "use_cls_token": False}
-    tk = WhitespaceTokenizer(component_config)
-    assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
-        "forecast",
-        "for",
-        "lunch",
-    ]
-
-    component_config = {"case_sensitive": True, "use_cls_token": False}
-    tk = WhitespaceTokenizer(component_config)
-    assert [t.text for t in tk.tokenize("Forecast for LUNCH")] == [
-        "Forecast",
-        "for",
-        "LUNCH",
-    ]
 
+def test_whitespace_training():
     _config = utilities.base_test_conf("supervised_embeddings")
+
     examples = [
         Message(
             "Any Mexican restaurant will do",
@@ -170,7 +108,9 @@ def test_whitespace_with_case():
 
     component_config = {"case_sensitive": False}
     tk = WhitespaceTokenizer(component_config)
+
     tk.train(TrainingData(training_examples=examples), _config)
+
     assert examples[0].data.get("tokens")[0].text == "any"
     assert examples[0].data.get("tokens")[1].text == "mexican"
     assert examples[0].data.get("tokens")[2].text == "restaurant"
diff --git a/tests/nlu/training/test_train.py b/tests/nlu/training/test_train.py
index 3bd187ab3f0e..6085b4451099 100644
--- a/tests/nlu/training/test_train.py
+++ b/tests/nlu/training/test_train.py
@@ -28,6 +28,7 @@ def pipelines_for_tests():
                 "SpacyNLP",
                 "MitieNLP",
                 "WhitespaceTokenizer",
+                "ConveRTTokenizer",
                 "MitieTokenizer",
                 "SpacyTokenizer",
                 "MitieFeaturizer",


Code review comments:
diff_hunk: @@ -36,25 +36,15 @@ methods that you should implement:
     :linenos:
 
 .. note::
-    If you create a custom tokenizer you should have the following entry in your default dict.
-
-    .. code-block:: none
-
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False,
-
-    To add the __CLS__ token you can use the function `add_cls_token()
-    <https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/tokenizers/tokenizer.py#L64/>`_.
+    If you create a custom tokenizer you should implement the methods of ``rasa.nlu.tokenizers.tokenizer.Tokenizer``.
+    The ``train`` and ``process`` methods are already implemented and you simply need to overwrite the ``tokenize``
+    method. ``train`` and ``process`` will automatically add a special token ``__CLS__`` to the end of list of tokens,
+    which is needed further down the pipeline.
 
 .. note::
-    If you create a custom featurizer you should have the following entry in your default dict.
-
-    .. code-block:: none
-
-        # if True return a sequence of features (return vector has size
-        # token-size x feature-dimension)
-        # if False token-size will be equal to 1
-        "return_sequence": False,
+    If you create a custom featurizer you should return a sequence of features.
+    E.g. your featurizer should return a matrix of size (token-size x feature-dimension).
path: docs/api/custom-nlu-components.rst
text: @user1:
```suggestion
    E.g. your featurizer should return a matrix of size (number-of-tokens x feature-dimension).
```

diff_hunk: @@ -23,76 +17,39 @@ class WhitespaceTokenizer(Tokenizer):
         "intent_split_symbol": "_",
         # Text will be tokenized with case sensitive as default
         "case_sensitive": True,
-        # add __CLS__ token to the end of the list of tokens
-        "use_cls_token": False,
     }
 
     def __init__(self, component_config: Dict[Text, Any] = None) -> None:
         """Construct a new tokenizer using the WhitespaceTokenizer framework."""
 
         super().__init__(component_config)
-        # flag to check whether to split intents
-        self.intent_tokenization_flag = self.component_config.get(
-            "intent_tokenization_flag"
-        )
-        # split symbol for intents
-        self.intent_split_symbol = self.component_config["intent_split_symbol"]
-        self.case_sensitive = self.component_config["case_sensitive"]
-
-    def train(
-        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
-    ) -> None:
-        for example in training_data.training_examples:
-            for attribute in MESSAGE_ATTRIBUTES:
-                if example.get(attribute) is not None:
-                    example.set(
-                        TOKENS_NAMES[attribute],
-                        self.tokenize(example.get(attribute), attribute),
-                    )
 
-    def process(self, message: Message, **kwargs: Any) -> None:
-
-        message.set(TOKENS_NAMES[TEXT_ATTRIBUTE], self.tokenize(message.text))
+        self.case_sensitive = self.component_config["case_sensitive"]
 
-    def tokenize(self, text: Text, attribute: Text = TEXT_ATTRIBUTE) -> List[Token]:
+    def tokenize(self, message: Message, attribute: Text) -> List[Token]:
+        text = message.get(attribute)
 
         if not self.case_sensitive:
             text = text.lower()
 
-        if attribute != INTENT_ATTRIBUTE:
-            # remove 'not a word character' if
-            words = re.sub(
-                # there is a space or an end of a string after it
-                r"[^\w#@&]+(?=\s|$)|"
-                # there is a space or beginning of a string before it
-                # not followed by a number
-                r"(\s|^)[^\w#@&]+(?=[^0-9\s])|"
-                # not in between numbers and not . or @ or & or - or #
-                # e.g. 10'000.00 or blabla@gmail.com
-                # and not url characters
-                r"(?<=[^0-9\s])[^\w._~:/?#\[\]()@!$&*+,;=-]+(?=[^0-9\s])",
-                " ",
-                text,
-            ).split()
-            # if we removed everything like smiles `:)`, use the whole text as 1 token
-            if not words:
-                words = [text]
-        else:
-            words = (
-                text.split(self.intent_split_symbol)
-                if self.intent_tokenization_flag
-                else [text]
-            )
-
-        running_offset = 0
-        tokens = []
-
-        for word in words:
-            word_offset = text.index(word, running_offset)
-            word_len = len(word)
-            running_offset = word_offset + word_len
-            tokens.append(Token(word, word_offset))
-
-        self.add_cls_token(tokens, attribute)
-
-        return tokens
+        # remove 'not a word character' if
+        words = re.sub(
+            # there is a space or an end of a string after it
+            r"[^\w#@&]+(?=\s|$)|"
+            # there is a space or beginning of a string before it
+            # not followed by a number
+            r"(\s|^)[^\w#@&]+(?=[^0-9\s])|"
+            # not in between numbers and not . or @ or & or - or #
+            # e.g. 10'000.00 or blabla@gmail.com
+            # and not url characters
+            r"(?<=[^0-9\s])[^\w._~:/?#\[\]()@!$&*+,;=-]+(?=[^0-9\s])",
+            " ",
+            text,
+        ).split()
+        # Fix for convert tokenizer, remove any single '&'
+        words = [w for w in words if w != "&"]
path: rasa/nlu/tokenizers/whitespace_tokenizer.py
text: @user1:
Do we still need this?

@author:
I guess, we will get a "dimensions do not match" kind of error if we remove it. Can quickly test it.

@user1:
If we use only one tokenizer which would be now ConveRT tokenizer, we shouldn't get it I guess. Earlier the problem was that the method of tokenization was different between ConveRTTokenizer and ConveRTFeaturizer. Now that we have it covered in one place only, I think it should be fine. Can you do a quick test please?

@author:
Yes, tested it. We do not need it anymore. Will remove it.

diff_hunk: @@ -0,0 +1,9 @@
+Remove option ``use_cls_token`` from tokenizers and option ``return_sequence`` from featurizers.
+
+By default all tokenizer add a special token (``__CLS__``) to the end of the list of tokens.
+This token will be used to capture the features of the whole utterance.
+
+The featurizers will return a matrix of size (token-length x feature-dimension) by default.
path: changelog/4978.removal.rst
text: @user1:
```suggestion
The featurizers will return a matrix of size (number-of-tokens x feature-dimension) by default.
```
I got confused initially if by `token-length` you meant length of an individual token. Suggested this for clarity.