Repository: RasaHQ/rasa
Pull request: Component based architecture
This PR contains a major refactoring of the architecture. It makes the different parts of the processing more composable and allows us to add new features in a clean way (e.g. sentiment analysis, different intent classifiers...). 

Benefits:
- simpler to add new features
- composable, e.g. you can use what ever intent classifier / entity extractor you want 
- reduced code duplication (in trainers and interpreters)
- testability of the components (still need to write tests for them, though 😄 )

Overview over the proposed changes:
- renamed `src` to `rasa_nlu` (the previous folder layout is uncommon and not supported by mayor IDEs)
- create components out of the existing intent classifiers, entity extractors and tokenizers (every component knows how to: init itself, load from file, store to file, train and process a message)
- refactored trainer & interpreter to be component neutral (they instantiate all the necessary components, feed the data through all of them and gather the result)
- added type annotations and comments to (almost) all public functions
- added ngram-character embeddings from @JoeyFaulkner
- fixes #210 (parse passed string messages)
- fixes #199 (return intent ranking instead of top intent only, only works for sklearn classifier)
- fixes #181 (entity only training)
- fixes #219 (conflicting parameters between metadata and config)

Open:
- [x] update documentation (will be done after change is approved)
- [x] update changelog & migration guide


Problem statement:
Entity Only Training
Sorry, maybe I am missing something, but your documentation states that the rasa_nlu_documentation contains 3 different examples: common training (intent + entity), intent training, and entity training. But under `entity_examples` all of the entity arrays are empty 

`{
        "text": "hey", 
        "intent": "greet", 
        "entities": []
      }, 
      {
        "text": "howdy", 
        "intent": "greet", 
        "entities": []
      }, 
      {
        "text": "hey there", 
        "intent": "greet", 
        "entities": []
      }`

except for these, which look like `common` examples:

`{
        "text": "i'm looking for a place in the north of town", 
        "intent": "restaurant_search", 
        "entities": [
          {
            "start": 31,
            "end": 36, 
            "value": "north", 
            "entity": "location"
          }
        ]
      }`

and then you have intent only examples

`{
        "text": "anywhere in the west", 
        "intent": "restaurant_search"
      }, 
      {
        "text": "central indian restaurant", 
        "intent": "restaurant_search"
      }, 
      {
        "text": "indeed", 
        "intent": "affirm"
      }`

I fail to see how any of these are just training the entities...

Multiple intents response
Hi, I'm coming from LUIS and noticed that interpreters returns only the more likely intent, instead of return the full intent list with scores.
My AI uses, sometimes, this list to make decisions of ambiguities.
rasa_nlu.classifiers.sklearn_intent_classifier.SklearnIntentClassifier#predict_prob return the full list. It's possible to return this list instead?

Fail properly if string is passed to parser instead of unicode
If a a string instead of unicode is passed to the interpreter the interpreter will fail at random points with a type error ala:
```
TypeError: Argument 'string' has incorrect type (expected unicode, got str)
```

This should be properly handled right at the beginning of the parsing (either by converting to unicode or by rejecting the argument).

config.json vs metadata.json
**rasa NLU version:** `0.7.0a7`

**Backend:** `MITIE`

**OS:** `macOS Sierra (latest)`

**Observed behaviour:** When running the command to start the HTTP server, `python -m rasa_nlu.server -c config.json --server_model_dirs=./model_YYYYMMDD-HHMMSS`, rasa takes the configuration cue from `metadata.json` in the model directory, instead of the `config.json` file specified on the commandline.

**Steps to reproduce behaviour:**
I train my models on another box, which has more raw processing power, then download the model to my local workstation to do tests with the trained model. The configuration while training on the server is different than when I work locally. The result is as follows, for `metadata.json`:
```
{
    "trained_at": "20170317-034420", 
    "should_fine_tune_spacy_ner": false, 
    "intent_classifier": "intent_classifier.dat", 
    "training_data": "training_data.json", 
    "language_name": "en", 
    "entity_synonyms": null, 
    "entity_extractor": "entity_extractor.dat", 
    "rasa_nlu_version": "0.7.3", 
    "feature_extractor": "../../total_word_feature_extractor.dat", 
    "backend": "mitie"
}
```
However, back on my local machine, the location of `feature_extractor` is different. As specified in the `config.json` file (the same holds for the `training_data`/`data` config variable):
```
{
  "backend": "mitie",
  "mitie_file": "./data/total_word_feature_extractor.dat",
  "path" : "./models",
  "data" : "./models/model_20170317-034420/traindata.json"
}
```
This throws an error (`Exception: Unable to load total_word_feature_extractor detector from total_word_feature_extractor.dat`), unsurprisingly, because it is locally in a different location/under a different path, vs on the training server.

**Expected behaviour:** I would expect the explicitly specified `config.json` to take precedence over the `metadata.json` configuration variables.

Additionally, though only tangentially related to this issue, it could perhaps help if the `File not found` error could be more explicit than `Could not open file`. But that's a nice-to-have and tangential to the current issue.

I'm open to contributing a fix, but I first wanted to bring the issue to light with the project maintainers.

Hints from issue discussion:
Alright, let's try to improve the documentation then. It states 

> In the rasa NLU data format, there are three lists of examples: common_examples, intent_examples, and entity_examples. The common_examples are used to train both the entity and the intent models. In many cases it’s fine to put all of your training examples in there. However, if you need lots and lots of examples to train a good entity recogniser, that can mess up your intent model because your classes are totally unbalanced. In that case it makes sense to split up these lists.

Can you explain what is left unclear?
How do I train entities only? if I need `Lots and lots of examples to train a good entity recognizer` what would that look like. In the docs/examples you show intent only training and intent + entity, but you never show entity only.

What are labels as entity examples are in fact entity + intent.
@wrathagom Currently that is not possible. (You might put all your examples into the `entity_examples` section and leave the `intent_examples` section empty, but I am not sure what will happen if you try to train that).

I am currently working on a pipeline implementation that allows you to specify the different components separately, hence you would be able to skip the intent classification step entirely. 
Works in the pipeline implementation, returned intent will be `null`.
I think we can make that an option, do you mind sharing the LUIS format?
could you please help me with the format for multiple intents?
I am not quite sure what you are referring to. Do you mean the implementation of returning a list of intents including probability instead of the most probable one?
yea, i mean the implementation of returning a list of intents including probability.
What is the format LUIS is using for this?
This is an example

```json
{
    "query": "us$ 5000.00",
    "topScoringIntent": {
        "intent": "Money Answer",
        "score": 0.9999938
    },
    "intents": [
        {
            "intent": "Money Answer",
            "score": 0.9999938
        },
        {
            "intent": "Number Answer",
            "score": 0.04435608
        }
    ],
    "entities": [
        {
            "entity": "us $",
            "type": "currency",
            "startIndex": 0,
            "endIndex": 2,
            "score": 0.9984345
        },
        {
            "entity": "5000 . 00",
            "type": "number",
            "startIndex": 4,
            "endIndex": 10,
            "score": 0.9975406
        }
    ]
}
```
Implemented on `pipeline` branch. Only works when sklearn intent classifier is used (mitie doesn't return the probabilities for all classes).

Commit message: fixed pepe8 error

Patch to review:
diff --git a/.gitignore b/.gitignore
index 94cef71d1a97..ae759afd19a8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,11 +5,9 @@ data/*
 !data/examples
 docs/_build
 server/
-*.log
 scala/
 mongodb/
 .cache/
-build/lib/
 build/
 *.egg-info/
 jnk/
@@ -26,13 +24,9 @@ lastmile_ai/learn/plots/
 .coveralls.yml
 .idea/
 *.iml
-rasa_nlu
 out/
 .vscode/
-rasa_nlu_logs.json
 tmp_training_data.json
 .DS_Store
 models/
 test_models/model_*
-tmp_config_file.json
-rasa_nlu_log-*
diff --git a/_pytest/conftest.py b/_pytest/conftest.py
index 2bb1157f7d4f..fda589fbb0f6 100644
--- a/_pytest/conftest.py
+++ b/_pytest/conftest.py
@@ -1,7 +1,25 @@
+import logging
 import pytest
 import spacy
+from mitie import total_word_feature_extractor
+
+from rasa_nlu.data_router import InterpreterBuilder
+
+
+logging.basicConfig(level="DEBUG")
 
 
 @pytest.fixture(scope="session")
 def spacy_nlp_en():
     return spacy.load("en", parser=False)
+
+
+@pytest.fixture(scope="session")
+def mitie_feature_extractor():
+    mitie_file = "data/total_word_feature_extractor.dat"
+    return total_word_feature_extractor(mitie_file)
+
+
+@pytest.fixture(scope="session")
+def interpreter_builder():
+    return InterpreterBuilder(use_cache=True)
diff --git a/_pytest/test_components.py b/_pytest/test_components.py
new file mode 100644
index 000000000000..30a0811dd3c7
--- /dev/null
+++ b/_pytest/test_components.py
@@ -0,0 +1,70 @@
+import pytest
+
+from rasa_nlu import config
+from rasa_nlu import registry
+from rasa_nlu.components import Component, fill_args
+
+
+@pytest.mark.parametrize("component_class", registry.component_classes)
+def test_component_init_without_args(component_class):
+    component = component_class()
+    assert issubclass(component.__class__, Component), \
+        "Either component can't be initialized without args, or it doesn't subclass component but is registered as one."
+
+
+@pytest.mark.parametrize("component_class", registry.component_classes)
+def test_no_components_with_same_name(component_class):
+    names = map(lambda cls: cls.name, registry.component_classes)
+    assert names.count(component_class.name) == 1, \
+        "There is more than one component named {}".format(component_class.name)
+
+
+@pytest.mark.parametrize("model_template", registry.registered_model_templates)
+def test_all_components_in_mode_templates_exist(model_template):
+    components = registry.registered_model_templates[model_template]
+    for component in components:
+        assert component in registry.registered_components, "Model template contains unknown component."
+
+
+@pytest.mark.parametrize("component_class", registry.component_classes)
+def test_all_arguments_can_be_satisfied_during_init(component_class):
+    # All available context arguments that will ever be generated during init
+    component = component_class()
+    context_arguments = {}
+    for clz in registry.component_classes:
+        for ctx_arg in clz.context_provides.get("pipeline_init", []):
+            context_arguments[ctx_arg] = None
+
+    filled_args = fill_args(component.pipeline_init_args(), context_arguments, config.DEFAULT_CONFIG)
+    assert len(filled_args) == len(component.pipeline_init_args())
+
+
+@pytest.mark.parametrize("component_class", registry.component_classes)
+def test_all_arguments_can_be_satisfied_during_train(component_class):
+    # All available context arguments that will ever be generated during train
+    # it might still happen, that in a certain pipeline configuration arguments can not be satisfied!
+    component = component_class()
+    context_arguments = {"training_data": None}
+    for clz in registry.component_classes:
+        for ctx_arg in clz.context_provides.get("pipeline_init", []):
+            context_arguments[ctx_arg] = None
+        for ctx_arg in clz.context_provides.get("train", []):
+            context_arguments[ctx_arg] = None
+
+    filled_args = fill_args(component.train_args(), context_arguments, config.DEFAULT_CONFIG)
+    assert len(filled_args) == len(component.train_args())
+
+
+@pytest.mark.parametrize("component_class", registry.component_classes)
+def test_all_arguments_can_be_satisfied_during_parse(component_class):
+    # All available context arguments that will ever be generated during parse
+    component = component_class()
+    context_arguments = {"text": None}
+    for clz in registry.component_classes:
+        for ctx_arg in clz.context_provides.get("pipeline_init", []):
+            context_arguments[ctx_arg] = None
+        for ctx_arg in clz.context_provides.get("process", []):
+            context_arguments[ctx_arg] = None
+
+    filled_args = fill_args(component.process_args(), context_arguments, config.DEFAULT_CONFIG)
+    assert len(filled_args) == len(component.process_args())
diff --git a/_pytest/test_config.py b/_pytest/test_config.py
index f9175c7f7410..44fc601ebd6b 100644
--- a/_pytest/test_config.py
+++ b/_pytest/test_config.py
@@ -6,7 +6,7 @@
 import logging
 
 defaults = {
-  "backend": "mitie",
+  "model_template": None,
   "config": "config.json",
   "data": None,
   "emulate": None,
@@ -16,11 +16,14 @@
   "mitie_file": os.path.join("data", "total_word_feature_extractor.dat"),
   "num_threads": 1,
   "fine_tune_spacy_ner": False,
+  "max_number_of_ngrams": 7,
   "path": os.path.join(os.getcwd(), "models"),
   "port": 5000,
   "server_model_dirs": None,
   "token": None,
-  "response_log": os.path.join(os.getcwd(), "logs")
+  "pipeline": [],
+  "response_log": os.path.join(os.getcwd(), "logs"),
+  "luis_data_tokenizer": None,
 }
 
 
diff --git a/_pytest/test_emulators.py b/_pytest/test_emulators.py
index 07d5fcbb84bf..660274a61077 100644
--- a/_pytest/test_emulators.py
+++ b/_pytest/test_emulators.py
@@ -10,8 +10,7 @@ def test_luis_response():
     em = LUISEmulator()
     data = {
         "text": "I want italian food",
-        "intent": "inform",
-        "confidence": 0.4794813722432127,
+        "intent": {"name": "inform", "confidence": 0.4794813722432127},
         "entities": [{"entity": "cuisine", "value": "italian"}]
     }
     norm = em.normalise_response_json(data)
@@ -21,6 +20,7 @@ def test_luis_response():
             "intent": "inform",
             "score": 0.4794813722432127
         },
+        "intents": [{"intent": "inform", "score": 0.4794813722432127}],
         "entities": [
             {
                 "entity": e["value"],
@@ -45,8 +45,7 @@ def test_wit_response():
     em = WitEmulator()
     data = {
         "text": "I want italian food",
-        "intent": "inform",
-        "confidence": 0.4794813722432127,
+        "intent": {"name": "inform", "confidence": 0.4794813722432127},
         "entities": [{"entity": "cuisine", "value": "italian", "start": 7, "end": 14}]}
     norm = em.normalise_response_json(data)
     assert norm == [{
diff --git a/_pytest/test_featurizers.py b/_pytest/test_featurizers.py
index 1d3c9a6a8cc9..f2779b814ec9 100644
--- a/_pytest/test_featurizers.py
+++ b/_pytest/test_featurizers.py
@@ -3,7 +3,7 @@
 import numpy as np
 import pytest
 
-from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
+from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
 
 
 @pytest.mark.parametrize("sentence, expected", [
@@ -11,22 +11,22 @@
 ])
 def test_spacy_featurizer(spacy_nlp_en, sentence, expected):
     from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-    ftr = SpacyFeaturizer(spacy_nlp_en)
+    ftr = SpacyFeaturizer()
     doc = spacy_nlp_en(sentence)
-    vecs = ftr.features_for_doc(doc)
+    vecs = ftr.features_for_doc(doc, spacy_nlp_en)
     assert np.allclose(doc.vector[:5], expected, atol=1e-5)
     assert np.allclose(vecs, doc.vector, atol=1e-5)
 
 
-def test_mitie_featurizer():
-    from rasa_nlu.featurizers.mitie_featurizer import MITIEFeaturizer
+def test_mitie_featurizer(mitie_feature_extractor):
+    from rasa_nlu.featurizers.mitie_featurizer import MitieFeaturizer
 
     filename = os.environ.get('MITIE_FILE')
     if not filename or not os.path.isfile(filename):
-        filename = "data/total_word_feature_extractor.dat"
+        filename = os.path.join("data", "total_word_feature_extractor.dat")
 
-    ftr = MITIEFeaturizer(filename)
+    ftr = MitieFeaturizer.load(filename)
     sentence = "Hey how are you today"
-    tokens = MITIETokenizer().tokenize(sentence)
-    vecs = ftr.features_for_tokens(tokens)
+    tokens = MitieTokenizer().tokenize(sentence)
+    vecs = ftr.features_for_tokens(tokens, mitie_feature_extractor)
     assert np.allclose(vecs[:5], np.array([0., -4.4551446, 0.26073121, -1.46632245, -1.84205751]), atol=1e-5)
diff --git a/_pytest/test_interpreter.py b/_pytest/test_interpreter.py
index 3fd820227a0f..afd82af85c5e 100644
--- a/_pytest/test_interpreter.py
+++ b/_pytest/test_interpreter.py
@@ -1,19 +1,13 @@
 import pytest
 
 import utilities
-from rasa_nlu.utils.mitie import MITIE_BACKEND_NAME
-from rasa_nlu.utils.mitie import MITIE_SKLEARN_BACKEND_NAME
-from rasa_nlu.utils.spacy import SPACY_BACKEND_NAME
+from rasa_nlu import registry
 
 
-@pytest.mark.parametrize("backend_name", [
-    MITIE_BACKEND_NAME,
-    MITIE_SKLEARN_BACKEND_NAME,
-    SPACY_BACKEND_NAME,
-])
-def test_samples(backend_name, spacy_nlp_en):
-    interpreter = utilities.interpreter_for(spacy_nlp_en, utilities.base_test_conf(backend_name))
-    available_intents = ["greet", "restaurant_search", "affirm", "goodbye"]
+@pytest.mark.parametrize("pipeline_name", registry.registered_model_templates.keys())
+def test_samples(pipeline_name, interpreter_builder):
+    interpreter = utilities.interpreter_for(interpreter_builder, utilities.base_test_conf(pipeline_name))
+    available_intents = ["greet", "restaurant_search", "affirm", "goodbye", "None"]
     samples = [
         (
             u"good bye",
@@ -35,9 +29,9 @@ def test_samples(backend_name, spacy_nlp_en):
         result = interpreter.parse(text)
         assert result['text'] == text, \
             "Wrong text for sample '{}'".format(text)
-        assert result['intent'] in available_intents, \
+        assert result['intent']['name'] in available_intents, \
             "Wrong intent for sample '{}'".format(text)
-        assert result['confidence'] >= 0, \
+        assert result['intent']['confidence'] >= 0, \
             "Low confidence for sample '{}'".format(text)
 
         # This ensures the model doesn't detect entities that are not present
diff --git a/_pytest/test_multitenancy.py b/_pytest/test_multitenancy.py
index f2f188d4a1d2..8807a7c86e91 100644
--- a/_pytest/test_multitenancy.py
+++ b/_pytest/test_multitenancy.py
@@ -6,6 +6,7 @@
 import pytest
 
 from rasa_nlu.config import RasaNLUConfig
+from rasa_nlu.model import Trainer
 from rasa_nlu.server import create_app
 from utilities import ResponseTest
 
@@ -52,7 +53,7 @@ def app():
 def test_get_parse(client, response_test):
     response = client.get(response_test.endpoint)
     assert response.status_code == 200
-    assert all(prop in response.json for prop in ['entities', 'intent', 'text', 'confidence'])
+    assert all(prop in response.json for prop in ['entities', 'intent', 'text'])
 
 
 @pytest.mark.parametrize("response_test", [
@@ -92,7 +93,7 @@ def test_post_parse(client, response_test):
     response = client.post(response_test.endpoint,
                            data=json.dumps(response_test.payload), content_type='application/json')
     assert response.status_code == 200
-    assert all(prop in response.json for prop in ['entities', 'intent', 'text', 'confidence'])
+    assert all(prop in response.json for prop in ['entities', 'intent', 'text'])
 
 
 @pytest.mark.parametrize("response_test", [
@@ -117,16 +118,16 @@ def test_post_parse_invalid_model(client, response_test):
 if __name__ == '__main__':
     # Retrain different multitenancy models
     def train(cfg_name, model_name):
-        from rasa_nlu.train import create_trainer
         from rasa_nlu.train import create_persistor
-        from rasa_nlu.training_data import TrainingData
+        from rasa_nlu.converters import load_data
 
         config = RasaNLUConfig(cfg_name)
-        trainer = create_trainer(config)
-        persistor = create_persistor(config)
+        trainer = Trainer(config)
+        training_data = load_data(config['data'], config['language'])
 
-        training_data = TrainingData(config['data'], config['backend'], nlp=trainer.nlp)
+        trainer.validate()
         trainer.train(training_data)
+        persistor = create_persistor(config)
         trainer.persist(os.path.join("test_models", model_name), persistor, create_unique_subfolder=False)
 
     train("config_mitie.json", "test_model_mitie")
diff --git a/_pytest/test_server.py b/_pytest/test_server.py
index b9585859ae1a..581af86676f5 100644
--- a/_pytest/test_server.py
+++ b/_pytest/test_server.py
@@ -21,6 +21,7 @@ def app():
         "path": "./models",
         "server_model_dirs": {},
         "data": "./data/demo-restaurants.json",
+        "luis_data_tokenizer": "tokenizer_mitie",
         "emulate": "wit",
     }
     config = RasaNLUConfig(cmdline_args=_config)
diff --git a/_pytest/test_synonyms.py b/_pytest/test_synonyms.py
index ccf775f8a882..83ac036b6b30 100644
--- a/_pytest/test_synonyms.py
+++ b/_pytest/test_synonyms.py
@@ -1,4 +1,4 @@
-from rasa_nlu import Interpreter
+from rasa_nlu.extractors.entity_synonyms import EntitySynonymMapper
 
 
 def test_entity_synonyms():
@@ -19,7 +19,7 @@ def test_entity_synonyms():
         "end": 6
     }]
     ent_synonyms = {"chines": "chinese", "NYC": "New York City"}
-    Interpreter.replace_synonyms(entities, ent_synonyms)
+    EntitySynonymMapper(ent_synonyms).replace_synonyms(entities)
     assert len(entities) == 3
     assert entities[0]["value"] == "chinese"
     assert entities[1]["value"] == "chinese"
diff --git a/_pytest/test_tokenizers.py b/_pytest/test_tokenizers.py
index c14ecb783871..00a2c3f6ba65 100644
--- a/_pytest/test_tokenizers.py
+++ b/_pytest/test_tokenizers.py
@@ -19,8 +19,8 @@ def tokenize_sentence(sentence, expected_result):
 
 
 def test_mitie():
-    from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
-    tk = MITIETokenizer()
+    from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+    tk = MitieTokenizer()
 
     assert tk.tokenize(u"Hi. My name is rasa") == [u'Hi', u'My', u'name', u'is', u'rasa']
     assert tk.tokenize(u"ὦ ἄνδρες ᾿Αθηναῖοι") == [u'ὦ', u'ἄνδρες', u'᾿Αθηναῖοι']
diff --git a/_pytest/test_train.py b/_pytest/test_train.py
index 1e66e0bdcaa7..87010a66a2cb 100644
--- a/_pytest/test_train.py
+++ b/_pytest/test_train.py
@@ -1,79 +1,48 @@
 # -*- coding: utf-8 -*-
-import tempfile
 
 import pytest
 
-from rasa_nlu.utils.mitie import MITIE_BACKEND_NAME, MITIE_SKLEARN_BACKEND_NAME
-from rasa_nlu.utils.spacy import SPACY_BACKEND_NAME
 import utilities
+from rasa_nlu import registry
 
 
-def base_test_conf(backend):
-    return {
-        'write': utilities.temp_log_file_location(),
-        'port': 5022,
-        "backend": backend,
-        "path": tempfile.mkdtemp(),
-        "data": "./data/examples/rasa/demo-rasa.json"
-    }
-
-
-@pytest.mark.parametrize("backend_name", [
-    MITIE_BACKEND_NAME,
-    MITIE_SKLEARN_BACKEND_NAME,
-    SPACY_BACKEND_NAME,
-    ])
-def test_train_backend(backend_name, spacy_nlp_en):
-    _config = base_test_conf(backend_name)
+@pytest.mark.parametrize("template_name", registry.registered_model_templates.keys())
+def test_train_model(template_name, interpreter_builder):
+    _config = utilities.base_test_conf(template_name)
     (trained, persisted_path) = utilities.run_train(_config)
-    assert trained.entity_extractor is not None
-    assert trained.intent_classifier is not None
-    loaded = utilities.load_interpreter_for_model(spacy_nlp_en, _config, persisted_path)
-    assert loaded.extractor is not None
-    assert loaded.classifier is not None
+    assert trained.pipeline
+    loaded = utilities.load_interpreter_for_model(_config, persisted_path, interpreter_builder)
+    assert loaded.pipeline
 
 
-@pytest.mark.parametrize("backend_name", [
-    MITIE_BACKEND_NAME,
-    MITIE_SKLEARN_BACKEND_NAME,
-    SPACY_BACKEND_NAME,
-])
-def test_train_backend_noents(backend_name, spacy_nlp_en):
-    _config = base_test_conf(backend_name)
+@pytest.mark.parametrize("pipeline_name", registry.registered_model_templates.keys())
+def test_train_model_noents(pipeline_name, interpreter_builder):
+    _config = utilities.base_test_conf(pipeline_name)
     _config['data'] = "./data/examples/rasa/demo-rasa-noents.json"
     (trained, persisted_path) = utilities.run_train(_config)
-    assert trained.entity_extractor is None
-    assert trained.intent_classifier is not None
-    loaded = utilities.load_interpreter_for_model(spacy_nlp_en, _config, persisted_path)
-    assert loaded.extractor is None
-    assert loaded.classifier is not None
+    assert trained.pipeline
+    loaded = utilities.load_interpreter_for_model(_config, persisted_path, interpreter_builder)
+    assert loaded.pipeline
 
 
-@pytest.mark.parametrize("backend_name", [
-    MITIE_BACKEND_NAME,
-    SPACY_BACKEND_NAME,
-])
-def test_train_backend_multithread(backend_name, spacy_nlp_en):
-    # basic conf
-    _config = base_test_conf(backend_name)
+@pytest.mark.parametrize("template_name", registry.registered_model_templates.keys())
+def test_train_model_multithread(template_name, interpreter_builder):
+    _config = utilities.base_test_conf(template_name)
     _config['num_threads'] = 2
     (trained, persisted_path) = utilities.run_train(_config)
-    assert trained.entity_extractor is not None
-    assert trained.intent_classifier is not None
-    loaded = utilities.load_interpreter_for_model(spacy_nlp_en, _config, persisted_path)
-    assert loaded.extractor is not None
-    assert loaded.classifier is not None
+    assert trained.pipeline
+    loaded = utilities.load_interpreter_for_model(_config, persisted_path, interpreter_builder)
+    assert loaded.pipeline
 
 
-def test_train_spacy_sklearn_finetune_ner():
-    # basic conf
-    _config = base_test_conf(SPACY_BACKEND_NAME)
+def test_train_spacy_sklearn_finetune_ner(interpreter_builder):
+    _config = utilities.base_test_conf("spacy_sklearn")
     _config['fine_tune_spacy_ner'] = True
-    (trained, _) = utilities.run_train(_config)
-    assert trained.entity_extractor is not None
-    assert trained.intent_classifier is not None
-    doc = trained.nlp(u"I am living in New York now.")
-    entities = trained.entity_extractor.extract_entities(doc)
+    (trained, persisted_path) = utilities.run_train(_config)
+    assert trained.pipeline
+    loaded = utilities.load_interpreter_for_model(_config, persisted_path, interpreter_builder)
+    result = loaded.parse(u"I am living in New York now.")
+    entities = result['entities']
     # Although the model is trained on restaurant entities, we can use the entities (`GPE`, `DATE`)
     # from spacy since we are fine tuning. This should even be the case if the rasa-entity training data changes!
     assert {u'start': 15, u'end': 23, u'value': u'New York', u'entity': u'GPE'} in entities
diff --git a/_pytest/test_training_data.py b/_pytest/test_training_data.py
index 18aae300b093..f7882a0eae14 100644
--- a/_pytest/test_training_data.py
+++ b/_pytest/test_training_data.py
@@ -1,36 +1,50 @@
 # -*- coding: utf-8 -*-
-import os
 import tempfile
-import spacy
 
-from rasa_nlu.training_data import TrainingData
-from rasa_nlu.trainers import mitie_trainer_utils
+import pytest
 
+from rasa_nlu.converters import load_data
+from rasa_nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
 
-def test_luis_mitie():
-    td = TrainingData('data/examples/luis/demo-restaurants.json', 'mitie')
-    assert td.fformat == 'luis'
-    # some more assertions
 
+def test_luis_data_spacy():
+    td = load_data('data/examples/luis/demo-restaurants.json', "en", "tokenizer_spacy")
+    assert td.entity_examples != []
+    assert td.intent_examples != []
+    assert td.entity_synonyms == {}
 
-def test_wit_spacy(spacy_nlp_en):
-    td = TrainingData('data/examples/wit/demo-flights.json', 'spacy_sklearn', nlp=spacy_nlp_en)
-    assert td.fformat == 'wit'
 
+def test_luis_data_mitie():
+    td = load_data('data/examples/luis/demo-restaurants.json', "en", "tokenizer_mitie")
+    assert td.entity_examples != []
+    assert td.intent_examples != []
+    assert td.entity_synonyms == {}
 
-def test_rasa_whitespace():
-    td = TrainingData('data/examples/rasa/demo-rasa.json', '', 'en')
-    assert td.fformat == 'rasa_nlu'
 
+def test_luis_data_without_tokenizer():
+    with pytest.raises(ValueError):
+        load_data('data/examples/luis/demo-restaurants.json', "en")
 
-def test_api_mitie():
-    td = TrainingData('data/examples/api/', 'mitie', 'en')
-    assert td.fformat == 'api'
 
+def test_wit_data():
+    td = load_data('data/examples/wit/demo-flights.json', "en")
+    assert td.entity_examples != []
+    assert td.intent_examples == []
+    assert td.entity_synonyms == {}
 
-def test_api_mitie_sklearn():
-    td = TrainingData('data/examples/api/', 'mitie_sklearn', 'en')
-    assert td.fformat == 'api'
+
+def test_rasa_data():
+    td = load_data('data/examples/rasa/demo-rasa.json', "en")
+    assert td.entity_examples != []
+    assert td.intent_examples != []
+    assert td.entity_synonyms == {}
+
+
+def test_api_data():
+    td = load_data('data/examples/api/', "en")
+    assert td.entity_examples != []
+    assert td.intent_examples != []
+    assert td.entity_synonyms != {}
 
 
 def test_repeated_entities():
@@ -56,12 +70,12 @@ def test_repeated_entities():
     with tempfile.NamedTemporaryFile(suffix="_tmp_training_data.json") as f:
         f.write(data.encode("utf-8"))
         f.flush()
-        td = TrainingData(f.name, 'mitie', 'en')
+        td = load_data(f.name, "en")
         assert len(td.entity_examples) == 1
         example = td.entity_examples[0]
         entities = example["entities"]
         assert len(entities) == 1
-        start, end = mitie_trainer_utils.find_entity(entities[0], example["text"])
+        start, end = MitieEntityExtractor.find_entity(entities[0], example["text"])
         assert start == 9
         assert end == 10
 
@@ -89,12 +103,12 @@ def test_multiword_entities():
     with tempfile.NamedTemporaryFile(suffix="_tmp_training_data.json") as f:
         f.write(data.encode("utf-8"))
         f.flush()
-        td = TrainingData(f.name, 'mitie', 'en')
+        td = load_data(f.name, "en")
         assert len(td.entity_examples) == 1
         example = td.entity_examples[0]
         entities = example["entities"]
         assert len(entities) == 1
-        start, end = mitie_trainer_utils.find_entity(entities[0], example["text"])
+        start, end = MitieEntityExtractor.find_entity(entities[0], example["text"])
         assert start == 4
         assert end == 7
 
@@ -120,7 +134,7 @@ def test_nonascii_entities():
     with tempfile.NamedTemporaryFile(suffix="_tmp_training_data.json") as f:
         f.write(data.encode("utf-8"))
         f.flush()
-        td = TrainingData(f.name, 'mitie', 'en')
+        td = load_data(f.name, "en", luis_data_tokenizer="tokenizer_mitie")
         assert len(td.entity_examples) == 1
         example = td.entity_examples[0]
         entities = example["entities"]
@@ -132,41 +146,40 @@ def test_nonascii_entities():
         assert entity["entity"] == "description"
 
 
-def test_entities_synonyms():
-    data = u"""
-{
-  "rasa_nlu_data": {
-    "common_examples" : [
-      {
-        "text": "show me flights to New York City",
-        "intent": "unk",
-        "entities": [
-          {
-            "entity": "destination",
-            "start": 19,
-            "end": 32,
-            "value": "NYC"
-          }
-        ]
-      },
-      {
-        "text": "show me flights to NYC",
-        "intent": "unk",
-        "entities": [
-          {
-            "entity": "destination",
-            "start": 19,
-            "end": 22,
-            "value": "NYC"
-          }
-        ]
-      }
-    ]
-  }
-}"""
-    with tempfile.NamedTemporaryFile(suffix="_tmp_training_data.json") as f:
-        f.write(data.encode("utf-8"))
-        f.flush()
-        td = TrainingData(f.name, 'mitie', 'en')
-        assert len(td.entity_synonyms) == 1
-        assert td.entity_synonyms["new york city"] == "nyc"
+# def test_entities_synonyms():
+#     data = u"""
+# {
+#   "rasa_nlu_data": {
+#     "common_examples" : [
+#       {
+#         "text": "show me flights to New York City",
+#         "intent": "unk",
+#         "entities": [
+#           {
+#             "entity": "destination",
+#             "start": 19,
+#             "end": 32,
+#             "value": "NYC"
+#           }
+#         ]
+#       },
+#       {
+#         "text": "show me flights to NYC",
+#         "intent": "unk",
+#         "entities": [
+#           {
+#             "entity": "destination",
+#             "start": 19,
+#             "end": 22,
+#             "value": "NYC"
+#           }
+#         ]
+#       }
+#     ]
+#   }
+# }"""
+#     with tempfile.NamedTemporaryFile(suffix="_tmp_training_data.json") as f:
+#         f.write(data.encode("utf-8"))
+#         f.flush()
+#         td = load_data(f.name, "en")
+#         assert td.entity_synonyms["new york city"] == "nyc"
diff --git a/_pytest/utilities.py b/_pytest/utilities.py
index c8fce894b428..0dbde682c418 100644
--- a/_pytest/utilities.py
+++ b/_pytest/utilities.py
@@ -1,39 +1,39 @@
 import tempfile
 
+from rasa_nlu import registry
 from rasa_nlu.config import RasaNLUConfig
 from rasa_nlu.data_router import DataRouter
 from rasa_nlu.train import do_train
 
 
-def base_test_conf(backend):
-    return {
-        'write': temp_log_file_location(),
+def base_test_conf(pipeline_name):
+    return RasaNLUConfig(cmdline_args={
+        'response_log': temp_log_file_dir(),
         'port': 5022,
-        "backend": backend,
+        "pipeline": registry.registered_model_templates[pipeline_name],
         "path": tempfile.mkdtemp(),
         "data": "./data/examples/rasa/demo-rasa.json"
-    }
+    })
 
 
-def interpreter_for(nlp, config):
+def interpreter_for(interpreter_builder, config):
     (trained, path) = run_train(config)
-    interpreter = load_interpreter_for_model(nlp, config, path)
+    interpreter = load_interpreter_for_model(config, path, interpreter_builder)
     return interpreter
 
 
-def temp_log_file_location():
-    return tempfile.mkstemp(suffix="_rasa_nlu_logs.json")[1]
+def temp_log_file_dir():
+    return tempfile.mkdtemp(suffix="_rasa_nlu_logs")
 
 
-def run_train(_config):
-    config = RasaNLUConfig(cmdline_args=_config)
+def run_train(config):
     (trained, path) = do_train(config)
     return trained, path
 
 
-def load_interpreter_for_model(nlp, config, persisted_path):
+def load_interpreter_for_model(config, persisted_path, interpreter_builder):
     metadata = DataRouter.read_model_metadata(persisted_path, config)
-    return DataRouter.create_interpreter(metadata, nlp)
+    return interpreter_builder.create_interpreter(metadata, config)
 
 
 class ResponseTest(object):
diff --git a/config_mitie.json b/config_mitie.json
index 7fb61fce0158..89154f36248a 100644
--- a/config_mitie.json
+++ b/config_mitie.json
@@ -1,5 +1,5 @@
 {
-  "backend": "mitie",
+  "model_template": "mitie",
   "mitie_file": "./data/total_word_feature_extractor.dat",
   "path" : "./models",
   "data" : "./data/examples/rasa/demo-rasa.json"
diff --git a/config_mitie_sklearn.json b/config_mitie_sklearn.json
index 710230237417..a7cfe0028a24 100644
--- a/config_mitie_sklearn.json
+++ b/config_mitie_sklearn.json
@@ -1,5 +1,5 @@
 {
-  "backend": "mitie_sklearn",
+  "model_template": "mitie_sklearn",
   "mitie_file": "./data/total_word_feature_extractor.dat",
   "path" : "./models",
   "data" : "./data/examples/rasa/demo-rasa.json"
diff --git a/config_spacy.json b/config_spacy.json
index 2fa266dad74e..0ad73d4b33a1 100644
--- a/config_spacy.json
+++ b/config_spacy.json
@@ -1,5 +1,5 @@
 {
-  "backend": "spacy_sklearn",
+  "model_template": "spacy_sklearn",
   "path" : "./models",
   "data" : "./data/examples/rasa/demo-rasa.json"
 }
diff --git a/docs/conf.py b/docs/conf.py
index 781ca514b1d8..41d9b6c4735c 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -12,8 +12,6 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys
-import os
 import sphinx_rtd_theme
 
 # If extensions (or modules to document with autodoc) are in another directory,
diff --git a/rasa_nlu/__init__.py b/rasa_nlu/__init__.py
new file mode 100644
index 000000000000..9a70bea8708c
--- /dev/null
+++ b/rasa_nlu/__init__.py
@@ -0,0 +1,3 @@
+import rasa_nlu.version
+
+__version__ = version.__version__
diff --git a/src/classifiers/__init__.py b/rasa_nlu/classifiers/__init__.py
similarity index 100%
rename from src/classifiers/__init__.py
rename to rasa_nlu/classifiers/__init__.py
diff --git a/rasa_nlu/classifiers/keyword_intent_classifier.py b/rasa_nlu/classifiers/keyword_intent_classifier.py
new file mode 100644
index 000000000000..17ef791e61a2
--- /dev/null
+++ b/rasa_nlu/classifiers/keyword_intent_classifier.py
@@ -0,0 +1,40 @@
+from rasa_nlu.components import Component
+
+
+class KeywordIntentClassifier(Component):
+
+    name = "intent_classifier_keyword"
+
+    context_provides = {
+        "process": ["intent"],
+    }
+
+    output_provides = ["intent"]
+
+    his = ["hello", "hi", "hey"]
+
+    byes = ["bye", "goodbye"]
+
+    def process(self, text):
+        # type: (str) -> dict
+
+        return {
+            "intent": {
+                "name": self.parse(text),
+                "confidence": 1.0,
+            }
+        }
+
+    def parse(self, text):
+        # type: (str) -> str
+
+        _text = text.lower()
+
+        def is_present(x): return x in _text
+
+        if any(map(is_present, self.his)):
+            return "greet"
+        elif any(map(is_present, self.byes)):
+            return "goodbye"
+        else:
+            return "None"
diff --git a/rasa_nlu/classifiers/mitie_intent_classifier.py b/rasa_nlu/classifiers/mitie_intent_classifier.py
new file mode 100644
index 000000000000..e2372e532308
--- /dev/null
+++ b/rasa_nlu/classifiers/mitie_intent_classifier.py
@@ -0,0 +1,66 @@
+import os
+
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class MitieIntentClassifier(Component):
+
+    name = "intent_classifier_mitie"
+
+    context_provides = {
+        "process": ["intent"],
+    }
+
+    output_provides = ["intent"]
+
+    def __init__(self, clf=None):
+        self.clf = clf
+
+    def train(self, training_data, mitie_file, num_threads):
+        # type: (TrainingData, str, Optional[int]) -> None
+        from mitie import tokenize, text_categorizer_trainer
+
+        trainer = text_categorizer_trainer(mitie_file)
+        trainer.num_threads = num_threads
+        for example in training_data.intent_examples:
+            tokens = tokenize(example["text"])
+            trainer.add_labeled_text(tokens, example["intent"])
+        self.clf = trainer.train()
+
+    def process(self, tokens, mitie_feature_extractor):
+        # type: ([str], mitie.total_word_feature_extractor) -> object
+        import mitie
+
+        intent, score = self.clf(tokens, mitie_feature_extractor)
+        return {
+            "intent": {
+                "name": intent,
+                "confidence": score,
+            }
+        }
+
+    @classmethod
+    def load(cls, model_dir, intent_classifier):
+        # type: (str, str) -> MitieIntentClassifier
+        from mitie import text_categorizer
+
+        if model_dir and intent_classifier:
+            classifier_file = os.path.join(model_dir, intent_classifier)
+            classifier = text_categorizer(classifier_file)
+            return MitieIntentClassifier(classifier)
+        else:
+            return MitieIntentClassifier()
+
+    def persist(self, model_dir):
+        # type: (str) -> object
+        import os
+
+        if self.clf:
+            classifier_file = os.path.join(model_dir, "intent_classifier.dat")
+            self.clf.save_to_disk(classifier_file, pure_model=True)
+            return {"intent_classifier": "intent_classifier.dat"}
+        else:
+            return {"intent_classifier": None}
diff --git a/rasa_nlu/classifiers/sklearn_intent_classifier.py b/rasa_nlu/classifiers/sklearn_intent_classifier.py
new file mode 100644
index 000000000000..ee93f2040e57
--- /dev/null
+++ b/rasa_nlu/classifiers/sklearn_intent_classifier.py
@@ -0,0 +1,138 @@
+import os
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+# How many intents are at max put into the output intent ranking, everything else will be cut off
+INTENT_RANKING_LENGTH = 10
+
+
+class SklearnIntentClassifier(Component):
+    """Intent classifier using the sklearn framework"""
+
+    name = "intent_classifier_sklearn"
+
+    context_provides = {
+        "process": ["intent", "intent_ranking"],
+    }
+
+    output_provides = ["intent"]
+
+    def __init__(self, clf=None, le=None):
+        """Construct a new intent classifier using the sklearn framework."""
+
+        if le is not None:
+            self.le = le
+        else:
+            from sklearn.preprocessing import LabelEncoder
+            self.le = LabelEncoder()
+        self.clf = clf
+
+    def transform_labels_str2num(self, labels):
+        # type: ([str]) -> [int]
+        """Transforms a list of strings into numeric label representation.
+
+        :param labels: List of labels to convert to numeric representation"""
+
+        return self.le.fit_transform(labels)
+
+    def transform_labels_num2str(self, y):
+        # type: ([int]) -> [str]
+        """Transforms a list of strings into numeric label representation.
+
+        :param y: List of labels to convert to numeric representation"""
+
+        return self.le.inverse_transform(y)
+
+    def train(self, training_data, intent_features, num_threads):
+        # type: (TrainingData, [float], int) -> None
+        """Train the intent classifier on a data set.
+
+        :param num_threads: number of threads used during training time"""
+        from sklearn.model_selection import GridSearchCV
+        from sklearn.svm import SVC
+
+        labels = [e["intent"] for e in training_data.intent_examples]
+
+        if len(set(labels)) < 2:
+            raise Exception("Can not train an intent classifier. Need at least 2 different classes.")
+        y = self.transform_labels_str2num(labels)
+        X = intent_features
+
+        tuned_parameters = [{'C': [1, 2, 5, 10, 20, 100], 'kernel': ['linear']}]
+        self.clf = GridSearchCV(SVC(C=1, probability=True),
+                                param_grid=tuned_parameters, n_jobs=num_threads,
+                                cv=2, scoring='f1_weighted')
+
+        self.clf.fit(X, y)
+
+    def process(self, intent_features):
+        # type: ([float]) -> dict
+        """Returns the most likely intent and its probability for the input text."""
+
+        X = intent_features.reshape(1, -1)
+        intent_ids, probabilities = self.predict(X)
+        intents = self.transform_labels_num2str(intent_ids)
+        # `predict` returns a matrix as it is supposed to work for multiple examples as well, hence we need to flatten
+        intents, probabilities = intents.flatten(), probabilities.flatten()
+        if intents.size > 0 and probabilities.size > 0:
+            ranking = zip(list(intents), list(probabilities))[:INTENT_RANKING_LENGTH]
+            return {
+                "intent": {
+                    "name": intents[0],
+                    "confidence": probabilities[0],
+                },
+                "intent_ranking": [{"name": intent, "confidence": score} for intent, score in ranking]
+            }
+        else:
+            return {"intent": None, "intent_ranking": []}
+
+    def predict_prob(self, X):
+        # type: (np.ndarray) -> np.ndarray
+        """Given a bow vector of an input text, predict the intent label. Returns probabilities for all labels.
+
+        :param X: bow of input text
+        :return: vector of probabilities containing one entry for each label"""
+
+        import numpy as np
+
+        return self.clf.predict_proba(X)
+
+    def predict(self, X):
+        """Given a bow vector of an input text, predict most probable label. Returns only the most likely label.
+
+        :param X: bow of input text
+        :return: tuple of first, the most probable label and second, its probability"""
+
+        import numpy as np
+
+        pred_result = self.predict_prob(X)
+        # sort the probabilities retrieving the indices of the elements in sorted order
+        sorted_indices = np.flip(np.argsort(pred_result, axis=1), axis=1)
+        return sorted_indices, pred_result[:, sorted_indices]
+
+    @classmethod
+    def load(cls, model_dir, intent_classifier):
+        # type: (str, str) -> SklearnIntentClassifier
+        import cloudpickle
+
+        if model_dir and intent_classifier:
+            classifier_file = os.path.join(model_dir, intent_classifier)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return SklearnIntentClassifier()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        import cloudpickle
+
+        classifier_file = os.path.join(model_dir, "intent_classifier.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "intent_classifier": "intent_classifier.pkl"
+        }
diff --git a/rasa_nlu/components.py b/rasa_nlu/components.py
new file mode 100644
index 000000000000..0674595f6079
--- /dev/null
+++ b/rasa_nlu/components.py
@@ -0,0 +1,125 @@
+import inspect
+
+from typing import Optional
+from typing import Type
+
+
+def load_component(component_clz, context, config):
+    # type: (Type[Component], dict, dict) -> Optional[Component]
+    """Calls a components load method to init it based on a previously persisted model."""
+
+    if component_clz is not None:
+        load_args = fill_args(component_clz.load_args(), context, config)
+        return component_clz.load(*load_args)
+    else:
+        return None
+
+
+def init_component(component, context, config):
+    # type: (Component, dict, dict) -> None
+    """Initializes a component using the attributes from the context and configuration."""
+
+    args = fill_args(component.pipeline_init_args(), context, config)
+    updates = component.pipeline_init(*args)
+    if updates:
+        context.update(updates)
+
+
+def fill_args(arguments, context, config):
+    # type: ([str], dict, dict) -> [object]
+    """Given a list of arguments, tries to look up these argument names in the config / context to fill the arguments"""
+
+    filled = []
+    for arg in arguments:
+        if arg in context:
+            filled.append(context[arg])
+        elif arg in config:
+            filled.append(config[arg])
+        else:
+            raise MissingArgumentError("Couldn't fill argument '{}' :(".format(arg))
+    return filled
+
+
+class MissingArgumentError(Exception):
+    """Raised when a function is called and not all parameters can be filled from the context / config.
+
+    Attributes:
+        message -- explanation of which parameter is missing
+    """
+
+    def __init__(self, message):
+        # type: (str) -> None
+        self.message = message
+
+
+class Component(object):
+    # Name of the component to be used when integrating it in a pipeline. E.g. `[ComponentA, ComponentB]`
+    # will be a proper pipeline definition where `ComponentA` is the name of the first component of the pipeline.
+    name = ""
+
+    # Defines what attributes the pipeline component will provide when called. The different keys indicate the
+    # different functions (`pipeline_init`, `train`, `process`) that are able to update the pipelines context.
+    # (mostly used to check if the pipeline is valid)
+    context_provides = {
+        "pipeline_init": [],
+        "train": [],
+        "process": [],
+    }
+
+    # Defines which of the attributes the component provides should be added to the final output json at the end of the
+    # pipeline. Every attribute in `output_provides` should be part of the above `context_provides['process']`. As it
+    # wouldn't make much sense to keep an attribute in the output that is not generated. Every other attribute provided
+    # in the context during the process step will be removed from the output json.
+    output_provides = []
+
+    @classmethod
+    def load(cls, *args):
+        # type: (...) -> 'cls'
+        return cls()
+
+    def pipeline_init(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def train(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def process(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def persist(self, model_dir):
+        # type: (str) -> Optional[dict]
+        pass
+
+    @classmethod
+    def cache_key(cls, model_metadata):
+        # type: (Metadata) -> Optional[str]
+        """This key is used to cache components.
+
+        If a model is unique to a model it should return None. Otherwise, an instantiation of the
+        component will be reused for all models where the metadata creates the same key."""
+        from rasa_nlu.model import Metadata
+
+        return None
+
+    def pipeline_init_args(self):
+        # type: () -> [str]
+        return filter(lambda arg: arg not in ["self"], inspect.getargspec(self.pipeline_init).args)
+
+    def train_args(self):
+        # type: () -> [str]
+        return filter(lambda arg: arg not in ["self"], inspect.getargspec(self.train).args)
+
+    def process_args(self):
+        # type: () -> [str]
+        return filter(lambda arg: arg not in ["self"], inspect.getargspec(self.process).args)
+
+    @classmethod
+    def load_args(cls):
+        # type: () -> [str]
+        return filter(lambda arg: arg not in ["cls"], inspect.getargspec(cls.load).args)
+
+    def __eq__(self, other):
+        return self.__dict__ == other.__dict__
diff --git a/src/config.py b/rasa_nlu/config.py
similarity index 60%
rename from src/config.py
rename to rasa_nlu/config.py
index f9d672acedc8..465d91ccea96 100644
--- a/src/config.py
+++ b/rasa_nlu/config.py
@@ -1,41 +1,44 @@
 import codecs
 import json
-import os
 import logging
+import os
+import warnings
 
 
 # Describes where to search for the configuration file if the location is not set by the user
-from rasa_nlu.utils.mitie import MITIE_BACKEND_NAME
-
 DEFAULT_CONFIG_LOCATION = "config.json"
 
 
+DEFAULT_CONFIG = {
+    "config": DEFAULT_CONFIG_LOCATION,
+    "model_template": None,
+    "data": None,
+    "emulate": None,
+    "language": "en",
+    "log_file": None,
+    "log_level": logging.INFO,
+    "mitie_file": os.path.join("data", "total_word_feature_extractor.dat"),
+    "num_threads": 1,
+    "fine_tune_spacy_ner": False,
+    "path": os.path.join(os.getcwd(), "models"),
+    "port": 5000,
+    "server_model_dirs": None,
+    "token": None,
+    "max_number_of_ngrams": 7,
+    "pipeline": [],
+    "response_log": os.path.join(os.getcwd(), "logs"),
+    "luis_data_tokenizer": None,
+}
+
+
 class RasaNLUConfig(object):
 
     def __init__(self, filename=None, env_vars=None, cmdline_args=None):
 
-        defaults = {
-          "backend": "mitie",
-          "config": DEFAULT_CONFIG_LOCATION,
-          "data": None,
-          "emulate": None,
-          "language": "en",
-          "log_file": None,
-          "log_level": logging.INFO,
-          "mitie_file": os.path.join("data", "total_word_feature_extractor.dat"),
-          "num_threads": 1,
-          "fine_tune_spacy_ner": False,
-          "path": os.path.join(os.getcwd(), "models"),
-          "port": 5000,
-          "server_model_dirs": None,
-          "token": None,
-          "response_log": os.path.join(os.getcwd(), "logs")
-        }
-
         if filename is None and os.path.isfile(DEFAULT_CONFIG_LOCATION):
             filename = DEFAULT_CONFIG_LOCATION
 
-        self.override(defaults)
+        self.override(DEFAULT_CONFIG)
         if filename is not None:
             with codecs.open(filename, encoding='utf-8') as f:
                 file_config = json.loads(f.read())
@@ -49,11 +52,17 @@ def __init__(self, filename=None, env_vars=None, cmdline_args=None):
             cmdline_config = {k: v for k, v in cmdline_args.items() if v is not None}
             self.override(cmdline_config)
 
+        if self.__dict__['model_template'] and not self.__dict__['pipeline']:
+            from rasa_nlu import registry
+            if self.__dict__['model_template'] in registry.registered_model_templates:
+                self.__dict__['pipeline'] = registry.registered_model_templates[self.__dict__['model_template']]
+            else:
+                warnings.warn("No model specified and unknown model_template " +
+                              "'{}' passed.".format(self.__dict__['model_template']))
+
         for key, value in self.items():
             setattr(self, key, value)
 
-        self.validate()
-
     def __getitem__(self, key):
         return self.__dict__[key]
 
@@ -84,8 +93,3 @@ def is_set(self, key):
 
     def override(self, new_dict):
         self.__dict__.update(new_dict)
-
-    def validate(self):
-        if self.backend == MITIE_BACKEND_NAME:
-            if not self.is_set("mitie_file"):
-                raise ValueError("backend set to 'mitie' but mitie_file not specified")
diff --git a/src/convert.py b/rasa_nlu/convert.py
similarity index 73%
rename from src/convert.py
rename to rasa_nlu/convert.py
index d5cd7fb3fac0..9a4fb9523a14 100644
--- a/src/convert.py
+++ b/rasa_nlu/convert.py
@@ -1,13 +1,14 @@
 import argparse
-import json
-from rasa_nlu.training_data import TrainingData
+import codecs
+
+from rasa_nlu.converters import load_data
 
 
 def create_argparser():
     parser = argparse.ArgumentParser(description='train a custom language parser')
     parser.add_argument('-d', '--data_file',
                         help='file or dir containing training data')
-    parser.add_argument('-b', '--backend', default='default', choices=['mitie', 'spacy_sklearn', 'default'],
+    parser.add_argument('-b', '--backend', default='tokenize_mitie', choices=['tokenize_mitie', 'tokenize_spacy'],
                         help='backend to use to tokenize text')
     parser.add_argument('-l', '--language', default=None, choices=['de', 'en'], help="model and data language")
     parser.add_argument('-o', '--out_file',
@@ -16,11 +17,11 @@ def create_argparser():
 
 
 def write_file(td, out_file):
-    with open(out_file, 'wb') as f:
+    with codecs.open(out_file, "w") as f:
         f.write(td.as_json(indent=2))
 
 if __name__ == "__main__":
     parser = create_argparser()
     args = parser.parse_args()
-    td = TrainingData(args.data_file, args.backend, args.language)
+    td = load_data(args.data_file, args.language, args.backend)
     write_file(td, args.out_file)
diff --git a/rasa_nlu/converters.py b/rasa_nlu/converters.py
new file mode 100644
index 000000000000..f57647b8ddc7
--- /dev/null
+++ b/rasa_nlu/converters.py
@@ -0,0 +1,200 @@
+import codecs
+import json
+import re
+import warnings
+
+from typing import Optional
+
+from rasa_nlu import util
+from rasa_nlu.tokenizers import Tokenizer
+from rasa_nlu.training_data import TrainingData
+
+# Different supported file formats and their identifier
+WIT_FILE_FORMAT = "wit"
+API_FILE_FORMAT = "api"
+LUIS_FILE_FORMAT = "luis"
+RASA_FILE_FORMAT = "rasa_nlu"
+UNK_FILE_FORMAT = "unk"
+
+
+def load_api_data(files):
+    # type: ([str]) -> TrainingData
+    """Loads training data stored in the API.ai data format."""
+
+    intent_examples = []
+    entity_examples = []
+    common_examples = []
+    entity_synonyms = {}
+    for filename in files:
+        with codecs.open(filename, encoding="utf-8-sig") as f:
+            data = json.loads(f.read())
+        # get only intents, skip the rest. The property name is the target class
+        if "userSays" in data:
+            intent = data.get("name")
+            for s in data["userSays"]:
+                text = "".join(map(lambda chunk: chunk["text"], s.get("data")))
+                # add entities to each token, if available
+                entities = []
+                for e in filter(lambda chunk: "alias" in chunk or "meta" in chunk, s.get("data")):
+                    start = text.find(e["text"])
+                    end = start + len(e["text"])
+                    val = text[start:end]
+                    entities.append(
+                        {
+                            "entity": e["alias"] if "alias" in e else e["meta"],
+                            "value": val,
+                            "start": start,
+                            "end": end
+                        }
+                    )
+
+                if intent and entities:
+                    common_examples.append({"text": text, "intent": intent, "entities": entities})
+                elif intent:
+                    intent_examples.append({"text": text, "intent": intent})
+                elif entities:
+                    entity_examples.append({"text": text, "intent": intent, "entities": entities})
+
+        # create synonyms dictionary
+        if "name" in data and "entries" in data:
+            for entry in data["entries"]:
+                if "value" in entry and "synonyms" in entry:
+                    for synonym in entry["synonyms"]:
+                        entity_synonyms[synonym] = entry["value"]
+    return TrainingData(intent_examples, entity_examples, common_examples, entity_synonyms)
+
+
+def load_luis_data(filename, tokenizer):
+    # type: (str, Optional[Tokenizer]) -> TrainingData
+    """Loads training data stored in the LUIS.ai data format."""
+
+    warnings.warn(
+        """LUIS data may not always be correctly imported because entity locations are specified by tokens.
+        If you use a tokenizer which behaves differently from LUIS's your entities might not be correct""")
+    if not tokenizer:
+        raise ValueError("Can not load luis data without a specified tokenizer " +
+                         "(e.g. using the configuration value `luis_data_tokenizer`)")
+
+    intent_examples = []
+    entity_examples = []
+    common_examples = []
+
+    with codecs.open(filename, encoding="utf-8-sig") as f:
+        data = json.loads(f.read())
+    for s in data["utterances"]:
+        text = s.get("text")
+        tokens = [t for t in tokenizer.tokenize(text)]
+        intent = s.get("intent")
+        entities = []
+        for e in s.get("entities") or []:
+            i, ii = e["startPos"], e["endPos"] + 1
+            _regex = u"\s*".join([re.escape(s) for s in tokens[i:ii]])
+            expr = re.compile(_regex)
+            m = expr.search(text)
+            start, end = m.start(), m.end()
+            val = text[start:end]
+            entities.append({"entity": e["entity"], "value": val, "start": start, "end": end})
+
+        if intent and entities:
+            common_examples.append({"text": text, "intent": intent, "entities": entities})
+        elif intent:
+            intent_examples.append({"text": text, "intent": intent})
+        elif entities:
+            entity_examples.append({"text": text, "intent": intent, "entities": entities})
+    return TrainingData(intent_examples, entity_examples, common_examples)
+
+
+def load_wit_data(filename):
+    # type: (str) -> TrainingData
+    """Loads training data stored in the WIT.ai data format."""
+
+    intent_examples = []
+    entity_examples = []
+    common_examples = []
+
+    with codecs.open(filename, encoding="utf-8-sig") as f:
+        data = json.loads(f.read())
+    for s in data["data"]:
+        entities = s.get("entities")
+        if entities is None:
+            continue
+        text = s.get("text")
+        intents = [e["value"] for e in entities if e["entity"] == 'intent']
+        intent = intents[0] if intents else None
+
+        entities = [e for e in entities if ("start" in e and "end" in e)]
+        for e in entities:
+            e["value"] = e["value"][1:-1]
+
+        if intent and entities:
+            common_examples.append({"text": text, "intent": intent, "entities": entities})
+        elif intent:
+            intent_examples.append({"text": text, "intent": intent})
+        elif entities:
+            entity_examples.append({"text": text, "intent": intent, "entities": entities})
+    return TrainingData(intent_examples, entity_examples, common_examples)
+
+
+def load_rasa_data(filename):
+    # type: (str) -> TrainingData
+    """Loads training data stored in the rasa NLU data format."""
+
+    with codecs.open(filename, encoding="utf-8-sig") as f:
+        data = json.loads(f.read())
+    common = data['rasa_nlu_data'].get("common_examples", list())
+    intent = data['rasa_nlu_data'].get("intent_examples", list())
+    entity = data['rasa_nlu_data'].get("entity_examples", list())
+
+    return TrainingData(intent, entity, common)
+
+
+def guess_format(files):
+    # type: ([str]) -> str
+    """Given a set of files, tries to guess which data format is used."""
+
+    for filename in files:
+        with codecs.open(filename, encoding="utf-8-sig") as f:
+            file_data = json.loads(f.read())
+        if "data" in file_data and type(file_data.get("data")) is list:
+            return WIT_FILE_FORMAT
+        elif "luis_schema_version" in file_data:
+            return LUIS_FILE_FORMAT
+        elif "userSays" in file_data:
+            return API_FILE_FORMAT
+        elif "rasa_nlu_data" in file_data:
+            return RASA_FILE_FORMAT
+
+    return UNK_FILE_FORMAT
+
+
+def resolve_data_files(resource_name):
+    # type: (str) -> [str]
+    """Lists all data files of the resource name (might be a file or directory)."""
+
+    try:
+        return util.recursively_find_files(resource_name)
+    except ValueError as e:
+        raise ValueError("Invalid training data file / folder specified. " + e.message)
+
+
+def load_data(resource_name, language, luis_data_tokenizer=None, fformat=None):
+    # type: (str, str, Optional[Tokenizer], Optional[str]) -> TrainingData
+    """Loads training data from disk. If no format is provided, the format will be guessed based on the files."""
+
+    files = resolve_data_files(resource_name)
+
+    if not fformat:
+        fformat = guess_format(files)
+
+    if fformat == LUIS_FILE_FORMAT:
+        from rasa_nlu.tokenizers import tokenizer_from_name
+        tokenizer = tokenizer_from_name(luis_data_tokenizer, language)
+        return load_luis_data(files[0], tokenizer)
+    elif fformat == WIT_FILE_FORMAT:
+        return load_wit_data(files[0])
+    elif fformat == API_FILE_FORMAT:
+        return load_api_data(files)
+    elif fformat == RASA_FILE_FORMAT:
+        return load_rasa_data(files[0])
+    else:
+        raise ValueError("unknown training file format : {0}".format(fformat))
diff --git a/src/data_router.py b/rasa_nlu/data_router.py
similarity index 63%
rename from src/data_router.py
rename to rasa_nlu/data_router.py
index 12a39a0b2ebd..fe1ba914d853 100644
--- a/src/data_router.py
+++ b/rasa_nlu/data_router.py
@@ -1,3 +1,4 @@
+import datetime
 import glob
 import json
 import logging
@@ -5,17 +6,52 @@
 import os
 import tempfile
 
-import datetime
 from flask import json
 
-from rasa_nlu.model import Model, Metadata, InvalidModelError
+import rasa_nlu.components
+from rasa_nlu import registry
 from rasa_nlu.config import RasaNLUConfig
+from rasa_nlu.model import Metadata, InvalidModelError, Interpreter
 from rasa_nlu.train import do_train
-from rasa_nlu.utils import mitie
-from rasa_nlu.utils import spacy
 from rasa_nlu.util import create_dir_for_file
 
 
+class InterpreterBuilder(object):
+    def __init__(self, use_cache=True):
+        self.use_cache = use_cache
+        # Reuse nlp and featurizers where possible to save memory
+        self.component_cache = {}
+
+    def __get_component(self, component_name, meta, context, model_config):
+        component_class = registry.get_component_class(component_name)
+        cache_key = component_class.cache_key(meta)
+        if cache_key is not None and self.use_cache and cache_key in self.component_cache:
+            component = self.component_cache[cache_key]
+        else:
+            component = registry.load_component_by_name(component_name, context, model_config)
+            if cache_key is not None and self.use_cache:
+                self.component_cache[cache_key] = component
+        return component
+
+    def create_interpreter(self, meta, config):
+        context = {"model_dir": meta.model_dir}
+
+        model_config = dict(config.items())
+        model_config.update(meta.metadata)
+
+        pipeline = []
+
+        for component_name in meta.pipeline:
+            try:
+                component = self.__get_component(component_name, meta, context, model_config)
+                rasa_nlu.components.init_component(component, context, model_config)
+                pipeline.append(component)
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to initialize component '{}'. {}".format(component_name, e.message))
+
+        return Interpreter(pipeline, context, model_config)
+
+
 class DataRouter(object):
     DEFAULT_MODEL_NAME = "default"
 
@@ -26,6 +62,7 @@ def __init__(self, config):
         self.model_dir = config['path']
         self.token = config['token']
         self.emulator = self.__create_emulator()
+        self.interpreter_builder = InterpreterBuilder()
         self.model_store = self.__create_model_store()
 
     @staticmethod
@@ -53,31 +90,6 @@ def _create_query_logger(response_log_dir):
             logging.info("Logging of requests is disabled. (No 'request_log' directory configured)")
             return None
 
-    @staticmethod
-    def featurizer_for_model(model, nlp):
-        """Initialize featurizer for backends. If it can NOT be shared between models, `None` should be returned."""
-
-        if model.backend_name() == spacy.SPACY_BACKEND_NAME:
-            from featurizers.spacy_featurizer import SpacyFeaturizer
-            return SpacyFeaturizer(nlp)
-        elif model.backend_name() == mitie.MITIE_BACKEND_NAME or \
-                model.backend_name() == mitie.MITIE_SKLEARN_BACKEND_NAME:
-            from rasa_nlu.featurizers.mitie_featurizer import MITIEFeaturizer
-            return MITIEFeaturizer(model.feature_extractor_path)
-        else:
-            return None
-
-    @staticmethod
-    def nlp_for_backend(backend, language):
-        """Initialize nlp for backends. If nlp can NOT be shared between models, `None` should be returned."""
-
-        if spacy.SPACY_BACKEND_NAME == backend:
-            # spacy can share large memory objects across models
-            import spacy as sp
-            return sp.load(language, parser=False, entity=False, matcher=False)
-        else:
-            return None
-
     def __search_for_models(self):
         models = {}
         for metadata_path in glob.glob(os.path.join(self.config.path, '*/metadata.json')):
@@ -94,35 +106,22 @@ def __create_model_store(self):
         else:
             model_dict = self.config.server_model_dirs
 
-        # Reuse nlp and featurizers where possible to save memory
-        cache = {}
-
         model_store = {}
 
         for alias, model_path in model_dict.items():
             metadata = DataRouter.read_model_metadata(model_path, self.config)
-            cache_key = metadata.model_group()
-            if cache_key in cache:
-                nlp = cache[cache_key]['nlp']
-                featurizer = cache[cache_key]['featurizer']
-            else:
-                nlp = DataRouter.nlp_for_backend(metadata.backend_name(), metadata.language_name())
-                featurizer = DataRouter.featurizer_for_model(metadata, nlp)
-                cache[cache_key] = {'nlp': nlp, 'featurizer': featurizer}
-            interpreter = DataRouter.create_interpreter(metadata, nlp, featurizer)
-            model_store[alias] = Model(metadata, model_path, interpreter, nlp, featurizer)
+            interpreter = self.interpreter_builder.create_interpreter(metadata, self.config)
+            model_store[alias] = interpreter
         if not model_store:
-            meta = Metadata({}, "")
-            interpreter = DataRouter.create_interpreter(meta)
-            model_store[self.DEFAULT_MODEL_NAME] = Model(meta, "", interpreter)
+            meta = Metadata({"pipeline": ["intent_classifier_keyword"]}, "")
+            interpreter = self.interpreter_builder.create_interpreter(meta, self.config)
+            model_store[self.DEFAULT_MODEL_NAME] = interpreter
         return model_store
 
     @staticmethod
     def default_model_metadata():
         return {
-            "backend": None,
-            "language_name": None,
-            "feature_extractor": None
+            "language": None,
         }
 
     @staticmethod
@@ -149,41 +148,19 @@ def read_model_metadata(model_dir, config):
 
             return Metadata.load(model_dir)
 
-    @staticmethod
-    def create_interpreter(metadata, nlp=None, featurizer=None):
-        backend = metadata.backend_name()
-        if backend is None:
-            from interpreters.simple_interpreter import HelloGoodbyeInterpreter
-            return HelloGoodbyeInterpreter()
-        elif backend.lower() == mitie.MITIE_BACKEND_NAME:
-            logging.info("using mitie backend")
-            from interpreters.mitie_interpreter import MITIEInterpreter
-            return MITIEInterpreter.load(metadata, featurizer)
-        elif backend.lower() == mitie.MITIE_SKLEARN_BACKEND_NAME:
-            logging.info("using mitie_sklearn backend")
-            from interpreters.mitie_sklearn_interpreter import MITIESklearnInterpreter
-            return MITIESklearnInterpreter.load(metadata, featurizer)
-
-        elif backend.lower() == spacy.SPACY_BACKEND_NAME:
-            logging.info("using spacy + sklearn backend")
-            from interpreters.spacy_sklearn_interpreter import SpacySklearnInterpreter
-            return SpacySklearnInterpreter.load(metadata, nlp, featurizer)
-        else:
-            raise ValueError("unknown backend : {0}".format(backend))
-
     def __create_emulator(self):
         mode = self.config['emulate']
         if mode is None:
-            from emulators import NoEmulator
+            from rasa_nlu.emulators import NoEmulator
             return NoEmulator()
         elif mode.lower() == 'wit':
-            from emulators.wit import WitEmulator
+            from rasa_nlu.emulators.wit import WitEmulator
             return WitEmulator()
         elif mode.lower() == 'luis':
-            from emulators.luis import LUISEmulator
+            from rasa_nlu.emulators.luis import LUISEmulator
             return LUISEmulator()
         elif mode.lower() == 'api':
-            from emulators.api import ApiEmulator
+            from rasa_nlu.emulators.api import ApiEmulator
             return ApiEmulator()
         else:
             raise ValueError("unknown mode : {0}".format(mode))
@@ -197,7 +174,7 @@ def parse(self, data):
             raise InvalidModelError("No model found with alias '{}'".format(alias))
         else:
             model = self.model_store[alias]
-            response = model.interpreter.parse(data['text'])
+            response = model.parse(data['text'])
             if self.responses:
                 self.responses.info(json.dumps(response, sort_keys=True))
             return self.format_response(response)
diff --git a/src/download.py b/rasa_nlu/download.py
similarity index 98%
rename from src/download.py
rename to rasa_nlu/download.py
index 9619ffd42a4c..aef0a497e22f 100644
--- a/src/download.py
+++ b/rasa_nlu/download.py
@@ -22,6 +22,7 @@ def create_argparser():
 
 
 def download_mitie_fe_file(fe_file):
+    # type: (str) -> None
     """Download the mitie feature extractor needed to run & train mitie classifiers.
 
     See https://github.com/mit-nlp/MITIE#initial-setup """
diff --git a/src/emulators/__init__.py b/rasa_nlu/emulators/__init__.py
similarity index 100%
rename from src/emulators/__init__.py
rename to rasa_nlu/emulators/__init__.py
diff --git a/src/emulators/api.py b/rasa_nlu/emulators/api.py
similarity index 100%
rename from src/emulators/api.py
rename to rasa_nlu/emulators/api.py
diff --git a/rasa_nlu/emulators/luis.py b/rasa_nlu/emulators/luis.py
new file mode 100644
index 000000000000..5180140d0ff8
--- /dev/null
+++ b/rasa_nlu/emulators/luis.py
@@ -0,0 +1,40 @@
+from rasa_nlu.emulators import NoEmulator
+
+
+class LUISEmulator(NoEmulator):
+    def __init__(self):
+        super(LUISEmulator, self).__init__()
+        self.name = 'luis'
+
+    def _top_intent(self, data):
+        if data.get("intent"):
+            return {
+                "intent": data["intent"]["name"],
+                "score": data["intent"]["confidence"]
+            }
+        else:
+            return None
+
+    def _ranking(self, data):
+        if data.get("intent_ranking"):
+            return [{"intent": el["intent"], "score": el["confidence"]} for el in data["intent_ranking"]]
+        else:
+            return [self._top_intent(data)]
+
+    def normalise_response_json(self, data):
+        top_intent = self._top_intent(data)
+        ranking = self._ranking(data)
+        return {
+            "query": data["text"],
+            "topScoringIntent": top_intent,
+            "intents": ranking,
+            "entities": [
+                {
+                    "entity": e["value"],
+                    "type": e["entity"],
+                    "startIndex": None,
+                    "endIndex": None,
+                    "score": None
+                } for e in data["entities"]
+                ]
+        }
diff --git a/src/emulators/wit.py b/rasa_nlu/emulators/wit.py
similarity index 85%
rename from src/emulators/wit.py
rename to rasa_nlu/emulators/wit.py
index 10734293c28d..e82bd1ac798e 100644
--- a/src/emulators/wit.py
+++ b/rasa_nlu/emulators/wit.py
@@ -20,8 +20,8 @@ def normalise_response_json(self, data):
         return [
             {
                 "_text": data["text"],
-                "confidence": data["confidence"],
-                "intent": data["intent"],
+                "confidence": data["intent"]['confidence'],
+                "intent": data["intent"]['name'],
                 "entities": entities
             }
         ]
diff --git a/rasa_nlu/extractors/__init__.py b/rasa_nlu/extractors/__init__.py
new file mode 100644
index 000000000000..c5549d3850ed
--- /dev/null
+++ b/rasa_nlu/extractors/__init__.py
@@ -0,0 +1,2 @@
+class EntityExtractor(object):
+    pass
diff --git a/rasa_nlu/extractors/entity_synonyms.py b/rasa_nlu/extractors/entity_synonyms.py
new file mode 100644
index 000000000000..2e910434d46f
--- /dev/null
+++ b/rasa_nlu/extractors/entity_synonyms.py
@@ -0,0 +1,80 @@
+import codecs
+import json
+import os
+import warnings
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class EntitySynonymMapper(Component):
+    name = "ner_synonyms"
+
+    context_provides = {
+        "process": ["entities"],
+    }
+
+    output_provides = ["entities"]
+
+    def __init__(self, synonyms=None):
+        self.synonyms = synonyms if synonyms else {}
+
+    def train(self, training_data):
+        # type: (TrainingData) -> None
+
+        for key, value in training_data.entity_synonyms.items():
+            self.add_entities_if_synonyms(key, value)
+
+        for example in training_data.entity_examples:
+            for entity in example["entities"]:
+                entity_val = example["text"][entity["start"]:entity["end"]]
+                self.add_entities_if_synonyms(entity_val, entity.get("value"))
+
+    def process(self, entities):
+        # type: (dict) -> dict
+
+        updated_entities = entities[:]
+        self.replace_synonyms(updated_entities)
+
+        return {
+            "entities": updated_entities
+        }
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+
+        if self.synonyms:
+            entity_synonyms_file = os.path.join(model_dir, "index.json")
+            with open(entity_synonyms_file, 'w') as f:
+                json.dump(self.synonyms, f)
+            return {"entity_synonyms": "index.json"}
+        else:
+            return {"entity_synonyms": None}
+
+    @classmethod
+    def load(cls, model_dir, entity_synonyms):
+        # type: (str, str) -> EntitySynonymMapper
+
+        if model_dir and entity_synonyms:
+            entity_synonyms_file = os.path.join(model_dir, "index.json")
+            if os.path.isfile(entity_synonyms_file):
+                with codecs.open(entity_synonyms_file, encoding='utf-8') as f:
+                    synonyms = json.loads(f.read())
+                return EntitySynonymMapper(synonyms)
+            else:
+                warnings.warn("Failed to load synonyms file from '{}'".format(entity_synonyms_file))
+        return EntitySynonymMapper()
+
+    def replace_synonyms(self, entities):
+        for i in range(len(entities)):
+            entity_value = entities[i]["value"]
+            if entity_value.lower() in self.synonyms:
+                entities[i]["value"] = self.synonyms[entity_value.lower()]
+
+    def add_entities_if_synonyms(self, entity_a, entity_b):
+        if entity_b is not None:
+            original = entity_a.lower() if type(entity_a) == unicode else unicode(entity_a)
+            replacement = entity_b.lower() if type(entity_b) == unicode else unicode(entity_b)
+
+            if original != replacement:
+                self.synonyms[original] = replacement
diff --git a/rasa_nlu/extractors/mitie_entity_extractor.py b/rasa_nlu/extractors/mitie_entity_extractor.py
new file mode 100644
index 000000000000..c279e24c4366
--- /dev/null
+++ b/rasa_nlu/extractors/mitie_entity_extractor.py
@@ -0,0 +1,111 @@
+import os
+import re
+
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.extractors import EntityExtractor
+from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa_nlu.training_data import TrainingData
+
+
+class MitieEntityExtractor(Component, EntityExtractor):
+    name = "ner_mitie"
+
+    context_provides = {
+        "process": ["entities"],
+    }
+
+    output_provides = ["entities"]
+
+    def __init__(self, ner=None):
+        self.ner = ner
+
+    def extract_entities(self, text, tokens, feature_extractor):
+        ents = []
+        offset = 0
+        if self.ner:
+            entities = self.ner.extract_entities(tokens, feature_extractor)
+            for e in entities:
+                _range = e[0]
+                _regex = u"\s*".join(re.escape(tokens[i]) for i in _range)
+                expr = re.compile(_regex)
+                m = expr.search(text[offset:])
+                start, end = m.start() + offset, m.end() + offset
+                entity_value = text[start:end]
+                offset += m.end()
+                ents.append({
+                    "entity": e[1],
+                    "value": entity_value,
+                    "start": start,
+                    "end": end
+                })
+
+        return ents
+
+    @staticmethod
+    def find_entity(ent, text):
+        from mitie import tokenize
+
+        tk = MitieTokenizer()
+        tokens, offsets = tk.tokenize_with_offsets(text)
+        if ent["start"] not in offsets:
+            message = u"Invalid entity {0} in example '{1}':".format(ent, text) + \
+                      u" entities must span whole tokens"
+            raise ValueError(message)
+        start = offsets.index(ent["start"])
+        _slice = text[ent["start"]:ent["end"]]
+        val_tokens = tokenize(_slice)
+        end = start + len(val_tokens)
+        return start, end
+
+    def train(self, training_data, mitie_file, num_threads):
+        # type: (TrainingData, str, Optional[int]) -> None
+        from mitie import ner_training_instance, ner_trainer, tokenize
+
+        trainer = ner_trainer(mitie_file)
+        trainer.num_threads = num_threads
+        found_one_entity = False
+        for example in training_data.entity_examples:
+            text = example["text"]
+            tokens = tokenize(text)
+            sample = ner_training_instance(tokens)
+            for ent in example["entities"]:
+                start, end = MitieEntityExtractor.find_entity(ent, text)
+                sample.add_entity(xrange(start, end), ent["entity"])
+                found_one_entity = True
+
+            trainer.add(sample)
+        # Mitie will fail to train if there is not a single entity tagged
+        if found_one_entity:
+            self.ner = trainer.train()
+
+    def process(self, text, tokens, mitie_feature_extractor):
+        # type: (str, [str], mitie.total_word_feature_extractor) -> dict
+        import mitie
+
+        return {
+            "entities": self.extract_entities(text, tokens, mitie_feature_extractor)
+        }
+
+    @classmethod
+    def load(cls, model_dir, entity_extractor):
+        # type: (str, str) -> MitieEntityExtractor
+        from mitie import named_entity_extractor
+
+        if model_dir and entity_extractor:
+            entity_extractor_file = os.path.join(model_dir, entity_extractor)
+            extractor = named_entity_extractor(entity_extractor_file)
+            return MitieEntityExtractor(extractor)
+        else:
+            return MitieEntityExtractor()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+
+        if self.ner:
+            entity_extractor_file = os.path.join(model_dir, "entity_extractor.dat")
+            self.ner.save_to_disk(entity_extractor_file, pure_model=True)
+            return {"entity_extractor": "entity_extractor.dat"}
+        else:
+            return {"entity_extractor": None}
diff --git a/rasa_nlu/extractors/spacy_entity_extractor.py b/rasa_nlu/extractors/spacy_entity_extractor.py
new file mode 100644
index 000000000000..f00152377c3e
--- /dev/null
+++ b/rasa_nlu/extractors/spacy_entity_extractor.py
@@ -0,0 +1,153 @@
+from __future__ import unicode_literals, print_function
+
+import os
+import random
+
+import pathlib
+import warnings
+
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.extractors import EntityExtractor
+from rasa_nlu.training_data import TrainingData
+
+
+class SpacyEntityExtractor(Component, EntityExtractor):
+    name = "ner_spacy"
+
+    context_provides = {
+        "process": ["entities"],
+    }
+
+    output_provides = ["entities"]
+
+    def __init__(self, ner=None, fine_tune_spacy_ner=False):
+        self.ner = ner
+        self.fine_tune_spacy_ner = fine_tune_spacy_ner
+
+    def train(self, spacy_nlp, training_data, fine_tune_spacy_ner):
+        # type: (Language, TrainingData, Optional[bool]) -> None
+        from spacy.language import Language
+        self.fine_tune_spacy_ner = fine_tune_spacy_ner
+        if training_data.num_entity_examples > 0:
+            train_data = self._convert_examples(training_data.entity_examples)
+            ent_types = [[ent["entity"] for ent in ex["entities"]] for ex in training_data.entity_examples]
+            entity_types = list(set(sum(ent_types, [])))
+
+            self.ner = self._train_from_scratch(spacy_nlp, entity_types, train_data)
+
+    def process(self, spacy_doc, spacy_nlp):
+        # type: (Doc) -> dict
+        from spacy.tokens import Doc
+
+        return {
+            "entities": self.extract_entities(spacy_doc, spacy_nlp)
+        }
+
+    def extract_entities(self, doc, nlp):
+        # type: (Doc) -> [dict]
+        from spacy.tokens import Doc
+
+        if self.ner is not None:
+            self.ner(doc)
+
+            # REMOVE THIS, as soon as we are able again to fine tune spacy models instead of combining them
+            if nlp.entity is not None and self.fine_tune_spacy_ner:
+                sp_doc = nlp(doc.text)
+                spacy_ents = sp_doc.ents
+                for spacy_ent in spacy_ents:
+                    for e in doc.ents:
+                        if e.start_char <= spacy_ent.start_char < e.end_char or \
+                              e.start_char <= spacy_ent.end_char < e.end_char:
+                            break
+                    else:
+                        doc.ents += (spacy_ent,)
+
+            entities = [
+                {
+                    "entity": ent.label_,
+                    "value": ent.text,
+                    "start": ent.start_char,
+                    "end": ent.end_char
+                }
+                for ent in doc.ents]
+            return entities
+        else:
+            return []
+
+    @classmethod
+    def load(cls, model_dir, entity_extractor, fine_tune_spacy_ner, spacy_nlp):
+        # type: (str, str, Language) -> SpacyEntityExtractor
+        from spacy.language import Language
+        from spacy.pipeline import EntityRecognizer
+
+        if model_dir and entity_extractor:
+            ner_dir = os.path.join(model_dir, entity_extractor)
+            ner = EntityRecognizer.load(pathlib.Path(ner_dir), spacy_nlp.vocab)
+            return SpacyEntityExtractor(ner, fine_tune_spacy_ner)
+        else:
+            return SpacyEntityExtractor()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+        import json
+
+        if self.ner:
+            ner_dir = os.path.join(model_dir, 'ner')
+            if not os.path.exists(ner_dir):
+                os.mkdir(ner_dir)
+
+            entity_extractor_config_file = os.path.join(ner_dir, "config.json")
+            entity_extractor_file = os.path.join(ner_dir, "model")
+
+            with open(entity_extractor_config_file, 'w') as f:
+                json.dump(self.ner.cfg, f)
+            self.ner.model.dump(entity_extractor_file)
+            return {
+                "entity_extractor": "ner",
+                "fine_tune_spacy_ner": self.fine_tune_spacy_ner,
+            }
+        else:
+            return {"entity_extractor": None}
+
+    def _convert_examples(self, entity_examples):
+        def convert_entity(ent):
+            return ent["start"], ent["end"], ent["entity"]
+
+        def convert_example(ex):
+            return ex["text"], [convert_entity(ent) for ent in ex["entities"]]
+
+        return [convert_example(ex) for ex in entity_examples]
+
+    def _train_from_scratch(self, nlp, entity_types, train_data):
+        from spacy.pipeline import EntityRecognizer
+
+        ner = EntityRecognizer(nlp.vocab, entity_types=entity_types)
+        self._update_ner_model(ner, nlp, train_data)
+        return ner
+
+    # TODO: this is not used at the moment as there is an issue in the latest spacy version. Currently there is no way
+    # TODO: to directly fine tune the entity model as that will result in a malloc error during parse time
+    def _fine_tune(self, nlp, entity_types, train_data):
+        if nlp.entity:
+            ner = nlp.entity
+            for entity_type in entity_types:
+                if entity_type not in ner.cfg['actions']['1']:
+                    ner.add_label(entity_type)
+            self._update_ner_model(ner, nlp, train_data)
+            return ner
+        else:
+            warnings.warn("Failed to fine tune model. There was no model to fine tune. ")
+            return None
+
+    def _update_ner_model(self, ner, nlp, train_data):
+        from spacy.gold import GoldParse
+
+        for itn in range(5):
+            random.shuffle(train_data)
+            for raw_text, entity_offsets in train_data:
+                doc = nlp.make_doc(raw_text)
+                gold = GoldParse(doc, entities=entity_offsets)
+                ner.update(doc, gold)
diff --git a/src/featurizers/__init__.py b/rasa_nlu/featurizers/__init__.py
similarity index 100%
rename from src/featurizers/__init__.py
rename to rasa_nlu/featurizers/__init__.py
diff --git a/rasa_nlu/featurizers/mitie_featurizer.py b/rasa_nlu/featurizers/mitie_featurizer.py
new file mode 100644
index 000000000000..a4b3172e7ae2
--- /dev/null
+++ b/rasa_nlu/featurizers/mitie_featurizer.py
@@ -0,0 +1,62 @@
+from rasa_nlu.components import Component
+from rasa_nlu.featurizers import Featurizer
+from rasa_nlu.training_data import TrainingData
+
+
+class MitieFeaturizer(Featurizer, Component):
+    name = "intent_featurizer_mitie"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    def ndim(self, feature_extractor):
+        # type: (mitie.total_word_feature_extractor) -> int
+        import mitie
+
+        return feature_extractor.num_dimensions
+
+    def train(self, training_data, mitie_feature_extractor):
+        # type: (TrainingData, mitie.total_word_feature_extractor) -> dict
+        import mitie
+
+        sentences = [e["text"] for e in training_data.intent_examples]
+        features = self.features_for_sentences(sentences, mitie_feature_extractor)
+        return {
+            "intent_features": features
+        }
+
+    def process(self, tokens, mitie_feature_extractor):
+        # type: ([str], mitie.total_word_feature_extractor) -> dict
+        import mitie
+
+        features = self.features_for_tokens(tokens, mitie_feature_extractor)
+        return {
+            "intent_features": features
+        }
+
+    def features_for_tokens(self, tokens, feature_extractor):
+        # type: ([str], mitie.total_word_feature_extractor) -> np.ndarray
+        import numpy as np
+        import mitie
+
+        vec = np.zeros(self.ndim(feature_extractor))
+        for token in tokens:
+            vec += feature_extractor.get_feature_vector(token)
+        if tokens:
+            return vec / len(tokens)
+        else:
+            return vec
+
+    def features_for_sentences(self, sentences, feature_extractor):
+        # type: ([str], mitie.total_word_feature_extractor) -> np.ndarray
+        import mitie
+        import numpy as np
+
+        X = np.zeros((len(sentences), self.ndim(feature_extractor)))
+
+        for idx, sentence in enumerate(sentences):
+            tokens = mitie.tokenize(sentence)
+            X[idx, :] = self.features_for_tokens(tokens, feature_extractor)
+        return X
diff --git a/rasa_nlu/featurizers/ngram_featurizer.py b/rasa_nlu/featurizers/ngram_featurizer.py
new file mode 100644
index 000000000000..809b38059d54
--- /dev/null
+++ b/rasa_nlu/featurizers/ngram_featurizer.py
@@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return NGramFeaturizer()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        classifier_file = os.path.join(model_dir, "ngram_featurizer.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "ngram_featurizer": "ngram_featurizer.pkl"
+        }
+
+    def _ngrams_to_use(self, num_ngrams):
+        if num_ngrams == 0:
+            return None
+        elif num_ngrams is not None:
+            return self.all_ngrams[:num_ngrams]
+        else:
+            return self.all_ngrams
+
+    def _get_best_ngrams(self, sentences, labels, spacy_nlp):
+        """Returns an ordered list of the best character ngrams for an intent classification problem"""
+
+        oov_strings = self._remove_in_vocab_words(sentences, spacy_nlp)
+        ngrams = self._generate_all_ngrams(oov_strings)
+        return self._sort_applicable_ngrams(ngrams, sentences, labels, spacy_nlp)
+
+    def _remove_in_vocab_words(self, sentences, spacy_nlp):
+        """Automatically removes words with digits in them, that may be a
+        hyperlink or that _are_ in vocabulary for the nlp"""
+
+        new_sents = []
+        for sentence in sentences:
+            new_sents.append(self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp))
+        return new_sents
+
+    def _remove_hyperlinks(self, sentence):
+        return re.sub(r'^https?:\/\/.*[\r\n]*', '', sentence, flags=re.MULTILINE)
+
+    def _remove_punctuation(self, sentence):
+        return ''.join([letter for letter in sentence if letter not in punctuation])
+
+    def _remove_in_vocab_words_of_sentence(self, sentence, spacy_nlp):
+        """Automatically removes words with digits in them, hyperlink and in-vocab-words."""
+
+        cleaned_sentence = self._remove_hyperlinks(self._remove_punctuation(sentence))
+
+        # keep only out-of-vocab 'non_word' words
+        doc = spacy_nlp(cleaned_sentence)
+        non_words = ' '.join([q.lower_ for q in doc if not q.has_vector])
+
+        # remove digits and extra spaces
+        non_words = ''.join([letter for letter in non_words if not letter.isdigit()])
+        non_words = ' '.join([word for word in non_words.split(' ') if word != ''])
+
+        # add cleaned sentence to list of these sentences
+        return non_words
+
+    def _sort_applicable_ngrams(self, list_of_ngrams, sentences, labels, spacy_nlp):
+        """Given an intent classification problem and a list of ngrams, creates ordered list of most useful ngrams."""
+
+        if list_of_ngrams:
+            from sklearn import linear_model, preprocessing
+            import numpy as np
+
+            usable_labels = []
+            for label in np.unique(labels):
+                lab_sents = np.array(sentences)[np.array(labels) == label]
+                if len(lab_sents) < 10:
+                    continue
+                usable_labels.append(label)
+
+            mask = [label in usable_labels for label in labels]
+            sentences = np.array(sentences)[mask]
+            labels = np.array(labels)[mask]
+
+            X = np.array(self._ngrams_in_sentences(sentences, spacy_nlp, list_of_ngrams))
+            intent_encoder = preprocessing.LabelEncoder()
+            intent_encoder.fit(labels)
+            y = intent_encoder.transform(labels)
+
+            clf = linear_model.RandomizedLogisticRegression(C=1)
+            clf.fit(X, y)
+            scores = clf.scores_
+            sort_idx = [i[0] for i in sorted(enumerate(scores), key=lambda x: -1 * x[1])]
+
+            return np.array(list_of_ngrams)[sort_idx]
+        else:
+            return []
+
+    def _ngrams_in_sentences(self, sentences, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        all_vectors = []
+        for sentence in sentences:
+            presence_vector = self._ngrams_in_sentence(sentence, spacy_nlp, ngrams)
+            all_vectors.append(presence_vector)
+        return all_vectors
+
+    def _ngrams_in_sentence(self, sentence, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        import numpy as np
+
+        cleaned_sentence = self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp)
+        presence_vector = np.zeros(len(ngrams))
+        idx_array = [idx for idx in xrange(len(ngrams)) if ngrams[idx] in cleaned_sentence]
+        presence_vector[idx_array] = 1
+        return presence_vector
+
+    def _generate_all_ngrams(self, list_of_strings):
+        """Takes a list of strings and generates all character ngrams.
+
+        Generated ngrams are at least 3 characters (and at most 17), occur at least 5 times
+        and occur independently of longer superset ngrams at least once."""
+
+        features = {}
+        counters = {}
+
+        for n in xrange(self.n_gram_min_length, self.n_gram_max_length):
+            candidates = []
+            features[n] = []
+            counters[n] = Counter()
+
+            # generate all possible n length ngrams
+            for text in list_of_strings:
+                text = text.replace(',', ' ')
+                for word in text.lower().split(' '):
+                    cands = [word[i:i + n] for i in xrange(len(word) - n)]
+                    for cand in cands:
+                        counters[n][cand] += 1
+                        if cand not in candidates:
+                            candidates.append(cand)
+
+            # iterate over these candidates picking only the applicable ones
+            for can in candidates:
+                if counters[n][can] > self.n_gram_min_occurrences:
+                    features[n].append(can)
+                    begin = can[:-1]
+                    end = can[1:]
+                    if n > self.n_gram_min_length:
+                        if counters[n - 1][begin] == counters[n][can] and begin in features[n - 1]:
+                            features[n - 1].remove(begin)
+                        if counters[n - 1][end] == counters[n][can] and end in features[n - 1]:
+                            features[n - 1].remove(end)
+
+        return [item for sublist in features.values() for item in sublist]
+
+    def _create_bow_vecs(self, intent_features, sentences, spacy_nlp, max_ngrams=None):
+        """Given a set of sentences, returns a feature vector for each sentence.
+
+        The first $k$ elements are from the `intent_features`,
+        the rest are {1,0} elements denoting whether an ngram is in sentence."""
+
+        import numpy as np
+
+        ngrams_to_use = self._ngrams_to_use(max_ngrams)
+        if not ngrams_to_use:
+            return intent_features
+        extras = np.array(self._ngrams_in_sentences(sentences, spacy_nlp, ngrams=ngrams_to_use))
+        total = np.hstack((intent_features, extras))
+        return total
+
+    def _cross_validation(self, sentences, labels, intent_features, spacy_nlp, max_ngrams):
+        """choose the best number of ngrams to include in bow.
+
+        Given an intent classification problem and a set of ordered ngrams (ordered in terms
+        of importance by pick_applicable_ngrams) we choose the best number of ngrams to include
+        in our bow vecs by cross validation."""
+
+        from sklearn import preprocessing
+        from sklearn.linear_model import LogisticRegression
+        from sklearn.model_selection import cross_val_score
+        import numpy as np
+
+        clf2 = LogisticRegression()
+        intent_encoder = preprocessing.LabelEncoder()
+        intent_encoder.fit(labels)
+        y = intent_encoder.transform(labels)
+        cv_splits = min(10, np.min(np.bincount(y)))
+        if cv_splits >= 3:
+            logging.debug("Started ngram cross-validation to find best number of ngrams to use...")
+            triers = map(int, np.floor(np.linspace(1, max_ngrams, 8)))
+            no_ngrams_X = self._create_bow_vecs(intent_features, sentences, spacy_nlp, max_ngrams=0)
+            no_ngrams_score = np.mean(cross_val_score(clf2, no_ngrams_X, y, cv=cv_splits))
+            scores = []
+            for n in triers:
+                X = self._create_bow_vecs(intent_features, sentences, spacy_nlp, max_ngrams=n)
+                score = np.mean(cross_val_score(clf2, X, y, cv=cv_splits))
+                scores.append(score)
+                logging.debug("Evaluating usage of {} ngrams. Score: {}".format(n, score))
+            n_top = triers[np.argmax(scores)]
+            logging.debug("Score without ngrams: {}".format(no_ngrams_score))
+            logging.info("Best score with {} ngrams: {}".format(n_top, np.max(scores)))
+            return n_top
+        else:
+            warnings.warn("Can't cross-validate ngram featurizer. There aren't enough examples per intent (at least 3)")
+            return max_ngrams
diff --git a/rasa_nlu/featurizers/spacy_featurizer.py b/rasa_nlu/featurizers/spacy_featurizer.py
new file mode 100644
index 000000000000..fdbdbc393dd5
--- /dev/null
+++ b/rasa_nlu/featurizers/spacy_featurizer.py
@@ -0,0 +1,64 @@
+from rasa_nlu.featurizers import Featurizer
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class SpacyFeaturizer(Featurizer, Component):
+    name = "intent_featurizer_spacy"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    def ndim(self, nlp):
+        # type: (Language) -> int
+        from spacy.language import Language
+
+        return nlp.vocab.vectors_length
+
+    def train(self, spacy_nlp, training_data):
+        # type: (Language, TrainingData) -> dict
+        from spacy.language import Language
+
+        sentences = [e["text"] for e in training_data.intent_examples]
+        features = self.features_for_sentences(sentences, spacy_nlp)
+        return {
+            "intent_features": features
+        }
+
+    def process(self, spacy_doc, spacy_nlp):
+        # type: (Doc, Language) -> dict
+        from spacy.language import Language
+        from spacy.tokens import Doc
+
+        features = self.features_for_doc(spacy_doc, spacy_nlp)
+        return {
+            "intent_features": features
+        }
+
+    def features_for_doc(self, doc, nlp):
+        # type: (Doc, Language) -> np.ndarray
+        import numpy as np
+        from spacy.language import Language
+        from spacy.tokens import Doc
+
+        vec = []
+        for token in doc:
+            if token.has_vector:
+                vec.append(token.vector)
+        if vec:
+            return np.sum(vec, axis=0) / len(vec)
+        else:
+            return np.zeros(self.ndim(nlp))
+
+    def features_for_sentences(self, sentences, nlp):
+        # type: ([str], Language) -> np.ndarray
+        import numpy as np
+        from spacy.language import Language
+
+        X = np.zeros((len(sentences), self.ndim(nlp)))
+        for idx, sentence in enumerate(sentences):
+            doc = nlp(sentence)
+            X[idx, :] = self.features_for_doc(doc, nlp)
+        return X
diff --git a/rasa_nlu/model.py b/rasa_nlu/model.py
new file mode 100644
index 000000000000..39553b1d292b
--- /dev/null
+++ b/rasa_nlu/model.py
@@ -0,0 +1,272 @@
+import datetime
+import json
+import logging
+import os
+
+from typing import Optional
+
+import rasa_nlu.components
+from rasa_nlu.components import Component
+from rasa_nlu.config import RasaNLUConfig
+from rasa_nlu.persistor import Persistor
+from rasa_nlu.training_data import TrainingData
+
+
+class InvalidModelError(Exception):
+    """Raised when a model failed to load.
+
+    Attributes:
+        message -- explanation of why the model is invalid
+    """
+
+    def __init__(self, message):
+        self.message = message
+
+
+class Metadata(object):
+    """Captures all necessary information about a model to load it and prepare it for usage."""
+
+    @staticmethod
+    def load(model_dir):
+        # type: (str) -> 'Metadata'
+        """Loads the metadata from a models directory."""
+
+        with open(os.path.join(model_dir, 'metadata.json'), 'rb') as f:
+            data = json.loads(f.read().decode('utf-8'))
+        return Metadata(data, model_dir)
+
+    def __init__(self, metadata, model_dir):
+        # type: (dict, Optional[str]) -> None
+
+        self.metadata = metadata
+        self.model_dir = model_dir
+
+    def __prepend_path(self, prop):
+        if self.metadata.get(prop) is not None:
+            return os.path.normpath(os.path.join(self.model_dir, self.metadata[prop]))
+        else:
+            return None
+
+    @property
+    def language(self):
+        # type: () -> Optional[str]
+        """Language of the underlying model"""
+
+        return self.metadata.get('language')
+
+    @property
+    def pipeline(self):
+        # type: () -> [str]
+        """Names of the processing pipeline elements."""
+
+        if 'pipeline' in self.metadata:
+            return self.metadata['pipeline']
+        elif 'model_template' in self.metadata:
+            from rasa_nlu import registry
+            return registry.registered_model_templates.get(self.metadata.get('model_template'))
+        else:
+            return []
+
+    def persist(self, model_dir):
+        # type: (str) -> None
+        """Persists the metadata of a model to a given directory."""
+
+        metadata = self.metadata.copy()
+
+        metadata.update({
+            "trained_at": datetime.datetime.now().strftime('%Y%m%d-%H%M%S'),
+            "rasa_nlu_version": rasa_nlu.__version__,
+        })
+
+        with open(os.path.join(model_dir, 'metadata.json'), 'w') as f:
+            f.write(json.dumps(metadata, indent=4))
+
+
+class Trainer(object):
+    """Given a pipeline specification and configuration this trainer will load the data and train all components."""
+
+    # Officially supported languages (others might be used, but might fail)
+    SUPPORTED_LANGUAGES = ["de", "en"]
+
+    def __init__(self, config):
+        from rasa_nlu.registry import get_component_class
+
+        self.config = config
+        self.training_data = None
+        self.pipeline = []
+
+        # Transform the passed names of the pipeline components into classes
+        for component_name in config.pipeline:
+            component_class = get_component_class(component_name)
+            if component_class is not None:
+                self.pipeline.append(component_class())
+            else:
+                raise Exception("Unregistered component '{}'. Failed to start trainer.".format(component_name))
+
+    def validate(self):
+        # type: () -> None
+        """Validates a pipeline before it is run. Ensures, that all arguments are present to train the pipeline."""
+
+        # Validate the init phase
+        context = {}
+
+        for component in self.pipeline:
+            try:
+                rasa_nlu.components.fill_args(component.pipeline_init_args(), context, self.config)
+                updates = component.context_provides.get("pipeline_init", [])
+                for u in updates:
+                    context[u] = None
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to validate at component '{}'. {}".format(component.name, e.message))
+
+        after_init_context = context.copy()
+
+        context["training_data"] = None     # Prepare context for testing the training phase
+
+        for component in self.pipeline:
+            try:
+                rasa_nlu.components.fill_args(component.train_args(), context, self.config)
+                updates = component.context_provides.get("train", [])
+                for u in updates:
+                    context[u] = None
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to validate at component '{}'. {}".format(component.name, e.message))
+
+        # Reset context to test processing phase and prepare for training phase
+        context = after_init_context
+        context["text"] = None
+
+        for component in self.pipeline:
+            try:
+                rasa_nlu.components.fill_args(component.process_args(), context, self.config)
+                updates = component.context_provides.get("process", [])
+                for u in updates:
+                    context[u] = None
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to validate at component '{}'. {}".format(component.name, e.message))
+
+    def train(self, data):
+        # type: (TrainingData) -> Interpreter
+        """Trains the underlying pipeline by using the provided training data."""
+
+        self.training_data = data
+
+        context = {}
+
+        for component in self.pipeline:
+            args = rasa_nlu.components.fill_args(component.pipeline_init_args(), context, self.config)
+            updates = component.pipeline_init(*args)
+            if updates:
+                context.update(updates)
+
+        init_context = context.copy()
+
+        context["training_data"] = data
+
+        for component in self.pipeline:
+            args = rasa_nlu.components.fill_args(component.train_args(), context, self.config)
+            updates = component.train(*args)
+            if updates:
+                context.update(updates)
+
+        return Interpreter(self.pipeline, context=init_context, config=self.config)
+
+    def persist(self, path, persistor=None, create_unique_subfolder=True):
+        # type: (str, Optional[Persistor], bool) -> str
+        """Persist all components of the pipeline to the passed path. Returns the directory of the persited model."""
+
+        timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
+        metadata = {
+            "language": self.config["language"],
+            "pipeline": [component.name for component in self.pipeline],
+        }
+
+        if create_unique_subfolder:
+            dir_name = os.path.join(path, "model_" + timestamp)
+            os.makedirs(dir_name)
+        else:
+            dir_name = path
+
+        metadata.update(self.training_data.persist(dir_name))
+
+        for component in self.pipeline:
+            update = component.persist(dir_name)
+            if update:
+                metadata.update(update)
+
+        Metadata(metadata, dir_name).persist(dir_name)
+
+        if persistor is not None:
+            persistor.send_tar_to_s3(dir_name)
+        logging.info("Successfully saved model into '{}'".format(dir_name))
+        return dir_name
+
+
+class Interpreter(object):
+    """Use a trained pipeline of components to parse text messages"""
+
+    # Defines all attributes (and their default values) that will be returned by `parse`
+    default_output_attributes = {"intent": None, "entities": [], "text": ""}
+
+    @staticmethod
+    def load(meta, rasa_config):
+        # type: (Metadata, RasaNLUConfig) -> Interpreter
+        """Load a stored model and its components defined by the provided metadata."""
+        from rasa_nlu.registry import load_component_by_name
+
+        context = {"model_dir": meta.model_dir}
+
+        config = dict(rasa_config.items())
+        config.update(meta.metadata)
+
+        pipeline = []
+
+        for component_name in meta.pipeline:
+            try:
+                component = load_component_by_name(component_name, context, config)
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to create/load component '{}'. {}".format(component_name, e.message))
+            try:
+                rasa_nlu.components.init_component(component, context, config)
+                pipeline.append(component)
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to initialize component '{}'. {}".format(component.name, e.message))
+
+        return Interpreter(pipeline, context, config, meta)
+
+    def __init__(self, pipeline, context, config, meta=None):
+        # type: ([Component], dict, dict, Optional[Metadata]) -> None
+
+        self.pipeline = pipeline
+        self.context = context
+        self.config = config
+        self.meta = meta
+        self.output_attributes = [output for component in pipeline for output in component.output_provides]
+
+    def parse(self, text):
+        # type: (basestring) -> dict
+        """Parse the input text, classify it and return an object containing its intent and entities."""
+
+        if type(text) is str:
+            text = unicode(text, "utf-8")
+
+        current_context = self.context.copy()
+
+        current_context.update({
+            "text": text,
+        })
+
+        for component in self.pipeline:
+            try:
+                args = rasa_nlu.components.fill_args(component.process_args(), current_context, self.config)
+                updates = component.process(*args)
+                if updates:
+                    current_context.update(updates)
+            except rasa_nlu.components.MissingArgumentError as e:
+                raise Exception("Failed to parse at component '{}'. {}".format(component.name, e.message))
+
+        result = self.default_output_attributes.copy()
+        all_attributes = self.default_output_attributes.keys() + self.output_attributes
+        # Ensure only keys of `all_attributes` are present and no other keys are returned
+        result.update({key: current_context[key] for key in all_attributes if key in current_context})
+        return result
diff --git a/src/persistor.py b/rasa_nlu/persistor.py
similarity index 77%
rename from src/persistor.py
rename to rasa_nlu/persistor.py
index bb24020c1b9b..92957ccea1a8 100644
--- a/src/persistor.py
+++ b/rasa_nlu/persistor.py
@@ -1,22 +1,27 @@
-import boto3
-import botocore
-import tarfile
 import os
 import shutil
+import tarfile
+
+import boto3
+import botocore
 
 
 class Persistor(object):
+    """Store models on S3 and fetch them when needed instead of storing them on the local disk."""
+
     def __init__(self, data_dir, aws_region, bucket_name):
         self.data_dir = data_dir
         self.s3 = boto3.resource('s3', region_name=aws_region)
         self.bucket_name = bucket_name
         try:
             self.s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': aws_region})
-        except botocore.exceptions.ClientError, e:
+        except botocore.exceptions.ClientError as e:
             pass  # bucket already exists
         self.bucket = self.s3.Bucket(bucket_name)
 
     def send_tar_to_s3(self, target_dir):
+        # type: (str) -> None
+        """Uploads a model persisted in the `target_dir` to s3."""
 
         if not os.path.isdir(target_dir):
             raise ValueError('target_dir %r not found.' % target_dir)
@@ -28,6 +33,9 @@ def send_tar_to_s3(self, target_dir):
         self.s3.Object(self.bucket_name, filekey).put(Body=open(tarname, 'rb'))
 
     def fetch_and_extract(self, filename):
+        # type: (str) -> None
+        """Downloads a model that has previously been persisted to s3."""
+
         with open(filename, 'wb') as f:
             self.bucket.download_fileobj(filename, f)
         with tarfile.open(filename, "r:gz") as tar:
diff --git a/rasa_nlu/registry.py b/rasa_nlu/registry.py
new file mode 100644
index 000000000000..c9f63d4e974e
--- /dev/null
+++ b/rasa_nlu/registry.py
@@ -0,0 +1,76 @@
+"""This is a somewhat delicate package. It contains all registered components and preconfigured templates.
+
+Hence, it imports all of the components. To avoid cycles, no component should import this in module scope."""
+from typing import Optional
+from typing import Type
+
+from rasa_nlu.classifiers.keyword_intent_classifier import KeywordIntentClassifier
+from rasa_nlu.classifiers.mitie_intent_classifier import MitieIntentClassifier
+from rasa_nlu.classifiers.sklearn_intent_classifier import SklearnIntentClassifier
+from rasa_nlu.extractors.entity_synonyms import EntitySynonymMapper
+from rasa_nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
+from rasa_nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
+from rasa_nlu.featurizers.mitie_featurizer import MitieFeaturizer
+from rasa_nlu.featurizers.ngram_featurizer import NGramFeaturizer
+from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
+from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa_nlu.utils.mitie_utils import MitieNLP
+from rasa_nlu.utils.spacy_utils import SpacyNLP
+
+# Classes of all known components. If a new component should be added, its class needs to be listed here.
+component_classes = [
+    SpacyNLP, SpacyEntityExtractor, SklearnIntentClassifier, SpacyFeaturizer,
+    MitieNLP, MitieEntityExtractor, MitieIntentClassifier, MitieFeaturizer, MitieTokenizer,
+    KeywordIntentClassifier, EntitySynonymMapper, NGramFeaturizer]
+
+# Mapping from a components name to its class to allow name based lookup.
+registered_components = {component.name: component for component in component_classes}
+
+# To simplify usage, there are a couple of model templates, that already add necessary components in the right order.
+# They also implement the preexisting `backends`.
+registered_model_templates = {
+    "spacy_sklearn": [
+        "init_spacy",
+        "ner_spacy",
+        "ner_synonyms",
+        "intent_featurizer_spacy",
+        "intent_classifier_sklearn",
+    ],
+    "mitie": [
+        "init_mitie",
+        "tokenizer_mitie",
+        "ner_mitie",
+        "ner_synonyms",
+        "intent_featurizer_mitie",
+        "intent_classifier_mitie",
+    ],
+    "mitie_sklearn": [
+        "init_mitie",
+        "tokenizer_mitie",
+        "ner_mitie",
+        "ner_synonyms",
+        "intent_featurizer_mitie",
+        "intent_classifier_sklearn",
+    ],
+    "keyword": [
+        "intent_classifier_keyword",
+    ]
+}
+
+
+def get_component_class(component_name):
+    # type: (str) -> Optional[Type[Component]]
+    """Resolve component name to a registered components class."""
+    from rasa_nlu.components import Component
+
+    return registered_components.get(component_name)
+
+
+def load_component_by_name(component_name, context, config):
+    # type: (str, dict, dict) -> Optional[Component]
+    """Resolves a components name and calls it's load method to init it based on a previously persisted model."""
+    from rasa_nlu.components import load_component
+    from rasa_nlu.components import Component
+
+    component_clz = get_component_class(component_name)
+    return load_component(component_clz, context, config)
diff --git a/src/server.py b/rasa_nlu/server.py
similarity index 98%
rename from src/server.py
rename to rasa_nlu/server.py
index 402717b11d35..db5f0be15e39 100644
--- a/src/server.py
+++ b/rasa_nlu/server.py
@@ -35,6 +35,8 @@ def create_arg_parser():
 
 
 def requires_auth(f):
+    """Wraps a request handler with token authentication."""
+
     @wraps(f)
     def decorated(*args, **kwargs):
         token = request.args.get('token', '')
diff --git a/rasa_nlu/tokenizers/__init__.py b/rasa_nlu/tokenizers/__init__.py
new file mode 100644
index 000000000000..9e0aa395a4a2
--- /dev/null
+++ b/rasa_nlu/tokenizers/__init__.py
@@ -0,0 +1,19 @@
+def tokenizer_from_name(name, language):
+    from rasa_nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
+    from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+    from rasa_nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
+
+    if name == MitieTokenizer.name:
+        return MitieTokenizer()
+    elif name == SpacyTokenizer.name:
+        import spacy
+        nlp = spacy.load(language, parser=False, entity=False)
+        return SpacyTokenizer(nlp)
+    elif name == WhitespaceTokenizer.name:
+        return WhitespaceTokenizer()
+
+
+class Tokenizer(object):
+    def tokenize(self, text):
+        # type: (str) -> [str]
+        raise NotImplementedError
diff --git a/src/tokenizers/mitie_tokenizer.py b/rasa_nlu/tokenizers/mitie_tokenizer.py
similarity index 65%
rename from src/tokenizers/mitie_tokenizer.py
rename to rasa_nlu/tokenizers/mitie_tokenizer.py
index 079ef07c4e2c..89ea80a1a560 100644
--- a/src/tokenizers/mitie_tokenizer.py
+++ b/rasa_nlu/tokenizers/mitie_tokenizer.py
@@ -1,19 +1,36 @@
-import logging
 import re
 
-from mitie import tokenize
-
 from rasa_nlu.tokenizers import Tokenizer
+from rasa_nlu.components import Component
+
 
+class MitieTokenizer(Tokenizer, Component):
+    name = "tokenizer_mitie"
+
+    context_provides = {
+        "process": ["tokens"],
+    }
 
-class MITIETokenizer(Tokenizer):
     def __init__(self):
         pass
 
     def tokenize(self, text):
+        # type: (str) -> [str]
+        from mitie import tokenize
+
         return [w.decode('utf-8') for w in tokenize(text.encode('utf-8'))]
 
+    def process(self, text):
+        # type: (str) -> dict
+
+        return {
+            "tokens": self.tokenize(text)
+        }
+
     def tokenize_with_offsets(self, text):
+        # type: (str) -> ([str], [int])
+        from mitie import tokenize
+
         _text = text.encode('utf-8')
         offsets = []
         offset = 0
diff --git a/rasa_nlu/tokenizers/spacy_tokenizer.py b/rasa_nlu/tokenizers/spacy_tokenizer.py
new file mode 100644
index 000000000000..646d4c096791
--- /dev/null
+++ b/rasa_nlu/tokenizers/spacy_tokenizer.py
@@ -0,0 +1,31 @@
+from rasa_nlu.tokenizers import Tokenizer
+from rasa_nlu.components import Component
+
+
+class SpacyTokenizer(Tokenizer, Component):
+    name = "tokenizer_spacy"
+
+    context_provides = {
+        "process": ["tokens"],
+    }
+
+    def __init__(self, nlp=None):
+        self.nlp = nlp
+
+    def pipeline_init(self, spacy_nlp):
+        # type: (Language) -> None
+        from spacy.language import Language
+
+        self.nlp = spacy_nlp
+
+    def process(self, text):
+        # type: (str) -> dict
+
+        return {
+            "tokens": self.tokenize(text)
+        }
+
+    def tokenize(self, text):
+        # type: (str) -> [str]
+
+        return [t.text for t in self.nlp(text)]
diff --git a/rasa_nlu/tokenizers/whitespace_tokenizer.py b/rasa_nlu/tokenizers/whitespace_tokenizer.py
new file mode 100644
index 000000000000..63f41723d06b
--- /dev/null
+++ b/rasa_nlu/tokenizers/whitespace_tokenizer.py
@@ -0,0 +1,22 @@
+from rasa_nlu.tokenizers import Tokenizer
+from rasa_nlu.components import Component
+
+
+class WhitespaceTokenizer(Tokenizer, Component):
+    name = "tokenizer_whitespace"
+
+    context_provides = {
+        "process": ["tokens"],
+    }
+
+    def process(self, text):
+        # type: (str) -> dict
+
+        return {
+            "tokens": self.tokenize(text)
+        }
+
+    def tokenize(self, text):
+        # type: (str) -> [str]
+
+        return text.split()
diff --git a/rasa_nlu/train.py b/rasa_nlu/train.py
new file mode 100644
index 000000000000..68637ea4b909
--- /dev/null
+++ b/rasa_nlu/train.py
@@ -0,0 +1,71 @@
+import argparse
+import logging
+import os
+
+from rasa_nlu.converters import load_data
+from rasa_nlu.model import Trainer
+
+from rasa_nlu.config import RasaNLUConfig
+from typing import Optional
+
+
+def create_argparser():
+    parser = argparse.ArgumentParser(description='train a custom language parser')
+
+    parser.add_argument('-b', '--model_template', default=None, choices=['mitie', 'spacy_sklearn', 'keyword'],
+                        help='model template to use to interpret text (default: built in keyword matcher).')
+    parser.add_argument('-p', '--pipeline', default=None,
+                        help='pipeline to use for the message processing.')
+    parser.add_argument('-o', '--output', default=None, help="path where model files will be saved")
+    parser.add_argument('-d', '--data', default=None, help="file containing training data")
+    parser.add_argument('-c', '--config', required=True, help="config file")
+    parser.add_argument('-l', '--language', default=None, choices=['de', 'en'], help="model and data language")
+    parser.add_argument('-t', '--num_threads', default=1, type=int,
+                        help="number of threads to use during model training")
+    parser.add_argument('-m', '--mitie_file', default=None,
+                        help='file with mitie total_word_feature_extractor')
+    return parser
+
+
+def create_persistor(config):
+    # type: (RasaNLUConfig) -> Optional[Persistor]
+    """Create a remote peristor to store the model if the configuration requests it."""
+
+    persistor = None
+    if "bucket_name" in config:
+        from rasa_nlu.persistor import Persistor
+        persistor = Persistor(config['path'], config['aws_region'], config['bucket_name'])
+
+    return persistor
+
+
+def init():
+    # type: () -> RasaNLUConfig
+    """Combines passed arguments to create rasa NLU config."""
+
+    parser = create_argparser()
+    args = parser.parse_args()
+    config = RasaNLUConfig(args.config, os.environ, vars(args))
+    return config
+
+
+def do_train(config):
+    # type: (RasaNLUConfig) -> (Trainer, str)
+    """Loads the trainer and the data and runs the training of the specified model."""
+
+    trainer = Trainer(config)
+    persistor = create_persistor(config)
+    training_data = load_data(config['data'], config['language'], luis_data_tokenizer=config['luis_data_tokenizer'])
+    trainer.validate()
+    trainer.train(training_data)
+    persisted_path = trainer.persist(config['path'], persistor)
+    return trainer, persisted_path
+
+
+if __name__ == '__main__':
+
+    config = init()
+    logging.basicConfig(level=config['log_level'])
+
+    do_train(config)
+    logging.info("done")
diff --git a/rasa_nlu/training_data.py b/rasa_nlu/training_data.py
new file mode 100644
index 000000000000..7cb56d9a2620
--- /dev/null
+++ b/rasa_nlu/training_data.py
@@ -0,0 +1,95 @@
+# -*- coding: utf-8 -*-
+
+import json
+import os
+import warnings
+from itertools import groupby
+
+
+class TrainingData(object):
+    """Holds loaded intent and entity training data."""
+
+    # Validation will ensure and warn if these lower limits are not met
+    MIN_EXAMPLES_PER_INTENT = 2
+    MIN_EXAMPLES_PER_ENTITY = 2
+
+    def __init__(self,
+                 intent_examples_only=None,
+                 entity_examples_only=None,
+                 common_examples=None,
+                 entity_synonyms=None):
+        self.intent_examples_only = intent_examples_only if intent_examples_only else []
+        self.entity_examples_only = entity_examples_only if entity_examples_only else []
+        self.common_examples = common_examples if common_examples else []
+        self.entity_synonyms = entity_synonyms if entity_synonyms else {}
+
+        self.validate()
+
+    @property
+    def intent_examples(self):
+        return self.intent_examples_only + self.common_examples
+
+    @property
+    def entity_examples(self):
+        return self.entity_examples_only + self.common_examples
+
+    @property
+    def num_entity_examples(self):
+        # type: () -> int
+        """Returns the number of entity examples."""
+
+        return len([e for e in self.entity_examples if len(e["entities"]) > 0])
+
+    def as_json(self, **kwargs):
+        # type: (dict) -> str
+        """Represent this set of training examples as json adding the passed meta information."""
+
+        return json.dumps({
+            "rasa_nlu_data": {
+                "common_examples": self.common_examples,
+                "intent_examples": self.intent_examples_only,
+                "entity_examples": self.entity_examples_only,
+            }
+        }, **kwargs)
+
+    def persist(self, dir_name):
+        # type: (str) -> dict
+        """Persists this training data to disk and returnes necessary information to load it again."""
+
+        data_file = os.path.join(dir_name, "training_data.json")
+        with open(data_file, 'w') as f:
+            f.write(self.as_json(indent=2))
+
+        return {
+            "training_data": "training_data.json"
+        }
+
+    def sorted_entity_examples(self):
+        # type: () -> [dict]
+        """Sorts the entity examples by the annotated entity."""
+
+        return sorted([entity for ex in self.entity_examples for entity in ex["entities"]], key=lambda e: e["entity"])
+
+    def sorted_intent_examples(self):
+        # type: () -> [dict]
+        """Sorts the intent examples by the name of the intent."""
+
+        return sorted(self.intent_examples, key=lambda e: e["intent"])
+
+    def validate(self):
+        # type: () -> None
+        """Ensures that the loaded training data is valid, e.g. has a minimum of certain training examples."""
+
+        examples = self.sorted_intent_examples()
+        for intent, group in groupby(examples, lambda e: e["intent"]):
+            size = len(list(group))
+            if size < self.MIN_EXAMPLES_PER_INTENT:
+                template = u"Intent '{0}' has only {1} training examples! minimum is {2}, training may fail."
+                warnings.warn(template.format(intent, size, self.MIN_EXAMPLES_PER_INTENT))
+
+        sorted_entity_examples = self.sorted_entity_examples()
+        for entity, group in groupby(sorted_entity_examples, lambda e: e["entity"]):
+            size = len(list(group))
+            if size < self.MIN_EXAMPLES_PER_ENTITY:
+                template = u"Entity '{0}' has only {1} training examples! minimum is {2}, training may fail."
+                warnings.warn(template.format(entity, size, self.MIN_EXAMPLES_PER_ENTITY))
diff --git a/src/util.py b/rasa_nlu/util.py
similarity index 72%
rename from src/util.py
rename to rasa_nlu/util.py
index 921445724890..f35486c2c873 100644
--- a/src/util.py
+++ b/rasa_nlu/util.py
@@ -1,7 +1,22 @@
 import os
 
+from typing import Optional
+
+
+def relative_normpath(f, path):
+    # type: (Optional[str], str) -> Optional[str]
+    """Return the path of file relative to `path`."""
+
+    if f is not None:
+        return os.path.normpath(os.path.relpath(f, path))
+    else:
+        return None
+
 
 def create_dir_for_file(file_path):
+    # type: (str) -> None
+    """Creates any missing parent directories of this files path."""
+
     try:
         os.makedirs(os.path.dirname(file_path))
     except OSError:
@@ -10,7 +25,11 @@ def create_dir_for_file(file_path):
 
 
 def recursively_find_files(resource_name):
-    """resource_name can be a folder or a file. In both cases we will return a list of files"""
+    # type: (str) -> [str]
+    """Traverse directory hirachry to find files.
+
+    `resource_name` can be a folder or a file. In both cases we will return a list of files."""
+
     if not resource_name:
         raise ValueError("Resource name '{}' must be an existing directory or file.".format(resource_name))
     elif os.path.isfile(resource_name):
@@ -35,12 +54,3 @@ def recursively_find_files(resource_name):
         raise ValueError("Could not locate the resource '{}'.".format(os.path.abspath(resource_name)))
     else:
         raise ValueError("Resource name must be an existing directory or file")
-
-
-def add_entities_if_synonyms(synonyms_dict, entity_a, entity_b):
-    if entity_b is not None:
-        original = entity_a.lower() if type(entity_a) == unicode else unicode(entity_a)
-        replacement = entity_b.lower() if type(entity_b) == unicode else unicode(entity_b)
-
-        if original != replacement:
-            synonyms_dict[original] = replacement
diff --git a/src/extractors/__init__.py b/rasa_nlu/utils/__init__.py
similarity index 100%
rename from src/extractors/__init__.py
rename to rasa_nlu/utils/__init__.py
diff --git a/rasa_nlu/utils/mitie_utils.py b/rasa_nlu/utils/mitie_utils.py
new file mode 100644
index 000000000000..c8ebdfc9116f
--- /dev/null
+++ b/rasa_nlu/utils/mitie_utils.py
@@ -0,0 +1,46 @@
+from mitie import total_word_feature_extractor
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.model import Metadata
+
+
+class MitieNLP(Component):
+
+    name = "init_mitie"
+
+    context_provides = {
+        "pipeline_init": ["mitie_feature_extractor"],
+    }
+
+    def __init__(self, extractor=None):
+        self.extractor = extractor
+
+    @classmethod
+    def cache_key(cls, model_metadata):
+        # type: (Metadata) -> str
+
+        return cls.name + "-" + str(model_metadata.metadata.get("mitie_feature_extractor_fingerprint"))
+
+    def pipeline_init(self, mitie_file):
+        # type: (str) -> dict
+
+        if self.extractor is None:
+            self.extractor = total_word_feature_extractor(mitie_file)
+        MitieNLP.ensure_proper_language_model(self.extractor)
+        return {"mitie_feature_extractor": self.extractor}
+
+    @staticmethod
+    def ensure_proper_language_model(extractor):
+        # type: (Optional[mitie.total_word_feature_extractor]) -> None
+        import mitie
+
+        if extractor is None:
+            raise Exception("Failed to load MITIE feature extractor. Loading the model returned 'None'.")
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+
+        return {
+            "mitie_feature_extractor_fingerprint": self.extractor.fingerprint
+        }
diff --git a/rasa_nlu/utils/spacy_utils.py b/rasa_nlu/utils/spacy_utils.py
new file mode 100644
index 000000000000..4e6c26ebd020
--- /dev/null
+++ b/rasa_nlu/utils/spacy_utils.py
@@ -0,0 +1,59 @@
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.model import Metadata
+
+
+class SpacyNLP(Component):
+    name = "init_spacy"
+
+    context_provides = {
+        "pipeline_init": ["spacy_nlp"],
+        "process": ["spacy_doc"],
+    }
+
+    def __init__(self, nlp=None):
+        # type: (Optional[Language]) -> None
+        from spacy.language import Language
+
+        self.nlp = nlp
+
+    @classmethod
+    def cache_key(cls, model_metadata):
+        # type: (Metadata) -> str
+
+        return cls.name + "-" + model_metadata.language
+
+    def pipeline_init(self, language, fine_tune_spacy_ner):
+        # type: (str, Optional[bool]) -> dict
+        import spacy
+
+        # If fine tuning is disabled, we do not need to load the spacy entity model
+        if self.nlp is None:
+            if fine_tune_spacy_ner:
+                self.nlp = spacy.load(language, parser=False)
+            else:
+                self.nlp = spacy.load(language, parser=False, entity=False)
+        SpacyNLP.ensure_proper_language_model(self.nlp)
+        return {"spacy_nlp": self.nlp}
+
+    def process(self, text):
+        # type: (str) -> dict
+
+        return {
+            "spacy_doc": self.nlp(text, entity=False)  # need to set entity=false, otherwise it interferes with our NER
+        }
+
+    @staticmethod
+    def ensure_proper_language_model(nlp):
+        # type: (Optional[Language]) -> None
+        """Checks if the spacy language model is properly loaded. Raises an exception if the model is invalid."""
+        from spacy.language import Language
+
+        if nlp is None:
+            raise Exception("Failed to load spacy language model. Loading the model returned 'None'.")
+        if nlp.path is None:
+            # Spacy sets the path to `None` if it did not load the model from disk.
+            # In this case `nlp` is an unusable stub.
+            raise Exception("Failed to load spacy language model for lang '{}'. ".format(nlp.lang) +
+                            "Make sure you have downloaded the correct model (https://spacy.io/docs/usage/).")
diff --git a/rasa_nlu/version.py b/rasa_nlu/version.py
new file mode 100644
index 000000000000..a2c9b2c07d25
--- /dev/null
+++ b/rasa_nlu/version.py
@@ -0,0 +1 @@
+__version__ = '0.8.0a1'
diff --git a/src/wsgi.py b/rasa_nlu/wsgi.py
similarity index 100%
rename from src/wsgi.py
rename to rasa_nlu/wsgi.py
diff --git a/requirements.txt b/requirements.txt
index 1842103e6158..299359199064 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,7 @@
 -e .
 gevent==1.2.1
 flask==0.12
+boto3==1.4.4
 git+https://github.com/mit-nlp/MITIE.git#egg=mitie
 spacy==1.6.0
 scikit-learn==0.18.1
@@ -10,3 +11,4 @@ pytest-services==1.1.14
 pytest-flask==0.10.0
 scipy==0.19.0
 pytest==3.0.7
+typing==3.5.3.0
diff --git a/setup.py b/setup.py
index 0657a8a2306a..efa3a1e27697 100644
--- a/setup.py
+++ b/setup.py
@@ -1,7 +1,7 @@
 from setuptools import setup
 
 __version__ = None   # Avoids IDE errors, but actual version is read from version.py
-exec(open('src/version.py').read())
+exec(open('rasa_nlu/version.py').read())
 
 tests_requires = [
     "pytest-pep8",
@@ -15,6 +15,7 @@
     "pathlib",
     "cloudpickle",
     "boto3",
+    "typing",
 ]
 
 extras_requires = {
@@ -33,11 +34,8 @@
         'rasa_nlu.emulators',
         'rasa_nlu.extractors',
         'rasa_nlu.featurizers',
-        'rasa_nlu.interpreters',
-        'rasa_nlu.trainers',
         'rasa_nlu.tokenizers',
     ],
-    package_dir={'rasa_nlu': 'src'},
     version=__version__,
     install_requires=install_requires,
     tests_require=tests_requires,
diff --git a/src/__init__.py b/src/__init__.py
deleted file mode 100644
index d11a695c2029..000000000000
--- a/src/__init__.py
+++ /dev/null
@@ -1,25 +0,0 @@
-from pkg_resources import get_distribution
-import codecs
-import json
-import os
-import version
-
-__version__ = version.__version__
-
-
-class Interpreter(object):
-    def parse(self, text):
-        raise NotImplementedError()
-
-    @staticmethod
-    def load_synonyms(entity_synonyms_file):
-        if entity_synonyms_file and os.path.isfile(entity_synonyms_file):
-            with codecs.open(entity_synonyms_file, encoding='utf-8') as infile:
-                return json.loads(infile.read())
-
-    @staticmethod
-    def replace_synonyms(entities, entity_synonyms):
-        for i in range(len(entities)):
-            entity_value = entities[i]["value"]
-            if entity_value.lower() in entity_synonyms:
-                entities[i]["value"] = entity_synonyms[entity_value.lower()]
diff --git a/src/classifiers/sklearn_intent_classifier.py b/src/classifiers/sklearn_intent_classifier.py
deleted file mode 100644
index 99f014cdc231..000000000000
--- a/src/classifiers/sklearn_intent_classifier.py
+++ /dev/null
@@ -1,86 +0,0 @@
-import logging
-
-import numpy as np
-from sklearn.model_selection import GridSearchCV
-from sklearn.model_selection import train_test_split
-from sklearn.preprocessing import LabelEncoder
-from sklearn.svm import SVC
-
-
-class SklearnIntentClassifier(object):
-    """Intent classifier using the sklearn framework"""
-
-    def __init__(self, uses_probabilities=True, max_num_threads=1):
-        """Construct a new intent classifier using the sklearn framework.
-
-        :param uses_probabilities: defines if the model should be trained
-                                           to be able to predict label probabilities
-        :param max_num_threads: number of threads used during training time
-        :type uses_probabilities: bool"""
-
-        self.le = LabelEncoder()
-        self.uses_probabilities = uses_probabilities
-        self.tuned_parameters = [{'C': [1, 2, 5, 10, 20, 100], 'kernel': ['linear']}]
-        self.score = 'f1'
-        self.clf = GridSearchCV(SVC(C=1, probability=uses_probabilities),
-                                self.tuned_parameters, n_jobs=max_num_threads,
-                                cv=2, scoring='%s_weighted' % self.score)
-
-    def transform_labels_str2num(self, labels):
-        """Transforms a list of strings into numeric label representation.
-
-        :param labels: List of labels to convert to numeric representation
-        :type labels: list of str"""
-
-        y = self.le.fit_transform(labels)
-        return y
-
-    def transform_labels_num2str(self, y):
-        """Transforms a list of strings into numeric label representation.
-
-        :param labels: List of labels to convert to numeric representation
-        :type labels: list of str"""
-
-        labels = self.le.inverse_transform(y)
-        return labels
-
-    def train(self, X, y, test_split_size=0.1):
-        """Train the intent classifier on a data set.
-
-        :param test_split_size: defines the percentage of examples to reserve for testing
-        :param X: Train data set
-        :param y: Train labels (numeric)"""
-
-        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_split_size, random_state=0)
-        self.clf.fit(X_train, y_train)
-
-        # Test the trained model
-        if test_split_size != 0.0:
-            logging.info("Score of intent model on test data: %s " % self.clf.score(X_test, y_test))
-
-    def predict_prob(self, X):
-        """Given a bow vector of an input text, predict the intent label. Returns probabilities for all labels.
-
-        :param X: bow of input text
-        :return: vector of probabilities containing one entry for each label"""
-
-        if hasattr(self, 'uses_probabilities') and self.uses_probabilities:
-            return self.clf.predict_proba(X)
-        else:
-            y_pred_indices = self.clf.predict(X)
-            # convert representation to one-hot. all labels are zero, only the predicted label gets assigned prob=1
-            y_pred = np.zeros((np.size(X, 0), len(self.le.classes_)))
-            y_pred[np.arange(y_pred.shape[0]), y_pred_indices] = 1
-            return y_pred
-
-    def predict(self, X):
-        """Given a bow vector of an input text, predict most probable label. Returns only the most likely label.
-
-        :param X: bow of input text
-        :return: tuple of first, the most probable label and second, its probability"""
-
-        pred_result = self.predict_prob(X)
-        max_indicies = np.argmax(pred_result, axis=1)
-        # retrieve the index of the intent with the highest probability
-        max_values = pred_result[:, max_indicies].flatten()
-        return max_indicies, max_values
diff --git a/src/emulators/luis.py b/src/emulators/luis.py
deleted file mode 100644
index 8a41238fe9b8..000000000000
--- a/src/emulators/luis.py
+++ /dev/null
@@ -1,25 +0,0 @@
-from rasa_nlu.emulators import NoEmulator
-
-
-class LUISEmulator(NoEmulator):
-    def __init__(self):
-        super(LUISEmulator, self).__init__()
-        self.name = 'luis'
-
-    def normalise_response_json(self, data):
-        return {
-            "query": data["text"],
-            "topScoringIntent": {
-                "intent": data["intent"],
-                "score": data["confidence"]
-            },
-            "entities": [
-                {
-                    "entity": e["value"],
-                    "type": e["entity"],
-                    "startIndex": None,
-                    "endIndex": None,
-                    "score": None
-                } for e in data["entities"]
-                ]
-        }
diff --git a/src/extractors/spacy_entity_extractor.py b/src/extractors/spacy_entity_extractor.py
deleted file mode 100644
index c11afbf19dbc..000000000000
--- a/src/extractors/spacy_entity_extractor.py
+++ /dev/null
@@ -1,90 +0,0 @@
-from __future__ import unicode_literals, print_function
-
-import random
-
-import pathlib
-from spacy.gold import GoldParse
-from spacy.pipeline import EntityRecognizer
-import warnings
-
-
-class SpacyEntityExtractor(object):
-    def __init__(self, nlp=None, extractor_file=None, should_fine_tune_spacy_ner=False):
-        self.nlp = nlp
-        if extractor_file:
-            self.ner = EntityRecognizer.load(pathlib.Path(extractor_file), nlp.vocab)
-        else:
-            self.ner = None
-
-        self.should_fine_tune_spacy_ner = should_fine_tune_spacy_ner
-
-    def convert_examples(self, entity_examples):
-        def convert_entity(ent):
-            return ent["start"], ent["end"], ent["entity"]
-
-        def convert_example(ex):
-            return ex["text"], [convert_entity(ent) for ent in ex["entities"]]
-
-        return [convert_example(ex) for ex in entity_examples]
-
-    def train(self, nlp, entity_examples, should_fine_tune_spacy_ner=False):
-        train_data = self.convert_examples(entity_examples)
-        ent_types = [[ent["entity"] for ent in ex["entities"]] for ex in entity_examples]
-        entity_types = list(set(sum(ent_types, [])))
-
-        self.should_fine_tune_spacy_ner = should_fine_tune_spacy_ner
-        self.ner = self._train_from_scratch(nlp, entity_types, train_data)
-
-    def _train_from_scratch(self, nlp, entity_types, train_data):
-        ner = EntityRecognizer(nlp.vocab, entity_types=entity_types)
-        self._update_ner_model(ner, nlp, train_data)
-        return ner
-
-    # TODO: this is not used at the moment as there is an issue in the latest spacy version. Currently there is no way
-    # TODO: to directly fine tune the entity model as that will result in a malloc error during parse time
-    def _fine_tune(self, nlp, entity_types, train_data):
-        if nlp.entity:
-            ner = nlp.entity
-            for entity_type in entity_types:
-                if entity_type not in ner.cfg['actions']['1']:
-                    ner.add_label(entity_type)
-            self._update_ner_model(ner, nlp, train_data)
-            return ner
-        else:
-            warnings.warn("Failed to fine tune model. There was no model to fine tune. ")
-            return None
-
-    def _update_ner_model(self, ner, nlp, train_data):
-        for itn in range(5):
-            random.shuffle(train_data)
-            for raw_text, entity_offsets in train_data:
-                doc = nlp.make_doc(raw_text)
-                gold = GoldParse(doc, entities=entity_offsets)
-                ner.update(doc, gold)
-
-    def extract_entities(self, doc):
-        if self.ner is not None:
-            self.ner(doc)
-
-            if self.nlp.entity is not None and self.should_fine_tune_spacy_ner:
-                sp_doc = self.nlp(doc.text)
-                spacy_ents = sp_doc.ents
-                for spacy_ent in spacy_ents:
-                    for e in doc.ents:
-                        if e.start_char <= spacy_ent.start_char < e.end_char or \
-                              e.start_char <= spacy_ent.end_char < e.end_char:
-                            break
-                    else:
-                        doc.ents += (spacy_ent,)
-
-            entities = [
-                {
-                    "entity": ent.label_,
-                    "value": ent.text,
-                    "start": ent.start_char,
-                    "end": ent.end_char
-                }
-                for ent in doc.ents]
-            return entities
-        else:
-            return []
diff --git a/src/featurizers/mitie_featurizer.py b/src/featurizers/mitie_featurizer.py
deleted file mode 100644
index ed969b2c0ba1..000000000000
--- a/src/featurizers/mitie_featurizer.py
+++ /dev/null
@@ -1,27 +0,0 @@
-from mitie import *
-import numpy as np
-
-from rasa_nlu.featurizers import Featurizer
-
-
-class MITIEFeaturizer(Featurizer):
-    def __init__(self, fe_file):
-        self.feature_extractor = total_word_feature_extractor(fe_file)
-        self.ndim = self.feature_extractor.num_dimensions
-
-    def features_for_tokens(self, tokens):
-        vec = np.zeros(self.ndim)
-        for token in tokens:
-            vec += self.feature_extractor.get_feature_vector(token)
-        if tokens:
-            return vec / len(tokens)
-        else:
-            return vec
-
-    def features_for_sentences(self, sentences):
-        X = np.zeros((len(sentences), self.ndim))
-
-        for idx, sentence in enumerate(sentences):
-            tokens = tokenize(sentence)
-            X[idx, :] = self.features_for_tokens(tokens)
-        return X
diff --git a/src/featurizers/spacy_featurizer.py b/src/featurizers/spacy_featurizer.py
deleted file mode 100644
index c53b79a14f98..000000000000
--- a/src/featurizers/spacy_featurizer.py
+++ /dev/null
@@ -1,26 +0,0 @@
-import numpy as np
-
-from rasa_nlu.featurizers import Featurizer
-
-
-class SpacyFeaturizer(Featurizer):
-    def __init__(self, nlp):
-        self.nlp = nlp
-        self.ndim = 300
-
-    def features_for_doc(self, doc):
-        vec = []
-        for token in doc:
-            if token.has_vector:
-                vec.append(token.vector)
-        if vec:
-            return np.sum(vec, axis=0) / len(vec)
-        else:
-            return np.zeros(self.ndim)
-
-    def features_for_sentences(self, sentences):
-        X = np.zeros((len(sentences), self.ndim))
-        for idx, sentence in enumerate(sentences):
-            doc = self.nlp(sentence)
-            X[idx, :] = self.features_for_doc(doc)
-        return X
diff --git a/src/interpreters/__init__.py b/src/interpreters/__init__.py
deleted file mode 100644
index e69de29bb2d1..000000000000
diff --git a/src/interpreters/mitie_interpreter.py b/src/interpreters/mitie_interpreter.py
deleted file mode 100644
index 3cb89c63c7b9..000000000000
--- a/src/interpreters/mitie_interpreter.py
+++ /dev/null
@@ -1,65 +0,0 @@
-from mitie import *
-from rasa_nlu.featurizers.mitie_featurizer import MITIEFeaturizer
-
-from rasa_nlu import Interpreter
-from rasa_nlu.interpreters.mitie_interpreter_utils import get_entities
-from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
-
-
-class MITIEInterpreter(Interpreter):
-    @staticmethod
-    def load(meta, featurizer=None):
-        """
-        :type meta: rasa_nlu.model.Metadata
-        :rtype: MITIEInterpreter
-        """
-        if meta.entity_extractor_path:
-            extractor = named_entity_extractor(meta.entity_extractor_path)
-        else:
-            extractor = None
-
-        if meta.intent_classifier_path:
-            classifier = text_categorizer(meta.intent_classifier_path)
-        else:
-            classifier = None
-
-        if featurizer is None:
-            featurizer = MITIEFeaturizer(meta.feature_extractor_path)
-
-        if meta.entity_synonyms_path:
-            entity_synonyms = Interpreter.load_synonyms(meta.entity_synonyms_path)
-        else:
-            entity_synonyms = None
-
-        return MITIEInterpreter(
-            classifier,
-            extractor,
-            featurizer,
-            entity_synonyms)
-
-    def __init__(self,
-                 intent_classifier=None,
-                 entity_extractor=None,
-                 featurizer=None,
-                 entity_synonyms=None):
-        self.extractor = entity_extractor
-        self.featurizer = featurizer
-        self.classifier = intent_classifier
-        self.ent_synonyms = entity_synonyms
-        self.tokenizer = MITIETokenizer()
-
-    def get_intent(self, tokens):
-        if self.classifier:
-            label, score = self.classifier(tokens, self.featurizer.feature_extractor)
-        else:
-            label, score = "None", 0.0
-        return label, score
-
-    def parse(self, text):
-        tokens = self.tokenizer.tokenize(text)
-        intent, score = self.get_intent(tokens)
-        entities = get_entities(text, tokens, self.extractor, self.featurizer)
-        if self.ent_synonyms:
-            Interpreter.replace_synonyms(entities, self.ent_synonyms)
-
-        return {'text': text, 'intent': intent, 'entities': entities, 'confidence': score}
diff --git a/src/interpreters/mitie_interpreter_utils.py b/src/interpreters/mitie_interpreter_utils.py
deleted file mode 100644
index 13909871652c..000000000000
--- a/src/interpreters/mitie_interpreter_utils.py
+++ /dev/null
@@ -1,24 +0,0 @@
-import re
-
-
-def get_entities(text, tokens, extractor, featurizer):
-    ents = []
-    offset = 0
-    if extractor:
-        entities = extractor.extract_entities(tokens, featurizer.feature_extractor)
-        for e in entities:
-            _range = e[0]
-            _regex = u"\s*".join(re.escape(tokens[i]) for i in _range)
-            expr = re.compile(_regex)
-            m = expr.search(text[offset:])
-            start, end = m.start() + offset, m.end() + offset
-            entity_value = text[start:end]
-            offset += m.end()
-            ents.append({
-                "entity": e[1],
-                "value": entity_value,
-                "start": start,
-                "end": end
-            })
-
-    return ents
diff --git a/src/interpreters/mitie_sklearn_interpreter.py b/src/interpreters/mitie_sklearn_interpreter.py
deleted file mode 100644
index a1bf62005c07..000000000000
--- a/src/interpreters/mitie_sklearn_interpreter.py
+++ /dev/null
@@ -1,76 +0,0 @@
-import os
-
-import cloudpickle
-from mitie import named_entity_extractor
-
-from rasa_nlu import Interpreter
-from rasa_nlu.featurizers.mitie_featurizer import MITIEFeaturizer
-from rasa_nlu.interpreters.mitie_interpreter_utils import get_entities
-from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
-
-
-class MITIESklearnInterpreter(Interpreter):
-    @staticmethod
-    def load(meta, featurizer=None):
-        """
-        :type meta: rasa_nlu.model.Metadata
-        :rtype: MITIESklearnInterpreter
-        """
-        if meta.entity_extractor_path:
-            extractor = named_entity_extractor(meta.entity_extractor_path)
-        else:
-            extractor = None
-
-        if featurizer is None:
-            featurizer = MITIEFeaturizer(meta.feature_extractor_path)
-
-        if meta.intent_classifier_path:
-            with open(meta.intent_classifier_path, 'rb') as f:
-                classifier = cloudpickle.load(f)
-        else:
-            classifier = None
-        if meta.entity_synonyms_path:
-            entity_synonyms = Interpreter.load_synonyms(meta.entity_synonyms_path)
-        else:
-            entity_synonyms = None
-
-        return MITIESklearnInterpreter(
-            classifier,
-            extractor,
-            featurizer,
-            entity_synonyms)
-
-    def __init__(self,
-                 intent_classifier=None,
-                 entity_extractor=None,
-                 featurizer=None,
-                 entity_synonyms=None):
-        self.extractor = entity_extractor
-        self.classifier = intent_classifier
-        self.featurizer = featurizer
-        self.tokenizer = MITIETokenizer()
-        self.ent_synonyms = entity_synonyms
-
-    def get_intent(self, sentence_tokens):
-        """Returns the most likely intent and its probability for the input text.
-
-        :param sentence_tokens: text to classify
-        :return: tuple of most likely intent name and its probability"""
-        if self.classifier:
-            X = self.featurizer.features_for_tokens(sentence_tokens).reshape(1, -1)
-            intent_ids, probabilities = self.classifier.predict(X)
-            intents = self.classifier.transform_labels_num2str(intent_ids)
-            intent, score = intents[0], probabilities[0]
-        else:
-            intent, score = "None", 0.0
-
-        return intent, score
-
-    def parse(self, text):
-        tokens = self.tokenizer.tokenize(text)
-        intent, probability = self.get_intent(tokens)
-        entities = get_entities(text, tokens, self.extractor, self.featurizer)
-        if self.ent_synonyms:
-            Interpreter.replace_synonyms(entities, self.ent_synonyms)
-
-        return {'text': text, 'intent': intent, 'entities': entities, 'confidence': probability}
diff --git a/src/interpreters/simple_interpreter.py b/src/interpreters/simple_interpreter.py
deleted file mode 100644
index 13f8613ab5e0..000000000000
--- a/src/interpreters/simple_interpreter.py
+++ /dev/null
@@ -1,22 +0,0 @@
-from rasa_nlu import Interpreter
-
-
-class HelloGoodbyeInterpreter(Interpreter):
-    def __init__(self):
-        self.name = "hello_goodbye"
-        self.his = ["hello", "hi", "hey"]
-        self.byes = ["bye", "goodbye"]
-
-    def parse(self, text):
-        _text = text.lower()
-
-        def is_present(x): return x in _text
-
-        if any(map(is_present, self.his)):
-            intent = "greet"
-        elif any(map(is_present, self.byes)):
-            intent = "goodbye"
-        else:
-            intent = "None"
-
-        return {'text': text, 'intent': intent, 'entities': [], 'confidence': 1.0}
diff --git a/src/interpreters/spacy_sklearn_interpreter.py b/src/interpreters/spacy_sklearn_interpreter.py
deleted file mode 100644
index d73f2fedd80e..000000000000
--- a/src/interpreters/spacy_sklearn_interpreter.py
+++ /dev/null
@@ -1,89 +0,0 @@
-from rasa_nlu import Interpreter
-import os
-import cloudpickle
-import spacy
-import json
-import codecs
-
-from rasa_nlu import Interpreter
-from rasa_nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
-from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-from rasa_nlu.utils.spacy import ensure_proper_language_model
-
-
-class SpacySklearnInterpreter(Interpreter):
-    @staticmethod
-    def load(meta, nlp, featurizer=None):
-        """
-        :type meta: rasa_nlu.model.Metadata
-        :type nlp: spacy.language.Language
-        :type featurizer: None or rasa_nlu.featurizers.spacy_featurizer.SpacyFeaturizer
-        :rtype: MITIEInterpreter
-        """
-        if meta.entity_extractor_path:
-            extractor = SpacyEntityExtractor(nlp, meta.entity_extractor_path,
-                                             meta.metadata.get("should_fine_tune_spacy_ner"))
-        else:
-            extractor = None
-        if meta.intent_classifier_path:
-            with open(meta.intent_classifier_path, 'rb') as f:
-                classifier = cloudpickle.load(f)
-        else:
-            classifier = None
-        if meta.entity_synonyms_path:
-            entity_synonyms = Interpreter.load_synonyms(meta.entity_synonyms_path)
-        else:
-            entity_synonyms = None
-
-        if featurizer is None:
-            featurizer = SpacyFeaturizer(nlp)
-        return SpacySklearnInterpreter(
-            classifier,
-            extractor,
-            entity_synonyms,
-            featurizer,
-            nlp)
-
-    def __init__(self,
-                 intent_classifier=None,
-                 entity_extractor=None,
-                 entity_synonyms=None,
-                 featurizer=None,
-                 nlp=None):
-        self.extractor = entity_extractor
-        self.classifier = intent_classifier
-        self.ent_synonyms = entity_synonyms
-        self.nlp = nlp
-        self.featurizer = featurizer
-
-        ensure_proper_language_model(nlp)
-
-    def get_intent(self, doc):
-        """Returns the most likely intent and its probability for the input text.
-
-        :param text: text to classify
-        :return: tuple of most likely intent name and its probability"""
-        if self.classifier:
-            X = self.featurizer.features_for_doc(doc).reshape(1, -1)
-            intent_ids, probabilities = self.classifier.predict(X)
-            intents = self.classifier.transform_labels_num2str(intent_ids)
-            intent, score = intents[0], probabilities[0]
-        else:
-            intent, score = "None", 0.0
-
-        return intent, score
-
-    def get_entities(self, doc):
-        if self.extractor:
-            return self.extractor.extract_entities(doc)
-        return []
-
-    def parse(self, text):
-        """Parse the input text, classify it and return an object containing its intent and entities."""
-        doc = self.nlp(text, entity=False)
-        intent, probability = self.get_intent(doc)
-        entities = self.get_entities(doc)
-        if self.ent_synonyms:
-            Interpreter.replace_synonyms(entities, self.ent_synonyms)
-
-        return {'text': text, 'intent': intent, 'entities': entities, 'confidence': probability}
diff --git a/src/model.py b/src/model.py
deleted file mode 100644
index 2e4052798aee..000000000000
--- a/src/model.py
+++ /dev/null
@@ -1,68 +0,0 @@
-import json
-import os
-
-
-class InvalidModelError(Exception):
-    """Raised when a model failed to load.
-
-    Attributes:
-        message -- explanation of why the model is invalid
-    """
-
-    def __init__(self, message):
-        self.message = message
-
-
-class Model(object):
-    def __init__(self, metadata, model_dir, interpreter=None, nlp=None, featurizer=None):
-        self.metadata = metadata
-        self.model_dir = model_dir
-        self.interpreter = interpreter
-        self.featurizer = featurizer
-        self.nlp = nlp
-
-
-class Metadata(object):
-    @staticmethod
-    def load(model_dir):
-        with open(os.path.join(model_dir, 'metadata.json'), 'rb') as f:
-            data = json.loads(f.read().decode('utf-8'))
-        return Metadata(data, model_dir)
-
-    def __init__(self, metadata, model_dir):
-        self.metadata = metadata
-        self.model_dir = model_dir
-
-    def __prepend_path(self, prop):
-        if self.metadata.get(prop) is not None:
-            return os.path.normpath(os.path.join(self.model_dir, self.metadata[prop]))
-        else:
-            return None
-
-    @property
-    def feature_extractor_path(self):
-        return self.__prepend_path("feature_extractor")
-
-    @property
-    def intent_classifier_path(self):
-        return self.__prepend_path("intent_classifier")
-
-    @property
-    def entity_extractor_path(self):
-        return self.__prepend_path("entity_extractor")
-
-    @property
-    def entity_synonyms_path(self):
-        return self.__prepend_path("entity_synonyms")
-
-    def language_name(self):
-        return self.metadata.get('language_name')
-
-    def backend_name(self):
-        return self.metadata.get('backend')
-
-    def model_group(self):
-        """Groups models by backend and language name."""
-        if self.language_name() is not None and self.backend_name() is not None:
-            return self.language_name() + "@" + self.backend_name()
-        return None
diff --git a/src/tokenizers/__init__.py b/src/tokenizers/__init__.py
deleted file mode 100644
index efdb23f5f3b8..000000000000
--- a/src/tokenizers/__init__.py
+++ /dev/null
@@ -1,3 +0,0 @@
-class Tokenizer(object):
-    def tokenize(self, text):
-        raise NotImplementedError
diff --git a/src/tokenizers/spacy_tokenizer.py b/src/tokenizers/spacy_tokenizer.py
deleted file mode 100644
index 56a8664bdb29..000000000000
--- a/src/tokenizers/spacy_tokenizer.py
+++ /dev/null
@@ -1,10 +0,0 @@
-from rasa_nlu.tokenizers import Tokenizer
-
-
-class SpacyTokenizer(Tokenizer):
-
-    def __init__(self, nlp):
-        self.nlp = nlp
-
-    def tokenize(self, text):
-        return [t.text for t in self.nlp(text)]
diff --git a/src/tokenizers/whitespace_tokenizer.py b/src/tokenizers/whitespace_tokenizer.py
deleted file mode 100644
index aefbd9d47579..000000000000
--- a/src/tokenizers/whitespace_tokenizer.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from rasa_nlu.tokenizers import Tokenizer
-
-
-class WhitespaceTokenizer(Tokenizer):
-    def tokenize(self, text):
-        return text.split()
diff --git a/src/train.py b/src/train.py
deleted file mode 100644
index f2f02c59f240..000000000000
--- a/src/train.py
+++ /dev/null
@@ -1,73 +0,0 @@
-import argparse
-import logging
-import os
-
-from rasa_nlu.config import RasaNLUConfig
-from rasa_nlu.training_data import TrainingData
-
-
-def create_argparser():
-    parser = argparse.ArgumentParser(description='train a custom language parser')
-
-    # TODO add args for training only entity extractor or only intent
-    parser.add_argument('-b', '--backend', default=None, choices=['mitie', 'spacy_sklearn', 'keyword'],
-                        help='backend to use to interpret text (default: built in keyword matcher).')
-    parser.add_argument('-p', '--path', default=None, help="path where model files will be saved")
-    parser.add_argument('-d', '--data', default=None, help="file containing training data")
-    parser.add_argument('-c', '--config', required=True, help="config file")
-    parser.add_argument('-l', '--language', default=None, choices=['de', 'en'], help="model and data language")
-    parser.add_argument('-t', '--num_threads', default=1, type=int,
-                        help="number of threads to use during model training")
-    parser.add_argument('-m', '--mitie_file', default=None,
-                        help='file with mitie total_word_feature_extractor')
-    return parser
-
-
-def create_trainer(config):
-    backend = config.backend.lower()
-    if backend == 'mitie':
-        from trainers.mitie_trainer import MITIETrainer
-        return MITIETrainer(config.mitie_file, config.language, config.num_threads)
-    if backend == 'mitie_sklearn':
-        from trainers.mitie_sklearn_trainer import MITIESklearnTrainer
-        return MITIESklearnTrainer(config.mitie_file, config.language, config.num_threads)
-    if backend == 'spacy_sklearn':
-        from trainers.spacy_sklearn_trainer import SpacySklearnTrainer
-        return SpacySklearnTrainer(config.language, config.num_threads, config.fine_tune_spacy_ner)
-    else:
-        raise NotImplementedError("other backend trainers not implemented yet")
-
-
-def create_persistor(config):
-    persistor = None
-    if "bucket_name" in config:
-        from rasa_nlu.persistor import Persistor
-        persistor = Persistor(config.path, config.aws_region, config.bucket_name)
-
-    return persistor
-
-
-def init():
-    parser = create_argparser()
-    args = parser.parse_args()
-    config = RasaNLUConfig(args.config, os.environ, vars(args))
-    return config
-
-
-def do_train(config):
-    """Loads the trainer and the data and runs the training of the specified model."""
-
-    trainer = create_trainer(config)
-
-    persistor = create_persistor(config)
-
-    training_data = TrainingData(config.data, config.backend, nlp=trainer.nlp)
-    trainer.train(training_data)
-    persited_path = trainer.persist(config.path, persistor)
-    return trainer, persited_path
-
-
-if __name__ == '__main__':
-    config = init()
-    do_train(config)
-    logging.info("done")
diff --git a/src/trainers/__init__.py b/src/trainers/__init__.py
deleted file mode 100644
index e69de29bb2d1..000000000000
diff --git a/src/trainers/mitie_sklearn_trainer.py b/src/trainers/mitie_sklearn_trainer.py
deleted file mode 100644
index 7cfad4829eca..000000000000
--- a/src/trainers/mitie_sklearn_trainer.py
+++ /dev/null
@@ -1,77 +0,0 @@
-import cloudpickle
-import datetime
-import json
-import os
-
-from rasa_nlu.featurizers.mitie_featurizer import MITIEFeaturizer
-from rasa_nlu.trainers.trainer import Trainer
-from training_utils import write_training_metadata
-from rasa_nlu.trainers import mitie_trainer_utils
-from rasa_nlu.trainers import sklearn_trainer_utils
-from rasa_nlu.utils.mitie import MITIE_SKLEARN_BACKEND_NAME
-
-
-class MITIESklearnTrainer(Trainer):
-    SUPPORTED_LANGUAGES = {"en"}
-
-    def __init__(self, fe_file, language_name, max_num_threads=1):
-        super(self.__class__, self).__init__(language_name, max_num_threads)
-        self.fe_file = fe_file
-        self.featurizer = MITIEFeaturizer(self.fe_file)
-
-    def start_and_end(self, text_tokens, entity_tokens):
-        size = len(entity_tokens)
-        max_loc = 1 + len(text_tokens) - size
-        locs = [i for i in range(max_loc) if text_tokens[i:i + size] == entity_tokens]
-        start, end = locs[0], locs[0] + len(entity_tokens)
-        return start, end
-
-    def train_entity_extractor(self, entity_examples):
-        self.entity_extractor = mitie_trainer_utils.train_entity_extractor(entity_examples,
-                                                                           self.fe_file,
-                                                                           self.max_num_threads,)
-
-    def train_intent_classifier(self, intent_examples, test_split_size=0.1):
-        self.intent_classifier = sklearn_trainer_utils.train_intent_classifier(intent_examples,
-                                                                               self.featurizer,
-                                                                               self.max_num_threads,
-                                                                               test_split_size)
-
-    def persist(self, path, persistor=None, create_unique_subfolder=True):
-        timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
-
-        if create_unique_subfolder:
-            dir_name = os.path.join(path, "model_" + timestamp)
-            os.makedirs(dir_name)
-        else:
-            dir_name = path
-
-        data_file = os.path.join(dir_name, "training_data.json")
-        classifier_file, entity_extractor_file = None, None
-        if self.intent_classifier:
-            classifier_file = os.path.join(dir_name, "intent_classifier.pkl")
-        if self.entity_extractor:
-            entity_extractor_file = os.path.join(dir_name, "entity_extractor.dat")
-        entity_synonyms_file = os.path.join(dir_name, "index.json") if self.training_data.entity_synonyms else None
-
-        write_training_metadata(dir_name, timestamp, data_file, MITIE_SKLEARN_BACKEND_NAME, 'en',
-                                classifier_file, entity_extractor_file, entity_synonyms_file,
-                                self.fe_file)
-
-        with open(data_file, 'w') as f:
-            f.write(self.training_data.as_json(indent=2))
-
-        if self.training_data.entity_synonyms:
-            with open(entity_synonyms_file, 'w') as f:
-                json.dump(self.training_data.entity_synonyms, f)
-
-        if self.intent_classifier:
-            with open(classifier_file, 'wb') as f:
-                cloudpickle.dump(self.intent_classifier, f)
-
-        if self.entity_extractor:
-            self.entity_extractor.save_to_disk(entity_extractor_file, pure_model=True)
-
-        if persistor is not None:
-            persistor.send_tar_to_s3(dir_name)
-        return dir_name
diff --git a/src/trainers/mitie_trainer.py b/src/trainers/mitie_trainer.py
deleted file mode 100644
index d9cd3fa885e3..000000000000
--- a/src/trainers/mitie_trainer.py
+++ /dev/null
@@ -1,65 +0,0 @@
-import datetime
-import json
-
-from mitie import *
-
-from rasa_nlu.trainers import mitie_trainer_utils
-from rasa_nlu.trainers.trainer import Trainer
-from rasa_nlu.utils.mitie import MITIE_BACKEND_NAME
-from training_utils import write_training_metadata
-
-
-class MITIETrainer(Trainer):
-    SUPPORTED_LANGUAGES = {"en"}
-
-    def __init__(self, fe_file, language_name, max_num_threads=1):
-        super(self.__class__, self).__init__(language_name, max_num_threads)
-        self.fe_file = fe_file
-
-    def train_entity_extractor(self, entity_examples):
-        self.entity_extractor = mitie_trainer_utils.train_entity_extractor(entity_examples,
-                                                                           self.fe_file,
-                                                                           self.max_num_threads)
-
-    def train_intent_classifier(self, intent_examples, test_split_size=0.1):
-        trainer = text_categorizer_trainer(self.fe_file)
-        trainer.num_threads = self.max_num_threads
-        for example in intent_examples:
-            tokens = tokenize(example["text"])
-            trainer.add_labeled_text(tokens, example["intent"])
-        self.intent_classifier = trainer.train()
-
-    def persist(self, path, persistor=None, create_unique_subfolder=True):
-        timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
-
-        if create_unique_subfolder:
-            dir_name = os.path.join(path, "model_" + timestamp)
-            os.makedirs(dir_name)
-        else:
-            dir_name = path
-
-        data_file = os.path.join(dir_name, "training_data.json")
-        entity_synonyms_file = os.path.join(dir_name, "index.json") if self.training_data.entity_synonyms else None
-        classifier_file, entity_extractor_file = None, None
-        if self.intent_classifier:
-            classifier_file = os.path.join(dir_name, "intent_classifier.dat")
-        if self.entity_extractor:
-            entity_extractor_file = os.path.join(dir_name, "entity_extractor.dat")
-
-        write_training_metadata(dir_name, timestamp, data_file, MITIE_BACKEND_NAME, 'en',
-                                classifier_file, entity_extractor_file, entity_synonyms_file, self.fe_file)
-
-        with open(data_file, 'w') as f:
-            f.write(self.training_data.as_json(indent=2))
-
-        self.intent_classifier.save_to_disk(classifier_file, pure_model=True)
-
-        if self.entity_extractor:
-            self.entity_extractor.save_to_disk(entity_extractor_file, pure_model=True)
-        if self.training_data.entity_synonyms:
-            with open(entity_synonyms_file, 'w') as f:
-                json.dump(self.training_data.entity_synonyms, f)
-
-        if persistor is not None:
-            persistor.send_tar_to_s3(dir_name)
-        return dir_name
diff --git a/src/trainers/mitie_trainer_utils.py b/src/trainers/mitie_trainer_utils.py
deleted file mode 100644
index 39f0896a82d7..000000000000
--- a/src/trainers/mitie_trainer_utils.py
+++ /dev/null
@@ -1,32 +0,0 @@
-from mitie import tokenize, ner_trainer, ner_training_instance
-
-from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
-
-
-def find_entity(ent, text):
-    tk = MITIETokenizer()
-    tokens, offsets = tk.tokenize_with_offsets(text)
-    if ent["start"] not in offsets:
-        message = u"invalid entity {0} in example '{1}':".format(ent, text) + \
-                  u" entities must span whole tokens"
-        raise ValueError(message)
-    start = offsets.index(ent["start"])
-    _slice = text[ent["start"]:ent["end"]]
-    val_tokens = tokenize(_slice)
-    end = start + len(val_tokens)
-    return start, end
-
-
-def train_entity_extractor(entity_examples, fe_file, max_num_threads):
-    trainer = ner_trainer(fe_file)
-    trainer.num_threads = max_num_threads
-    for example in entity_examples:
-        text = example["text"]
-        tokens = tokenize(text)
-        sample = ner_training_instance(tokens)
-        for ent in example["entities"]:
-            start, end = find_entity(ent, text)
-            sample.add_entity(xrange(start, end), ent["entity"])
-
-        trainer.add(sample)
-    return trainer.train()
diff --git a/src/trainers/sklearn_trainer_utils.py b/src/trainers/sklearn_trainer_utils.py
deleted file mode 100644
index f57cff3525e3..000000000000
--- a/src/trainers/sklearn_trainer_utils.py
+++ /dev/null
@@ -1,27 +0,0 @@
-from rasa_nlu.classifiers.sklearn_intent_classifier import SklearnIntentClassifier
-
-
-def train_intent_classifier(intent_examples, featurizer, max_num_threads, test_split_size=0.1):
-    """Trains an intent classifier using sklearn.
-
-    :param intent_examples: examples to train on
-    :param featurizer: featurizer
-    :param max_num_threads: number of threads to use
-    :param test_split_size: how many examples to retain for testing
-    :type intent_examples: list
-    :type featurizer: Featurizer
-    :return: trained classifier
-    :rtype: SklearnIntentClassifier
-    """
-
-    intent_classifier = SklearnIntentClassifier(max_num_threads=max_num_threads)
-    labels = [e["intent"] for e in intent_examples]
-    sentences = [e["text"] for e in intent_examples]
-
-    if len(set(labels)) < 2:
-        raise Exception("Can not train an intent classifier. Need at least 2 different classes.")
-    y = intent_classifier.transform_labels_str2num(labels)
-    X = featurizer.features_for_sentences(sentences)
-    intent_classifier.train(X, y, test_split_size)
-
-    return intent_classifier
diff --git a/src/trainers/spacy_sklearn_trainer.py b/src/trainers/spacy_sklearn_trainer.py
deleted file mode 100644
index 2c0faef0915d..000000000000
--- a/src/trainers/spacy_sklearn_trainer.py
+++ /dev/null
@@ -1,82 +0,0 @@
-import spacy
-import os
-import datetime
-import json
-import cloudpickle
-from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
-from rasa_nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
-from rasa_nlu.trainers import sklearn_trainer_utils
-from rasa_nlu.trainers.trainer import Trainer
-from training_utils import write_training_metadata
-from rasa_nlu.utils.spacy import ensure_proper_language_model, SPACY_BACKEND_NAME
-
-
-class SpacySklearnTrainer(Trainer):
-    SUPPORTED_LANGUAGES = {"en", "de"}
-
-    def __init__(self, language_name, max_num_threads=1, should_fine_tune_spacy_ner=False):
-        super(self.__class__, self).__init__(language_name, max_num_threads)
-        self.should_fine_tune_spacy_ner = should_fine_tune_spacy_ner
-        self.nlp = self._load_nlp_model(language_name, should_fine_tune_spacy_ner)
-        self.featurizer = SpacyFeaturizer(self.nlp)
-        ensure_proper_language_model(self.nlp)
-
-    def train_entity_extractor(self, entity_examples):
-        self.entity_extractor = SpacyEntityExtractor(self.nlp)
-        self.entity_extractor.train(self.nlp, entity_examples, self.should_fine_tune_spacy_ner)
-
-    def _load_nlp_model(self, language_name, should_fine_tune_spacy_ner):
-        # If fine tuning is disabled, we do not need to load the spacy entity model
-        if should_fine_tune_spacy_ner:
-            return spacy.load(language_name, parser=False)
-        else:
-            return spacy.load(language_name, parser=False, entity=False)
-
-    def train_intent_classifier(self, intent_examples, test_split_size=0.1):
-        self.intent_classifier = sklearn_trainer_utils.train_intent_classifier(intent_examples,
-                                                                               self.featurizer,
-                                                                               self.max_num_threads,
-                                                                               test_split_size)
-
-    def persist(self, path, persistor=None, create_unique_subfolder=True):
-        entity_extractor_file, entity_extractor_config_file = None, None
-        timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
-
-        if create_unique_subfolder:
-            dir_name = os.path.join(path, "model_" + timestamp)
-            os.makedirs(dir_name)
-        else:
-            dir_name = path
-
-        data_file = os.path.join(dir_name, "training_data.json")
-        entity_synonyms_file = os.path.join(dir_name, "index.json") if self.training_data.entity_synonyms else None
-        classifier_file, ner_dir, entity_extractor_config_file, entity_extractor_file = None, None, None, None
-        if self.intent_classifier:
-            classifier_file = os.path.join(dir_name, "intent_classifier.pkl")
-        if self.entity_extractor:
-            ner_dir = os.path.join(dir_name, 'ner')
-            if not os.path.exists(ner_dir):
-                os.mkdir(ner_dir)
-            entity_extractor_config_file = os.path.join(ner_dir, "config.json")
-            entity_extractor_file = os.path.join(ner_dir, "model")
-
-        write_training_metadata(dir_name, timestamp, data_file, SPACY_BACKEND_NAME, self.language_name,
-                                classifier_file, ner_dir, entity_synonyms_file,
-                                should_fine_tune_spacy_ner=self.should_fine_tune_spacy_ner)
-
-        with open(data_file, 'w') as f:
-            f.write(self.training_data.as_json(indent=2))
-        if self.intent_classifier:
-            with open(classifier_file, 'wb') as f:
-                cloudpickle.dump(self.intent_classifier, f)
-        if self.entity_extractor:
-            with open(entity_extractor_config_file, 'w') as f:
-                json.dump(self.entity_extractor.ner.cfg, f)
-            self.entity_extractor.ner.model.dump(entity_extractor_file)
-        if self.training_data.entity_synonyms:
-            with open(entity_synonyms_file, 'w') as f:
-                json.dump(self.training_data.entity_synonyms, f)
-
-        if persistor is not None:
-            persistor.send_tar_to_s3(dir_name)
-        return dir_name
diff --git a/src/trainers/trainer.py b/src/trainers/trainer.py
deleted file mode 100644
index ed98a9ab9117..000000000000
--- a/src/trainers/trainer.py
+++ /dev/null
@@ -1,33 +0,0 @@
-import logging
-
-
-class Trainer(object):
-    SUPPORTED_LANGUAGES = None
-
-    def __init__(self, language_name, max_num_threads=1):
-        self.intent_classifier = None
-        self.entity_extractor = None
-        self.language_name = language_name
-        self.max_num_threads = max_num_threads
-        self.training_data = None
-        self.nlp = None
-        self.ensure_language_support(language_name)
-
-    def ensure_language_support(self, language_name):
-        if language_name not in self.SUPPORTED_LANGUAGES:
-            supported = "', '".join(self.SUPPORTED_LANGUAGES)
-            logging.warn("Selected backend currently does not officially support language " +
-                         "'{}' (only '{}'). Things might break!".format(language_name, supported))
-
-    def train_entity_extractor(self, entity_examples):
-        raise NotImplementedError()
-
-    def train_intent_classifier(self, intent_examples, test_split_size=0.1):
-        raise NotImplementedError()
-
-    def train(self, data, test_split_size=0.1):
-        self.training_data = data
-        self.train_intent_classifier(data.intent_examples, test_split_size)
-
-        if self.training_data.num_entity_examples > 0:
-            self.train_entity_extractor(data.entity_examples)
diff --git a/src/trainers/training_utils.py b/src/trainers/training_utils.py
deleted file mode 100644
index 13b17b5b5ef9..000000000000
--- a/src/trainers/training_utils.py
+++ /dev/null
@@ -1,31 +0,0 @@
-import os
-import json
-import rasa_nlu
-
-
-def __relative_normpath(file, path):
-    if file is not None:
-        return os.path.normpath(os.path.relpath(file, path))
-    else:
-        return None
-
-
-def write_training_metadata(output_folder, timestamp, data_file, backend_name,
-                            language_name, intent_file, entity_file,
-                            entity_synonyms_file=None, feature_file=None, should_fine_tune_spacy_ner=False):
-
-    metadata = {
-        "trained_at": timestamp,
-        "training_data": os.path.basename(data_file),
-        "backend": backend_name,
-        "intent_classifier": __relative_normpath(intent_file, output_folder),
-        "entity_extractor": __relative_normpath(entity_file, output_folder),
-        "entity_synonyms": __relative_normpath(entity_synonyms_file, output_folder),
-        "feature_extractor": __relative_normpath(feature_file, output_folder),
-        "language_name": language_name,
-        "rasa_nlu_version": rasa_nlu.__version__,
-        "should_fine_tune_spacy_ner": should_fine_tune_spacy_ner,
-    }
-
-    with open(os.path.join(output_folder, 'metadata.json'), 'w') as f:
-        f.write(json.dumps(metadata, indent=4))
diff --git a/src/training_data.py b/src/training_data.py
deleted file mode 100644
index c222a635333e..000000000000
--- a/src/training_data.py
+++ /dev/null
@@ -1,225 +0,0 @@
-# -*- coding: utf-8 -*-
-
-import codecs
-import json
-import re
-import warnings
-from itertools import groupby
-
-from rasa_nlu.utils import mitie
-
-from rasa_nlu.utils import spacy
-
-from rasa_nlu import util
-
-
-# Different supported file formats and their identifier
-WIT_FILE_FORMAT = "wit"
-API_FILE_FORMAT = "api"
-LUIS_FILE_FORMAT = "luis"
-RASA_FILE_FORMAT = "rasa_nlu"
-UNK_FILE_FORMAT = "unk"
-
-
-class TrainingData(object):
-    # Validation will ensure and warn if these lower limits are not met
-    MIN_EXAMPLES_PER_INTENT = 2
-    MIN_EXAMPLES_PER_ENTITY = 2
-
-    def __init__(self, resource_name, backend, nlp=None, file_format=None):
-        self.intent_examples = []
-        self.entity_examples = []
-        self.entity_synonyms = {}
-        self.resource_name = resource_name
-        self.files = TrainingData.resolve_data_files(resource_name)
-        self.fformat = file_format if file_format is not None else TrainingData.guess_format(self.files)
-        self.tokenizer = None
-
-        self.init_tokenizer(backend, nlp)
-        self.load_data()
-        self.validate()
-
-    def init_tokenizer(self, backend, nlp):
-        if backend in [mitie.MITIE_BACKEND_NAME, mitie.MITIE_SKLEARN_BACKEND_NAME]:
-            from rasa_nlu.tokenizers.mitie_tokenizer import MITIETokenizer
-            self.tokenizer = MITIETokenizer()
-        elif backend in [spacy.SPACY_BACKEND_NAME]:
-            from rasa_nlu.tokenizers.spacy_tokenizer import SpacyTokenizer
-            self.tokenizer = SpacyTokenizer(nlp)
-        else:
-            from rasa_nlu.tokenizers.whitespace_tokenizer import WhitespaceTokenizer
-            self.tokenizer = WhitespaceTokenizer()
-            warnings.warn(
-                "backend not recognised by TrainingData : defaulting to tokenizing by splitting on whitespace")
-
-    @property
-    def num_entity_examples(self):
-        return len([e for e in self.entity_examples if len(e["entities"]) > 0])
-
-    @staticmethod
-    def resolve_data_files(resource_name):
-        try:
-            return util.recursively_find_files(resource_name)
-        except ValueError as e:
-            raise ValueError("Invalid training data file / folder specified. " + e.message)
-
-    def as_json(self, **kwargs):
-        return json.dumps({
-            "rasa_nlu_data": {
-                "intent_examples": self.intent_examples,
-                "entity_examples": self.entity_examples
-            }
-        }, **kwargs)
-
-    @staticmethod
-    def guess_format(files):
-        for filename in files:
-            with codecs.open(filename, encoding="utf-8-sig") as f:
-                file_data = json.loads(f.read())
-            if "data" in file_data and type(file_data.get("data")) is list:
-                return WIT_FILE_FORMAT
-            elif "luis_schema_version" in file_data:
-                return LUIS_FILE_FORMAT
-            elif "userSays" in file_data:
-                return API_FILE_FORMAT
-            elif "rasa_nlu_data" in file_data:
-                return RASA_FILE_FORMAT
-
-        return UNK_FILE_FORMAT
-
-    def load_data(self):
-        if self.fformat == LUIS_FILE_FORMAT:
-            self.load_luis_data(self.files[0])
-        elif self.fformat == WIT_FILE_FORMAT:
-            self.load_wit_data(self.files[0])
-        elif self.fformat == API_FILE_FORMAT:
-            self.load_api_data(self.files)
-        elif self.fformat == RASA_FILE_FORMAT:
-            self.load_rasa_data(self.files[0])
-        else:
-            raise ValueError("unknown training file format : {0}".format(self.fformat))
-
-    def load_wit_data(self, filename):
-        with codecs.open(filename, encoding="utf-8-sig") as f:
-            data = json.loads(f.read())
-        for s in data["data"]:
-            entities = s.get("entities")
-            if entities is None:
-                continue
-            text = s.get("text")
-            intents = [e["value"] for e in entities if e["entity"] == 'intent']
-            intent = intents[0] if intents else 'None'
-
-            entities = [e for e in entities if ("start" in e and "end" in e)]
-            for e in entities:
-                e["value"] = e["value"][1:-1]
-                # create synonyms dictionary
-                text_value = text[e["start"]:e["end"]]
-                util.add_entities_if_synonyms(self.entity_synonyms, text_value, e["value"])
-
-            self.intent_examples.append({"text": text, "intent": intent})
-            self.entity_examples.append({"text": text, "intent": intent, "entities": entities})
-
-    def load_luis_data(self, filename):
-        warnings.warn(
-            """LUIS data may not always be correctly imported because entity locations are specified by tokens.
-            If you use a tokenizer which behaves differently from LUIS's your entities might not be correct""")
-        with codecs.open(filename, encoding="utf-8-sig") as f:
-            data = json.loads(f.read())
-        for s in data["utterances"]:
-            text = s.get("text")
-            tokens = [t for t in self.tokenizer.tokenize(text)]
-            intent = s.get("intent")
-            entities = []
-            for e in s.get("entities") or []:
-                i, ii = e["startPos"], e["endPos"] + 1
-                _regex = u"\s*".join([re.escape(s) for s in tokens[i:ii]])
-                expr = re.compile(_regex)
-                m = expr.search(text)
-                start, end = m.start(), m.end()
-                val = text[start:end]
-                entities.append({"entity": e["entity"], "value": val, "start": start, "end": end})
-
-            self.intent_examples.append({"text": text, "intent": intent})
-            self.entity_examples.append({"text": text, "intent": intent, "entities": entities})
-
-    def load_api_data(self, files):
-        for filename in files:
-            with codecs.open(filename, encoding="utf-8-sig") as f:
-                data = json.loads(f.read())
-            # get only intents, skip the rest. The property name is the target class
-            if "userSays" in data:
-                intent = data.get("name")
-                for s in data["userSays"]:
-                    text = "".join(map(lambda chunk: chunk["text"], s.get("data")))
-                    # add entities to each token, if available
-                    entities = []
-                    for e in filter(lambda chunk: "alias" in chunk or "meta" in chunk, s.get("data")):
-                        start = text.find(e["text"])
-                        end = start + len(e["text"])
-                        val = text[start:end]
-                        entities.append(
-                            {
-                                "entity": e["alias"] if "alias" in e else e["meta"],
-                                "value": val,
-                                "start": start,
-                                "end": end
-                            }
-                        )
-
-                    self.intent_examples.append({"text": text, "intent": intent})
-                    self.entity_examples.append({"text": text, "intent": intent, "entities": entities})
-
-            # create synonyms dictionary
-            if "name" in data and "entries" in data:
-                for entry in data["entries"]:
-                    if "value" in entry and "synonyms" in entry:
-                        for synonym in entry["synonyms"]:
-                            util.add_entities_if_synonyms(self.entity_synonyms, synonym, entry["value"])
-
-    def load_rasa_data(self, filename):
-        with codecs.open(filename, encoding="utf-8-sig") as f:
-            data = json.loads(f.read())
-        common = data['rasa_nlu_data'].get("common_examples", list())
-        intent = data['rasa_nlu_data'].get("intent_examples", list())
-        entity = data['rasa_nlu_data'].get("entity_examples", list())
-
-        self.intent_examples = intent + common
-        self.entity_examples = entity + common
-
-        for example in self.entity_examples:
-            for entity in example["entities"]:
-                entity_val = example["text"][entity["start"]:entity["end"]]
-                util.add_entities_if_synonyms(self.entity_synonyms, entity_val, entity.get("value"))
-
-    def sorted_entity_examples(self):
-        return sorted([entity for ex in self.entity_examples for entity in ex["entities"]], key=lambda e: e["entity"])
-
-    def sorted_intent_examples(self):
-        return sorted(self.intent_examples, key=lambda e: e["intent"])
-
-    def validate(self):
-        examples = self.sorted_intent_examples()
-        for intent, group in groupby(examples, lambda e: e["intent"]):
-            size = len(list(group))
-            if size < self.MIN_EXAMPLES_PER_INTENT:
-                template = u"Intent '{0}' has only {1} training examples! minimum is {2}, training may fail."
-                warnings.warn(template.format(intent, size, self.MIN_EXAMPLES_PER_INTENT))
-
-        sorted_entity_examples = self.sorted_entity_examples()
-        for entity, group in groupby(sorted_entity_examples, lambda e: e["entity"]):
-            size = len(list(group))
-            if size < self.MIN_EXAMPLES_PER_ENTITY:
-                template = u"Entity '{0}' has only {1} training examples! minimum is {2}, training may fail."
-                warnings.warn(template.format(entity, size, self.MIN_EXAMPLES_PER_ENTITY))
-
-        for example in self.entity_examples:
-            text = example["text"]
-            text_tokens = self.tokenizer.tokenize(text)
-            for ent in example["entities"]:
-                ent_tokens = self.tokenizer.tokenize(text[ent["start"]:ent["end"]])
-                for token in ent_tokens:
-                    if token not in text_tokens:
-                        warnings.warn(
-                            "Token '{0}' does not appear in tokenized sentence {1}.".format(token, text_tokens) +
-                            "Entities must span whole tokens.")
diff --git a/src/utils/__init__.py b/src/utils/__init__.py
deleted file mode 100644
index e69de29bb2d1..000000000000
diff --git a/src/utils/mitie.py b/src/utils/mitie.py
deleted file mode 100644
index 140722115e93..000000000000
--- a/src/utils/mitie.py
+++ /dev/null
@@ -1,4 +0,0 @@
-MITIE_BACKEND_NAME = "mitie"
-
-
-MITIE_SKLEARN_BACKEND_NAME = "mitie_sklearn"
diff --git a/src/utils/spacy.py b/src/utils/spacy.py
deleted file mode 100644
index 18c7caf68d29..000000000000
--- a/src/utils/spacy.py
+++ /dev/null
@@ -1,12 +0,0 @@
-SPACY_BACKEND_NAME = "spacy_sklearn"
-
-
-def ensure_proper_language_model(nlp):
-    """Checks if the spacy language model is properly loaded. Raises an exception if the model is invalid."""
-
-    if nlp is None:
-        raise Exception("Failed to load spacy language model. Loading the model returned 'None'.")
-    if nlp.path is None:
-        # Spacy sets the path to `None` if it did not load the model from disk. In this case `nlp` is an unusable stub.
-        raise Exception("Failed to load spacy language model for lang '{}'. ".format(nlp.lang) +
-                        "Make sure you have downloaded the correct model (https://spacy.io/docs/usage/).")
diff --git a/src/version.py b/src/version.py
deleted file mode 100644
index 1ef13195a5e4..000000000000
--- a/src/version.py
+++ /dev/null
@@ -1 +0,0 @@
-__version__ = '0.7.3'
diff --git a/test_models/test_model_mitie/intent_classifier.pkl b/test_models/test_model_mitie/intent_classifier.pkl
deleted file mode 100644
index 0a545f7844a5..000000000000
Binary files a/test_models/test_model_mitie/intent_classifier.pkl and /dev/null differ
diff --git a/test_models/test_model_mitie/metadata.json b/test_models/test_model_mitie/metadata.json
index 2b172de079fa..bf8685386300 100644
--- a/test_models/test_model_mitie/metadata.json
+++ b/test_models/test_model_mitie/metadata.json
@@ -1,11 +1,18 @@
 {
-    "trained_at": "20170304-012220", 
-    "intent_classifier": "intent_classifier.dat", 
-    "training_data": "training_data.json", 
-    "language_name": "en", 
     "entity_synonyms": "index.json", 
+    "pipeline": [
+        "init_mitie", 
+        "tokenizer_mitie", 
+        "ner_mitie", 
+        "ner_synonyms", 
+        "intent_featurizer_mitie", 
+        "intent_classifier_mitie"
+    ], 
+    "intent_classifier": "intent_classifier.dat", 
     "entity_extractor": "entity_extractor.dat", 
-    "rasa_nlu_version": "0.7.0a4", 
-    "feature_extractor": "../../data/total_word_feature_extractor.dat", 
-    "backend": "mitie"
+    "language": "en", 
+    "trained_at": "20170316-140543", 
+    "training_data": "training_data.json", 
+    "mitie_feature_extractor_fingerprint": 10023965992282753551, 
+    "rasa_nlu_version": "0.8.0a1"
 }
\ No newline at end of file
diff --git a/test_models/test_model_mitie/ner/config.json b/test_models/test_model_mitie/ner/config.json
deleted file mode 100644
index 7c35f3f35700..000000000000
--- a/test_models/test_model_mitie/ner/config.json
+++ /dev/null
@@ -1 +0,0 @@
-{"entity_types": ["cuisine", "location"], "features": [[121], [157], [133], [169], [145], [157, 121], [121, 133], [127], [128], [165], [129], [141], [165, 129], [129, 165], [165, 129, 141], [153], [177], [170], [158], [122], [134], [146], [158, 122], [122, 134], [170, 158, 122], [158, 122, 134], [122, 134, 146], [171], [159], [123], [135], [147], [159, 123], [123, 135], [181], [183], [182], [181, 121], [183, 121], [182, 121], [182, 158, 122], [183, 159, 123], [180, 159], [182, 158], [183, 159], [182, 194], [183, 158], [193], [195], [194], [181, 193], [181, 194], [182, 193], [182, 193], [166], [166, 167], [120, 166, 167], [129], [141], [153], [165], [177], [127], [128], [166], [178], [166, 178], [166, 167], [178, 179], [120, 166, 167], [120, 132]]}
\ No newline at end of file
diff --git a/test_models/test_model_mitie/ner/model b/test_models/test_model_mitie/ner/model
deleted file mode 100644
index 9503d8c559d8..000000000000
Binary files a/test_models/test_model_mitie/ner/model and /dev/null differ
diff --git a/test_models/test_model_mitie/training_data.json b/test_models/test_model_mitie/training_data.json
index 3f9f2dc50938..bda33dba081f 100644
--- a/test_models/test_model_mitie/training_data.json
+++ b/test_models/test_model_mitie/training_data.json
@@ -1,5 +1,6 @@
 {
   "rasa_nlu_data": {
+    "common_examples": [], 
     "entity_examples": [
       {
         "text": "hey", 
diff --git a/test_models/test_model_mitie_sklearn/intent_classifier.pkl b/test_models/test_model_mitie_sklearn/intent_classifier.pkl
index be1614a63fed..3c8fa594ff76 100644
Binary files a/test_models/test_model_mitie_sklearn/intent_classifier.pkl and b/test_models/test_model_mitie_sklearn/intent_classifier.pkl differ
diff --git a/test_models/test_model_mitie_sklearn/metadata.json b/test_models/test_model_mitie_sklearn/metadata.json
index 051efff75522..56e65934b38c 100644
--- a/test_models/test_model_mitie_sklearn/metadata.json
+++ b/test_models/test_model_mitie_sklearn/metadata.json
@@ -1,11 +1,18 @@
 {
-    "trained_at": "20170304-012243", 
-    "intent_classifier": "intent_classifier.pkl", 
-    "training_data": "training_data.json", 
-    "language_name": "en", 
     "entity_synonyms": "index.json", 
+    "pipeline": [
+        "init_mitie", 
+        "tokenizer_mitie", 
+        "ner_mitie", 
+        "ner_synonyms", 
+        "intent_featurizer_mitie", 
+        "intent_classifier_sklearn"
+    ], 
+    "intent_classifier": "intent_classifier.pkl", 
     "entity_extractor": "entity_extractor.dat", 
-    "rasa_nlu_version": "0.7.0a4", 
-    "feature_extractor": "../../data/total_word_feature_extractor.dat", 
-    "backend": "mitie_sklearn"
+    "language": "en", 
+    "trained_at": "20170316-140607", 
+    "training_data": "training_data.json", 
+    "mitie_feature_extractor_fingerprint": 10023965992282753551, 
+    "rasa_nlu_version": "0.8.0a1"
 }
\ No newline at end of file
diff --git a/test_models/test_model_mitie_sklearn/training_data.json b/test_models/test_model_mitie_sklearn/training_data.json
index 3f9f2dc50938..bda33dba081f 100644
--- a/test_models/test_model_mitie_sklearn/training_data.json
+++ b/test_models/test_model_mitie_sklearn/training_data.json
@@ -1,5 +1,6 @@
 {
   "rasa_nlu_data": {
+    "common_examples": [], 
     "entity_examples": [
       {
         "text": "hey", 
diff --git a/test_models/test_model_spacy_sklearn/intent_classifier.pkl b/test_models/test_model_spacy_sklearn/intent_classifier.pkl
index 0977526d266b..c3eca3ea567f 100644
Binary files a/test_models/test_model_spacy_sklearn/intent_classifier.pkl and b/test_models/test_model_spacy_sklearn/intent_classifier.pkl differ
diff --git a/test_models/test_model_spacy_sklearn/metadata.json b/test_models/test_model_spacy_sklearn/metadata.json
index 490c9e1b566f..3dbccd0999a1 100644
--- a/test_models/test_model_spacy_sklearn/metadata.json
+++ b/test_models/test_model_spacy_sklearn/metadata.json
@@ -1,11 +1,17 @@
 {
-    "trained_at": "20170304-012229", 
-    "intent_classifier": "intent_classifier.pkl", 
-    "training_data": "training_data.json", 
-    "language_name": "en", 
     "entity_synonyms": "index.json", 
+    "pipeline": [
+        "init_spacy", 
+        "ner_spacy", 
+        "ner_synonyms", 
+        "intent_featurizer_spacy", 
+        "intent_classifier_sklearn"
+    ], 
+    "intent_classifier": "intent_classifier.pkl", 
     "entity_extractor": "ner", 
-    "rasa_nlu_version": "0.7.0a4", 
-    "feature_extractor": null, 
-    "backend": "spacy_sklearn"
+    "language": "en", 
+    "trained_at": "20170316-140552", 
+    "training_data": "training_data.json", 
+    "rasa_nlu_version": "0.8.0a1", 
+    "fine_tune_spacy_ner": false
 }
\ No newline at end of file
diff --git a/test_models/test_model_spacy_sklearn/ner/model b/test_models/test_model_spacy_sklearn/ner/model
index f9243efefaf1..25144e05bbe1 100644
Binary files a/test_models/test_model_spacy_sklearn/ner/model and b/test_models/test_model_spacy_sklearn/ner/model differ
diff --git a/test_models/test_model_spacy_sklearn/training_data.json b/test_models/test_model_spacy_sklearn/training_data.json
index 3f9f2dc50938..bda33dba081f 100644
--- a/test_models/test_model_spacy_sklearn/training_data.json
+++ b/test_models/test_model_spacy_sklearn/training_data.json
@@ -1,5 +1,6 @@
 {
   "rasa_nlu_data": {
+    "common_examples": [], 
     "entity_examples": [
       {
         "text": "hey", 


Code review comments:
diff_hunk: @@ -0,0 +1,76 @@
+"""This is a somewhat delicate package. It contains all registered components and preconfigured templates.
+
+Hence, it imports all of the components. To avoid cycles, no component should import this in module scope."""
+from typing import Optional
+from typing import Type
+
+from rasa_nlu.classifiers.keyword_intent_classifier import KeywordIntentClassifier
+from rasa_nlu.classifiers.mitie_intent_classifier import MitieIntentClassifier
+from rasa_nlu.classifiers.sklearn_intent_classifier import SklearnIntentClassifier
+from rasa_nlu.extractors.entity_synonyms import EntitySynonymMapper
+from rasa_nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
+from rasa_nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
+from rasa_nlu.featurizers.mitie_featurizer import MitieFeaturizer
+from rasa_nlu.featurizers.ngram_featurizer import NGramFeaturizer
+from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
+from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa_nlu.utils.mitie_utils import MitieNLP
+from rasa_nlu.utils.spacy_utils import SpacyNLP
+
+# Classes of all known components. If a new component should be added, its class needs to be listed here.
+component_classes = [
+    SpacyNLP, SpacyEntityExtractor, SklearnIntentClassifier, SpacyFeaturizer,
+    MitieNLP, MitieEntityExtractor, MitieIntentClassifier, MitieFeaturizer, MitieTokenizer,
+    KeywordIntentClassifier, EntitySynonymMapper, NGramFeaturizer]
+
+# Mapping from a components name to its class to allow name based lookup.
+registered_components = {component.name: component for component in component_classes}
+
+# To simplify usage, there are a couple of model templates, that already add necessary components in the right order.
+# They also implement the preexisting `backends`.
+registered_model_templates = {
path: rasa_nlu/registry.py
text: @user1:
what's the intended way for users to specify their pipeline?  - providing a list of component names in their config?

@author:
Yes. either that using the `pipeline` attribute and passing in an array of component names or by specifying a `model_template` which is then a preconfigured pipeline based on these templates. we could make that more clear by using one config key, e.g.:
support ` 'pipeline': ['init_spacy', ..., 'intent_classifier_sklearn']` and `'pipeline': 'mitie'` instead of a separate key to define a template.

@author:
I'd keep the pipeline definition using the explicit component names an "advanced" feature. Usually the users should use the templates. Simply because there are some oddities users might not expect (the same way I didn't expect that the `intent_classifier_mitie` doesn't need the featurizer and also won't use the features if you put the `intent_featurizer_spacy` in there ;) )

@user1:
yep cool - I like that. So we can have a 'named' pipeline which is just the name of a template, or if you pass a list then it's interpreted as a custom pipeline. 

@author:
sounds good :)

@author:
what do you think is the best naming, should we use `pipleline`, or avoid that term and go for `model`, other ideas?

@user1:
I think pipeline is fine, can't really think of anything better (maybe arch?)

diff_hunk: @@ -0,0 +1,71 @@
+import argparse
+import logging
+import os
+
+from rasa_nlu.converters import load_data
+from rasa_nlu.model import Trainer
+
+from rasa_nlu.config import RasaNLUConfig
+from typing import Optional
+
+
+def create_argparser():
+    parser = argparse.ArgumentParser(description='train a custom language parser')
+
+    parser.add_argument('-b', '--model_template', default=None, choices=['mitie', 'spacy_sklearn', 'keyword'],
+                        help='model template to use to interpret text (default: built in keyword matcher).')
+    parser.add_argument('-p', '--pipeline', default=None,
+                        help='pipeline to use for the message processing.')
+    parser.add_argument('-o', '--output', default=None, help="path where model files will be saved")
+    parser.add_argument('-d', '--data', default=None, help="file containing training data")
+    parser.add_argument('-c', '--config', required=True, help="config file")
+    parser.add_argument('-l', '--language', default=None, choices=['de', 'en'], help="model and data language")
+    parser.add_argument('-t', '--num_threads', default=1, type=int,
+                        help="number of threads to use during model training")
+    parser.add_argument('-m', '--mitie_file', default=None,
+                        help='file with mitie total_word_feature_extractor')
+    return parser
+
+
+def create_persistor(config):
+    # type: (RasaNLUConfig) -> Optional[Persistor]
+    """Create a remote peristor to store the model if the configuration requests it."""
path: rasa_nlu/train.py
text: @user1:
typo: peristor -> persistor

diff_hunk: @@ -0,0 +1,125 @@
+import inspect
+
+from typing import Optional
+from typing import Type
+
+
+def load_component(component_clz, context, config):
+    # type: (Type[Component], dict, dict) -> Optional[Component]
+    """Calls a components load method to init it based on a previously persisted model."""
+
+    if component_clz is not None:
+        load_args = fill_args(component_clz.load_args(), context, config)
+        return component_clz.load(*load_args)
+    else:
+        return None
+
+
+def init_component(component, context, config):
+    # type: (Component, dict, dict) -> None
+    """Initializes a component using the attributes from the context and configuration."""
+
+    args = fill_args(component.pipeline_init_args(), context, config)
+    updates = component.pipeline_init(*args)
+    if updates:
+        context.update(updates)
+
+
+def fill_args(arguments, context, config):
+    # type: ([str], dict, dict) -> [object]
+    """Given a list of arguments, tries to look up these argument names in the config / context to fill the arguments"""
+
+    filled = []
+    for arg in arguments:
+        if arg in context:
+            filled.append(context[arg])
+        elif arg in config:
+            filled.append(config[arg])
+        else:
+            raise MissingArgumentError("Couldn't fill argument '{}' :(".format(arg))
+    return filled
+
+
+class MissingArgumentError(Exception):
+    """Raised when a function is called and not all parameters can be filled from the context / config.
+
+    Attributes:
+        message -- explanation of which parameter is missing
+    """
+
+    def __init__(self, message):
+        # type: (str) -> None
+        self.message = message
+
+
+class Component(object):
+    # Name of the component to be used when integrating it in a pipeline. E.g. `[ComponentA, ComponentB]`
+    # will be a proper pipeline definition where `ComponentA` is the name of the first component of the pipeline.
+    name = ""
+
+    # Defines what attributes the pipeline component will provide when called. The different keys indicate the
+    # different functions (`pipeline_init`, `train`, `process`) that are able to update the pipelines context.
+    # (mostly used to check if the pipeline is valid)
+    context_provides = {
+        "pipeline_init": [],
+        "train": [],
+        "process": [],
+    }
+
+    # Defines which of the attributes the component provides should be added to the final output json at the end of the
+    # pipeline. Every attribute in `output_provides` should be part of the above `context_provides['process']`. As it
+    # wouldn't make much sense to keep an attribute in the output that is not generated. Every other attribute provided
+    # in the context during the process step will be removed from the output json.
+    output_provides = []
+
+    @classmethod
+    def load(cls, *args):
+        # type: (...) -> 'cls'
+        return cls()
+
+    def pipeline_init(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def train(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def process(self, *args):
+        # type: (...) -> Optional[dict]
+        pass
+
+    def persist(self, model_dir):
+        # type: (str) -> Optional[dict]
+        pass
+
+    @classmethod
+    def cache_key(cls, model_metadata):
+        # type: (Metadata) -> Optional[str]
+        """This key is used to cache components.
+
+        If a model is unique to a model it should return None. Otherwise, an instantiation of the
path: rasa_nlu/components.py
text: @user1:
I guess this should say "if a component is unique to a model"

diff_hunk: @@ -0,0 +1,76 @@
+"""This is a somewhat delicate package. It contains all registered components and preconfigured templates.
+
+Hence, it imports all of the components. To avoid cycles, no component should import this in module scope."""
+from typing import Optional
+from typing import Type
+
+from rasa_nlu.classifiers.keyword_intent_classifier import KeywordIntentClassifier
+from rasa_nlu.classifiers.mitie_intent_classifier import MitieIntentClassifier
+from rasa_nlu.classifiers.sklearn_intent_classifier import SklearnIntentClassifier
+from rasa_nlu.extractors.entity_synonyms import EntitySynonymMapper
+from rasa_nlu.extractors.mitie_entity_extractor import MitieEntityExtractor
+from rasa_nlu.extractors.spacy_entity_extractor import SpacyEntityExtractor
+from rasa_nlu.featurizers.mitie_featurizer import MitieFeaturizer
+from rasa_nlu.featurizers.ngram_featurizer import NGramFeaturizer
+from rasa_nlu.featurizers.spacy_featurizer import SpacyFeaturizer
+from rasa_nlu.tokenizers.mitie_tokenizer import MitieTokenizer
+from rasa_nlu.utils.mitie_utils import MitieNLP
+from rasa_nlu.utils.spacy_utils import SpacyNLP
+
+# Classes of all known components. If a new component should be added, its class needs to be listed here.
+component_classes = [
+    SpacyNLP, SpacyEntityExtractor, SklearnIntentClassifier, SpacyFeaturizer,
+    MitieNLP, MitieEntityExtractor, MitieIntentClassifier, MitieFeaturizer, MitieTokenizer,
+    KeywordIntentClassifier, EntitySynonymMapper, NGramFeaturizer]
+
+# Mapping from a components name to its class to allow name based lookup.
+registered_components = {component.name: component for component in component_classes}
+
+# To simplify usage, there are a couple of model templates, that already add necessary components in the right order.
+# They also implement the preexisting `backends`.
+registered_model_templates = {
+    "spacy_sklearn": [
+        "init_spacy",
+        "ner_spacy",
+        "ner_synonyms",
+        "intent_featurizer_spacy",
+        "intent_classifier_sklearn",
+    ],
+    "mitie": [
+        "init_mitie",
+        "tokenizer_mitie",
+        "ner_mitie",
+        "ner_synonyms",
+        "intent_featurizer_mitie",
path: rasa_nlu/registry.py
text: @user1:
I don't think we need this here, do we? the mitie intent classifier's `train` and `process` methods work with tokens directly, no need to featurize first

diff_hunk: @@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return NGramFeaturizer()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        classifier_file = os.path.join(model_dir, "ngram_featurizer.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "ngram_featurizer": "ngram_featurizer.pkl"
+        }
+
+    def _ngrams_to_use(self, num_ngrams):
+        if num_ngrams == 0:
+            return None
+        elif num_ngrams is not None:
+            return self.all_ngrams[:num_ngrams]
+        else:
+            return self.all_ngrams
+
+    def _get_best_ngrams(self, sentences, labels, spacy_nlp):
+        """Returns an ordered list of the best character ngrams for an intent classification problem"""
+
+        oov_strings = self._remove_in_vocab_words(sentences, spacy_nlp)
+        ngrams = self._generate_all_ngrams(oov_strings)
+        return self._sort_applicable_ngrams(ngrams, sentences, labels, spacy_nlp)
+
+    def _remove_in_vocab_words(self, sentences, spacy_nlp):
+        """Automatically removes words with digits in them, that may be a
+        hyperlink or that _are_ in vocabulary for the nlp"""
+
+        new_sents = []
+        for sentence in sentences:
+            new_sents.append(self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp))
+        return new_sents
+
+    def _remove_hyperlinks(self, sentence):
+        return re.sub(r'^https?:\/\/.*[\r\n]*', '', sentence, flags=re.MULTILINE)
+
+    def _remove_punctuation(self, sentence):
+        return ''.join([letter for letter in sentence if letter not in punctuation])
+
+    def _remove_in_vocab_words_of_sentence(self, sentence, spacy_nlp):
+        """Automatically removes words with digits in them, hyperlink and in-vocab-words."""
+
+        cleaned_sentence = self._remove_hyperlinks(self._remove_punctuation(sentence))
+
+        # keep only out-of-vocab 'non_word' words
+        doc = spacy_nlp(cleaned_sentence)
+        non_words = ' '.join([q.lower_ for q in doc if not q.has_vector])
+
+        # remove digits and extra spaces
+        non_words = ''.join([letter for letter in non_words if not letter.isdigit()])
+        non_words = ' '.join([word for word in non_words.split(' ') if word != ''])
+
+        # add cleaned sentence to list of these sentences
+        return non_words
+
+    def _sort_applicable_ngrams(self, list_of_ngrams, sentences, labels, spacy_nlp):
+        """Given an intent classification problem and a list of ngrams, creates ordered list of most useful ngrams."""
+
+        if list_of_ngrams:
+            from sklearn import linear_model, preprocessing
+            import numpy as np
+
+            usable_labels = []
+            for label in np.unique(labels):
+                lab_sents = np.array(sentences)[np.array(labels) == label]
+                if len(lab_sents) < 10:
path: rasa_nlu/featurizers/ngram_featurizer.py
text: @user1:
let's replace 10 with `min_intent_examples_for_ngram_classification` or something

@user2:
done

diff_hunk: @@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return NGramFeaturizer()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        classifier_file = os.path.join(model_dir, "ngram_featurizer.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "ngram_featurizer": "ngram_featurizer.pkl"
+        }
+
+    def _ngrams_to_use(self, num_ngrams):
+        if num_ngrams == 0:
+            return None
+        elif num_ngrams is not None:
+            return self.all_ngrams[:num_ngrams]
+        else:
+            return self.all_ngrams
+
+    def _get_best_ngrams(self, sentences, labels, spacy_nlp):
+        """Returns an ordered list of the best character ngrams for an intent classification problem"""
+
+        oov_strings = self._remove_in_vocab_words(sentences, spacy_nlp)
+        ngrams = self._generate_all_ngrams(oov_strings)
+        return self._sort_applicable_ngrams(ngrams, sentences, labels, spacy_nlp)
+
+    def _remove_in_vocab_words(self, sentences, spacy_nlp):
+        """Automatically removes words with digits in them, that may be a
+        hyperlink or that _are_ in vocabulary for the nlp"""
+
+        new_sents = []
+        for sentence in sentences:
+            new_sents.append(self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp))
+        return new_sents
+
+    def _remove_hyperlinks(self, sentence):
+        return re.sub(r'^https?:\/\/.*[\r\n]*', '', sentence, flags=re.MULTILINE)
+
+    def _remove_punctuation(self, sentence):
+        return ''.join([letter for letter in sentence if letter not in punctuation])
+
+    def _remove_in_vocab_words_of_sentence(self, sentence, spacy_nlp):
path: rasa_nlu/featurizers/ngram_featurizer.py
text: @user1:
I think `_remove_in_vocab_words_from_sentence` makes more sense

@user2:
agreed

diff_hunk: @@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return NGramFeaturizer()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        classifier_file = os.path.join(model_dir, "ngram_featurizer.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "ngram_featurizer": "ngram_featurizer.pkl"
+        }
+
+    def _ngrams_to_use(self, num_ngrams):
+        if num_ngrams == 0:
+            return None
+        elif num_ngrams is not None:
+            return self.all_ngrams[:num_ngrams]
+        else:
+            return self.all_ngrams
+
+    def _get_best_ngrams(self, sentences, labels, spacy_nlp):
+        """Returns an ordered list of the best character ngrams for an intent classification problem"""
+
+        oov_strings = self._remove_in_vocab_words(sentences, spacy_nlp)
+        ngrams = self._generate_all_ngrams(oov_strings)
+        return self._sort_applicable_ngrams(ngrams, sentences, labels, spacy_nlp)
+
+    def _remove_in_vocab_words(self, sentences, spacy_nlp):
+        """Automatically removes words with digits in them, that may be a
+        hyperlink or that _are_ in vocabulary for the nlp"""
+
+        new_sents = []
+        for sentence in sentences:
+            new_sents.append(self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp))
+        return new_sents
+
+    def _remove_hyperlinks(self, sentence):
+        return re.sub(r'^https?:\/\/.*[\r\n]*', '', sentence, flags=re.MULTILINE)
+
+    def _remove_punctuation(self, sentence):
+        return ''.join([letter for letter in sentence if letter not in punctuation])
+
+    def _remove_in_vocab_words_of_sentence(self, sentence, spacy_nlp):
+        """Automatically removes words with digits in them, hyperlink and in-vocab-words."""
+
+        cleaned_sentence = self._remove_hyperlinks(self._remove_punctuation(sentence))
+
+        # keep only out-of-vocab 'non_word' words
+        doc = spacy_nlp(cleaned_sentence)
+        non_words = ' '.join([q.lower_ for q in doc if not q.has_vector])
+
+        # remove digits and extra spaces
+        non_words = ''.join([letter for letter in non_words if not letter.isdigit()])
+        non_words = ' '.join([word for word in non_words.split(' ') if word != ''])
+
+        # add cleaned sentence to list of these sentences
+        return non_words
+
+    def _sort_applicable_ngrams(self, list_of_ngrams, sentences, labels, spacy_nlp):
+        """Given an intent classification problem and a list of ngrams, creates ordered list of most useful ngrams."""
+
+        if list_of_ngrams:
+            from sklearn import linear_model, preprocessing
+            import numpy as np
+
+            usable_labels = []
+            for label in np.unique(labels):
+                lab_sents = np.array(sentences)[np.array(labels) == label]
+                if len(lab_sents) < 10:
+                    continue
+                usable_labels.append(label)
+
+            mask = [label in usable_labels for label in labels]
+            sentences = np.array(sentences)[mask]
+            labels = np.array(labels)[mask]
+
+            X = np.array(self._ngrams_in_sentences(sentences, spacy_nlp, list_of_ngrams))
+            intent_encoder = preprocessing.LabelEncoder()
+            intent_encoder.fit(labels)
+            y = intent_encoder.transform(labels)
+
+            clf = linear_model.RandomizedLogisticRegression(C=1)
+            clf.fit(X, y)
+            scores = clf.scores_
+            sort_idx = [i[0] for i in sorted(enumerate(scores), key=lambda x: -1 * x[1])]
+
+            return np.array(list_of_ngrams)[sort_idx]
+        else:
+            return []
+
+    def _ngrams_in_sentences(self, sentences, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        all_vectors = []
+        for sentence in sentences:
+            presence_vector = self._ngrams_in_sentence(sentence, spacy_nlp, ngrams)
+            all_vectors.append(presence_vector)
+        return all_vectors
+
+    def _ngrams_in_sentence(self, sentence, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        import numpy as np
+
+        cleaned_sentence = self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp)
+        presence_vector = np.zeros(len(ngrams))
+        idx_array = [idx for idx in xrange(len(ngrams)) if ngrams[idx] in cleaned_sentence]
+        presence_vector[idx_array] = 1
+        return presence_vector
+
+    def _generate_all_ngrams(self, list_of_strings):
+        """Takes a list of strings and generates all character ngrams.
+
+        Generated ngrams are at least 3 characters (and at most 17), occur at least 5 times
+        and occur independently of longer superset ngrams at least once."""
+
+        features = {}
+        counters = {}
+
+        for n in xrange(self.n_gram_min_length, self.n_gram_max_length):
+            candidates = []
+            features[n] = []
+            counters[n] = Counter()
+
+            # generate all possible n length ngrams
+            for text in list_of_strings:
+                text = text.replace(',', ' ')
path: rasa_nlu/featurizers/ngram_featurizer.py
text: @user2:
why only replace commas and not all punctuation? @user1

@user3:
good point, fixed!

diff_hunk: @@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
+            with open(classifier_file, 'rb') as f:
+                return cloudpickle.load(f)
+        else:
+            return NGramFeaturizer()
+
+    def persist(self, model_dir):
+        # type: (str) -> dict
+        """Persist this model into the passed directory. Returns the metadata necessary to load the model again."""
+
+        classifier_file = os.path.join(model_dir, "ngram_featurizer.pkl")
+        with open(classifier_file, 'wb') as f:
+            cloudpickle.dump(self, f)
+
+        return {
+            "ngram_featurizer": "ngram_featurizer.pkl"
+        }
+
+    def _ngrams_to_use(self, num_ngrams):
+        if num_ngrams == 0:
+            return None
+        elif num_ngrams is not None:
+            return self.all_ngrams[:num_ngrams]
+        else:
+            return self.all_ngrams
+
+    def _get_best_ngrams(self, sentences, labels, spacy_nlp):
+        """Returns an ordered list of the best character ngrams for an intent classification problem"""
+
+        oov_strings = self._remove_in_vocab_words(sentences, spacy_nlp)
+        ngrams = self._generate_all_ngrams(oov_strings)
+        return self._sort_applicable_ngrams(ngrams, sentences, labels, spacy_nlp)
+
+    def _remove_in_vocab_words(self, sentences, spacy_nlp):
+        """Automatically removes words with digits in them, that may be a
+        hyperlink or that _are_ in vocabulary for the nlp"""
+
+        new_sents = []
+        for sentence in sentences:
+            new_sents.append(self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp))
+        return new_sents
+
+    def _remove_hyperlinks(self, sentence):
+        return re.sub(r'^https?:\/\/.*[\r\n]*', '', sentence, flags=re.MULTILINE)
+
+    def _remove_punctuation(self, sentence):
+        return ''.join([letter for letter in sentence if letter not in punctuation])
+
+    def _remove_in_vocab_words_of_sentence(self, sentence, spacy_nlp):
+        """Automatically removes words with digits in them, hyperlink and in-vocab-words."""
+
+        cleaned_sentence = self._remove_hyperlinks(self._remove_punctuation(sentence))
+
+        # keep only out-of-vocab 'non_word' words
+        doc = spacy_nlp(cleaned_sentence)
+        non_words = ' '.join([q.lower_ for q in doc if not q.has_vector])
+
+        # remove digits and extra spaces
+        non_words = ''.join([letter for letter in non_words if not letter.isdigit()])
+        non_words = ' '.join([word for word in non_words.split(' ') if word != ''])
+
+        # add cleaned sentence to list of these sentences
+        return non_words
+
+    def _sort_applicable_ngrams(self, list_of_ngrams, sentences, labels, spacy_nlp):
+        """Given an intent classification problem and a list of ngrams, creates ordered list of most useful ngrams."""
+
+        if list_of_ngrams:
+            from sklearn import linear_model, preprocessing
+            import numpy as np
+
+            usable_labels = []
+            for label in np.unique(labels):
+                lab_sents = np.array(sentences)[np.array(labels) == label]
+                if len(lab_sents) < 10:
+                    continue
+                usable_labels.append(label)
+
+            mask = [label in usable_labels for label in labels]
+            sentences = np.array(sentences)[mask]
+            labels = np.array(labels)[mask]
+
+            X = np.array(self._ngrams_in_sentences(sentences, spacy_nlp, list_of_ngrams))
+            intent_encoder = preprocessing.LabelEncoder()
+            intent_encoder.fit(labels)
+            y = intent_encoder.transform(labels)
+
+            clf = linear_model.RandomizedLogisticRegression(C=1)
+            clf.fit(X, y)
+            scores = clf.scores_
+            sort_idx = [i[0] for i in sorted(enumerate(scores), key=lambda x: -1 * x[1])]
+
+            return np.array(list_of_ngrams)[sort_idx]
+        else:
+            return []
+
+    def _ngrams_in_sentences(self, sentences, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        all_vectors = []
+        for sentence in sentences:
+            presence_vector = self._ngrams_in_sentence(sentence, spacy_nlp, ngrams)
+            all_vectors.append(presence_vector)
+        return all_vectors
+
+    def _ngrams_in_sentence(self, sentence, spacy_nlp, ngrams):
+        """Given a set of sentences, returns a vector of {1,0} values indicating ngram presence"""
+
+        import numpy as np
+
+        cleaned_sentence = self._remove_in_vocab_words_of_sentence(sentence, spacy_nlp)
+        presence_vector = np.zeros(len(ngrams))
+        idx_array = [idx for idx in xrange(len(ngrams)) if ngrams[idx] in cleaned_sentence]
+        presence_vector[idx_array] = 1
+        return presence_vector
+
+    def _generate_all_ngrams(self, list_of_strings):
+        """Takes a list of strings and generates all character ngrams.
+
+        Generated ngrams are at least 3 characters (and at most 17), occur at least 5 times
+        and occur independently of longer superset ngrams at least once."""
+
+        features = {}
+        counters = {}
+
+        for n in xrange(self.n_gram_min_length, self.n_gram_max_length):
+            candidates = []
+            features[n] = []
+            counters[n] = Counter()
+
+            # generate all possible n length ngrams
+            for text in list_of_strings:
+                text = text.replace(',', ' ')
+                for word in text.lower().split(' '):
+                    cands = [word[i:i + n] for i in xrange(len(word) - n)]
+                    for cand in cands:
+                        counters[n][cand] += 1
+                        if cand not in candidates:
+                            candidates.append(cand)
+
+            # iterate over these candidates picking only the applicable ones
+            for can in candidates:
+                if counters[n][can] > self.n_gram_min_occurrences:
+                    features[n].append(can)
+                    begin = can[:-1]
+                    end = can[1:]
+                    if n > self.n_gram_min_length:
+                        if counters[n - 1][begin] == counters[n][can] and begin in features[n - 1]:
+                            features[n - 1].remove(begin)
+                        if counters[n - 1][end] == counters[n][can] and end in features[n - 1]:
+                            features[n - 1].remove(end)
+
+        return [item for sublist in features.values() for item in sublist]
+
+    def _create_bow_vecs(self, intent_features, sentences, spacy_nlp, max_ngrams=None):
+        """Given a set of sentences, returns a feature vector for each sentence.
+
+        The first $k$ elements are from the `intent_features`,
+        the rest are {1,0} elements denoting whether an ngram is in sentence."""
+
+        import numpy as np
+
+        ngrams_to_use = self._ngrams_to_use(max_ngrams)
+        if not ngrams_to_use:
+            return intent_features
+        extras = np.array(self._ngrams_in_sentences(sentences, spacy_nlp, ngrams=ngrams_to_use))
+        total = np.hstack((intent_features, extras))
+        return total
+
+    def _cross_validation(self, sentences, labels, intent_features, spacy_nlp, max_ngrams):
+        """choose the best number of ngrams to include in bow.
+
+        Given an intent classification problem and a set of ordered ngrams (ordered in terms
+        of importance by pick_applicable_ngrams) we choose the best number of ngrams to include
+        in our bow vecs by cross validation."""
+
+        from sklearn import preprocessing
+        from sklearn.linear_model import LogisticRegression
+        from sklearn.model_selection import cross_val_score
+        import numpy as np
+
+        clf2 = LogisticRegression()
+        intent_encoder = preprocessing.LabelEncoder()
+        intent_encoder.fit(labels)
+        y = intent_encoder.transform(labels)
+        cv_splits = min(10, np.min(np.bincount(y)))
+        if cv_splits >= 3:
+            logging.debug("Started ngram cross-validation to find best number of ngrams to use...")
+            triers = map(int, np.floor(np.linspace(1, max_ngrams, 8)))
path: rasa_nlu/featurizers/ngram_featurizer.py
text: @user1:
maybe add an np.unique here? because if max_ngrams is < 8 won't we be duplicating computation? 

actually, why not just use `np.arange(1, max_ngrams)` ?

@user1:
also, why `triers` and not `num_ngrams`?

@user2:
Changed triers to num_ngrams and added np.unique. I think that doing CV for each added ngram sequentially would be too expensive, as on the test cases we were of order 100 extra ngrams and that would mean running cross validation a lot of times.

@user1:
ah understood - yes wasn't thinking about the case `max_ngrams >> 8`

diff_hunk: @@ -0,0 +1,271 @@
+import logging
+import os
+import re
+import time
+import warnings
+from collections import Counter
+from string import punctuation
+import cloudpickle
+from typing import Optional
+
+from rasa_nlu.components import Component
+from rasa_nlu.training_data import TrainingData
+
+
+class NGramFeaturizer(Component):
+    name = "intent_featurizer_ngrams"
+
+    context_provides = {
+        "train": ["intent_features"],
+        "process": ["intent_features"],
+    }
+
+    n_gram_min_length = 3
+
+    n_gram_max_length = 17
+
+    n_gram_min_occurrences = 5
+
+    def __init__(self):
+        self.best_num_ngrams = None
+        self.all_ngrams = None
+
+    def train(self, training_data, intent_features, spacy_nlp, max_number_of_ngrams):
+        # type: (TrainingData, [float], Language, Optional[int]) -> dict
+        from spacy.language import Language
+
+        start = time.time()
+        labels = [e['intent'] for e in training_data.intent_examples]
+        sentences = [e['text'] for e in training_data.intent_examples]
+        self.all_ngrams = self._get_best_ngrams(sentences, labels, spacy_nlp)
+        self.best_num_ngrams = self._cross_validation(
+            sentences, labels, intent_features, spacy_nlp, max_number_of_ngrams)
+        logging.debug("Ngram collection took {} seconds".format(time.time() - start))
+        stacked = self._create_bow_vecs(intent_features, sentences, spacy_nlp)
+        return {"intent_features": stacked}
+
+    def process(self, intent_features, text, spacy_nlp):
+        # type: ([float], str, Language) -> dict
+        from spacy.language import Language
+        import numpy as np
+
+        if self.all_ngrams:
+            ngrams_to_use = self._ngrams_to_use(self.best_num_ngrams)
+            if not ngrams_to_use:
+                return {"intent_features": intent_features}
+
+            extras = np.array(self._ngrams_in_sentence(text, spacy_nlp, ngrams_to_use))
+            total = np.hstack((intent_features, extras))
+            return {"intent_features": total}
+        else:
+            return {"intent_features": intent_features}
+
+    @classmethod
+    def load(cls, model_dir, ngram_featurizer):
+        # type: (str, str) -> NGramFeaturizer
+
+        if model_dir and ngram_featurizer:
+            classifier_file = os.path.join(model_dir, ngram_featurizer)
path: rasa_nlu/featurizers/ngram_featurizer.py
text: @user1:
I guess this should either be `filename` or `featurizer_file`

@user2:
done

diff_hunk: @@ -1,7 +1,22 @@
 import os
path: rasa_nlu/util.py
text: @user1:
should we move this code from ./util.py to utils/__init__.py ?