Repository: RasaHQ/rasa
Pull request: only suggest config that is required when doing 'rasa train nlu/core'
Fix #6456 

**Proposed changes**:
- When running `rasa train nlu`, only `pipeline` is suggested and not `policies`.
- When running `rasa train core`, only `policies` is suggested and not `pipeline`.
- As a side effect, for any custom importer there is now also the option to take an argument `nlu_or_core`. This argument will have the value `"core"` when running `rasa train core`, `"nlu"` when running `rasa train nlu` and `"both"` when running `rasa train`. So you can now choose to treat these cases differently in your custom importer.

**TODO**:
- [x] add tests for both `rasa train nlu` and `rasa train core` to make sure this is done correctly
- [x] write something in the docs about the new feature for custom importers

**Status (please check what you already did)**:
- [x] added some tests for the functionality
- [x] updated the documentation
- [ ] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/master/changelog) for instructions)
- [x] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
autoconfig chooses policies when training nlu only 
When training nlu:
```
new ❯ rasa train nlu
The configuration for policies was chosen automatically. It was written into the config file at 'config.yml'.
```
I wouldn't expect it to choose policies for me because I am not using them in my model. My expectations for the opposite would be the same: if only training core, I wouldn't expect it to give me an NLU pipeline.

Hints from issue discussion:

Commit message: updated test code style

Patch to review:
diff --git a/data/test_config/config_empty_en_after_dumping_core.yml b/data/test_config/config_empty_en_after_dumping_core.yml
new file mode 100644
index 000000000000..57a552f7b954
--- /dev/null
+++ b/data/test_config/config_empty_en_after_dumping_core.yml
@@ -0,0 +1,11 @@
+language: en
+pipeline:
+policies:
+# # No configuration for policies was provided. The following default policies were used to train your model.
+# # If you'd like to customize them, uncomment and adjust the policies.
+# # See https://rasa.com/docs/rasa/core/policies/ for more information.
+#   - name: MemoizationPolicy
+#   - name: TEDPolicy
+#     max_history: 5
+#     epochs: 100
+#   - name: RulePolicy
diff --git a/data/test_config/config_empty_en_after_dumping_nlu.yml b/data/test_config/config_empty_en_after_dumping_nlu.yml
new file mode 100644
index 000000000000..0e013c4cbe22
--- /dev/null
+++ b/data/test_config/config_empty_en_after_dumping_nlu.yml
@@ -0,0 +1,20 @@
+language: en
+pipeline:
+# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
+# # If you'd like to customize it, uncomment and adjust the pipeline.
+# # See https://rasa.com/docs/rasa/nlu/choosing-a-pipeline/ for more information.
+#   - name: ConveRTTokenizer
+#   - name: ConveRTFeaturizer
+#   - name: RegexFeaturizer
+#   - name: LexicalSyntacticFeaturizer
+#   - name: CountVectorsFeaturizer
+#   - name: CountVectorsFeaturizer
+#     analyzer: char_wb
+#     min_ngram: 1
+#     max_ngram: 4
+#   - name: DIETClassifier
+#     epochs: 100
+#   - name: EntitySynonymMapper
+#   - name: ResponseSelector
+#     epochs: 100
+policies:
diff --git a/data/test_config/config_empty_en_after_dumping_windows_core.yml b/data/test_config/config_empty_en_after_dumping_windows_core.yml
new file mode 100644
index 000000000000..57a552f7b954
--- /dev/null
+++ b/data/test_config/config_empty_en_after_dumping_windows_core.yml
@@ -0,0 +1,11 @@
+language: en
+pipeline:
+policies:
+# # No configuration for policies was provided. The following default policies were used to train your model.
+# # If you'd like to customize them, uncomment and adjust the policies.
+# # See https://rasa.com/docs/rasa/core/policies/ for more information.
+#   - name: MemoizationPolicy
+#   - name: TEDPolicy
+#     max_history: 5
+#     epochs: 100
+#   - name: RulePolicy
diff --git a/data/test_config/config_empty_en_after_dumping_windows_nlu.yml b/data/test_config/config_empty_en_after_dumping_windows_nlu.yml
new file mode 100644
index 000000000000..c39c5878be86
--- /dev/null
+++ b/data/test_config/config_empty_en_after_dumping_windows_nlu.yml
@@ -0,0 +1,19 @@
+language: en
+pipeline:
+# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
+# # If you'd like to customize it, uncomment and adjust the pipeline.
+# # See https://rasa.com/docs/rasa/nlu/choosing-a-pipeline/ for more information.
+#   - name: WhitespaceTokenizer
+#   - name: RegexFeaturizer
+#   - name: LexicalSyntacticFeaturizer
+#   - name: CountVectorsFeaturizer
+#   - name: CountVectorsFeaturizer
+#     analyzer: char_wb
+#     min_ngram: 1
+#     max_ngram: 4
+#   - name: DIETClassifier
+#     epochs: 100
+#   - name: EntitySynonymMapper
+#   - name: ResponseSelector
+#     epochs: 100
+policies:
diff --git a/docs/docs/model-configuration.mdx b/docs/docs/model-configuration.mdx
index 70cef328cf76..adb797b5f75a 100644
--- a/docs/docs/model-configuration.mdx
+++ b/docs/docs/model-configuration.mdx
@@ -21,6 +21,10 @@ The two other keys (`pipeline` and `policies`) can be left out or left empty. Wh
 running `rasa train`, the Suggested Config feature will select a default configuration
 for the missing key(s) which will then be used to train the model.
 
+Please note, that only the default configuration for `pipeline` will be automatically selected
+if you run `rasa train nlu`, and only the default configuration for `policies`
+will be selected if you run `rasa train core`.
+
 Example `config.yml` file:
 
 ```yaml-rasa (docs/sources/data/configs_for_docs/example_for_suggested_config.yml)
diff --git a/rasa/constants.py b/rasa/constants.py
index ba2365982e6d..1311ec2f9ca7 100644
--- a/rasa/constants.py
+++ b/rasa/constants.py
@@ -61,7 +61,11 @@
 CONFIG_MANDATORY_KEYS_CORE = []
 CONFIG_MANDATORY_KEYS_NLU = ["language"]
 CONFIG_MANDATORY_KEYS = CONFIG_MANDATORY_KEYS_CORE + CONFIG_MANDATORY_KEYS_NLU
-CONFIG_AUTOCONFIGURABLE_KEYS = ["policies", "pipeline"]
+CONFIG_AUTOCONFIGURABLE_KEYS_CORE = ["policies"]
+CONFIG_AUTOCONFIGURABLE_KEYS_NLU = ["pipeline"]
+CONFIG_AUTOCONFIGURABLE_KEYS = (
+    CONFIG_AUTOCONFIGURABLE_KEYS_CORE + CONFIG_AUTOCONFIGURABLE_KEYS_NLU
+)
 
 
 MINIMUM_COMPATIBLE_VERSION = "1.11.0a3"
diff --git a/rasa/importers/autoconfig.py b/rasa/importers/autoconfig.py
index 7033c14fc35c..8e6dda0de723 100644
--- a/rasa/importers/autoconfig.py
+++ b/rasa/importers/autoconfig.py
@@ -2,8 +2,8 @@
 import logging
 import os
 import sys
-
-from typing import Text, Dict, Any, List, Set
+from enum import Enum
+from typing import Text, Dict, Any, List, Set, Optional
 
 from rasa.cli import utils as cli_utils
 from rasa.constants import (
@@ -11,6 +11,10 @@
     DOCS_URL_PIPELINE,
     DOCS_URL_POLICIES,
     CONFIG_KEYS,
+    CONFIG_KEYS_CORE,
+    CONFIG_KEYS_NLU,
+    CONFIG_AUTOCONFIGURABLE_KEYS_CORE,
+    CONFIG_AUTOCONFIGURABLE_KEYS_NLU,
 )
 from rasa.utils import io as io_utils, common as common_utils
 
@@ -32,7 +36,15 @@
 }
 
 
-def get_configuration(config_file_path: Text) -> Dict[Text, Any]:
+class TrainingType(Enum):
+    NLU = 1
+    CORE = 2
+    BOTH = 3
+
+
+def get_configuration(
+    config_file_path: Text, training_type: Optional[TrainingType] = TrainingType.BOTH
+) -> Dict[Text, Any]:
     """Determine configuration from a configuration file.
 
     Keys that are provided and have a value in the file are kept. Keys that are not
@@ -40,6 +52,7 @@ def get_configuration(config_file_path: Text) -> Dict[Text, Any]:
 
     Args:
         config_file_path: The path to the configuration file.
+        training_type: NLU, CORE or BOTH depending on what is trained.
     """
     if not config_file_path or not os.path.exists(config_file_path):
         logger.debug("No configuration file was provided to the TrainingDataImporter.")
@@ -47,22 +60,42 @@ def get_configuration(config_file_path: Text) -> Dict[Text, Any]:
 
     config = io_utils.read_config_file(config_file_path)
 
-    missing_keys = _get_missing_config_keys(config)
-    keys_to_configure = _get_unspecified_autoconfigurable_keys(config)
+    missing_keys = _get_missing_config_keys(config, training_type)
+    keys_to_configure = _get_unspecified_autoconfigurable_keys(config, training_type)
 
     if keys_to_configure:
         config = _auto_configure(config, keys_to_configure)
-        _dump_config(config, config_file_path, missing_keys, keys_to_configure)
+        _dump_config(
+            config, config_file_path, missing_keys, keys_to_configure, training_type
+        )
 
     return config
 
 
-def _get_unspecified_autoconfigurable_keys(config: Dict[Text, Any]) -> Set[Text]:
-    return {k for k in CONFIG_AUTOCONFIGURABLE_KEYS if not config.get(k)}
+def _get_unspecified_autoconfigurable_keys(
+    config: Dict[Text, Any], training_type: Optional[TrainingType] = TrainingType.BOTH
+) -> Set[Text]:
+    if training_type == TrainingType.NLU:
+        all_keys = CONFIG_AUTOCONFIGURABLE_KEYS_NLU
+    elif training_type == TrainingType.CORE:
+        all_keys = CONFIG_AUTOCONFIGURABLE_KEYS_CORE
+    else:
+        all_keys = CONFIG_AUTOCONFIGURABLE_KEYS
+
+    return {k for k in all_keys if not config.get(k)}
+
 
+def _get_missing_config_keys(
+    config: Dict[Text, Any], training_type: Optional[TrainingType] = TrainingType.BOTH
+) -> Set[Text]:
+    if training_type == TrainingType.NLU:
+        all_keys = CONFIG_KEYS_NLU
+    elif training_type == TrainingType.CORE:
+        all_keys = CONFIG_KEYS_CORE
+    else:
+        all_keys = CONFIG_KEYS
 
-def _get_missing_config_keys(config: Dict[Text, Any]) -> Set[Text]:
-    return {k for k in CONFIG_KEYS if k not in config.keys()}
+    return {k for k in all_keys if k not in config.keys()}
 
 
 def _auto_configure(
@@ -109,6 +142,7 @@ def _dump_config(
     config_file_path: Text,
     missing_keys: Set[Text],
     auto_configured_keys: Set[Text],
+    training_type: Optional[TrainingType] = TrainingType.BOTH,
 ) -> None:
     """Dump the automatically configured keys into the config file.
 
@@ -125,10 +159,11 @@ def _dump_config(
         missing_keys: Keys that need to be added to the config file.
         auto_configured_keys: Keys for which a commented out auto configuration section
                               needs to be added to the config file.
+        training_type: NLU, CORE or BOTH depending on which is trained.
     """
 
     config_as_expected = _is_config_file_as_expected(
-        config_file_path, missing_keys, auto_configured_keys
+        config_file_path, missing_keys, auto_configured_keys, training_type,
     )
     if not config_as_expected:
         cli_utils.print_error(
@@ -161,7 +196,10 @@ def _dump_config(
 
 
 def _is_config_file_as_expected(
-    config_file_path: Text, missing_keys: Set[Text], auto_configured_keys: Set[Text]
+    config_file_path: Text,
+    missing_keys: Set[Text],
+    auto_configured_keys: Set[Text],
+    training_type: Optional[TrainingType] = TrainingType.BOTH,
 ) -> bool:
     try:
         content = io_utils.read_config_file(config_file_path)
@@ -170,8 +208,9 @@ def _is_config_file_as_expected(
 
     return (
         bool(content)
-        and missing_keys == _get_missing_config_keys(content)
-        and auto_configured_keys == _get_unspecified_autoconfigurable_keys(content)
+        and missing_keys == _get_missing_config_keys(content, training_type)
+        and auto_configured_keys
+        == _get_unspecified_autoconfigurable_keys(content, training_type)
     )
 
 
diff --git a/rasa/importers/importer.py b/rasa/importers/importer.py
index 83d7fd3be45b..1b121c014180 100644
--- a/rasa/importers/importer.py
+++ b/rasa/importers/importer.py
@@ -6,6 +6,7 @@
 from rasa.core.domain import Domain
 from rasa.core.training.structures import StoryGraph
 from rasa.nlu.training_data import TrainingData
+from rasa.importers.autoconfig import TrainingType
 import rasa.utils.io as io_utils
 import rasa.utils.common as common_utils
 
@@ -69,12 +70,13 @@ def load_from_config(
         config_path: Text,
         domain_path: Optional[Text] = None,
         training_data_paths: Optional[List[Text]] = None,
+        training_type: Optional[TrainingType] = TrainingType.BOTH,
     ) -> "TrainingDataImporter":
         """Loads a `TrainingDataImporter` instance from a configuration file."""
 
         config = io_utils.read_config_file(config_path)
         return TrainingDataImporter.load_from_dict(
-            config, config_path, domain_path, training_data_paths
+            config, config_path, domain_path, training_data_paths, training_type
         )
 
     @staticmethod
@@ -89,7 +91,7 @@ def load_core_importer_from_config(
         """
 
         importer = TrainingDataImporter.load_from_config(
-            config_path, domain_path, training_data_paths
+            config_path, domain_path, training_data_paths, TrainingType.CORE,
         )
 
         return CoreDataImporter(importer)
@@ -106,7 +108,7 @@ def load_nlu_importer_from_config(
         """
 
         importer = TrainingDataImporter.load_from_config(
-            config_path, domain_path, training_data_paths
+            config_path, domain_path, training_data_paths, TrainingType.NLU
         )
 
         return NluDataImporter(importer)
@@ -117,6 +119,7 @@ def load_from_dict(
         config_path: Text,
         domain_path: Optional[Text] = None,
         training_data_paths: Optional[List[Text]] = None,
+        training_type: Optional[TrainingType] = TrainingType.BOTH,
     ) -> "TrainingDataImporter":
         """Loads a `TrainingDataImporter` instance from a dictionary."""
 
@@ -126,7 +129,7 @@ def load_from_dict(
         importers = config.get("importers", [])
         importers = [
             TrainingDataImporter._importer_from_dict(
-                importer, config_path, domain_path, training_data_paths
+                importer, config_path, domain_path, training_data_paths, training_type
             )
             for importer in importers
         ]
@@ -134,7 +137,9 @@ def load_from_dict(
 
         if not importers:
             importers = [
-                RasaFileImporter(config_path, domain_path, training_data_paths)
+                RasaFileImporter(
+                    config_path, domain_path, training_data_paths, training_type
+                )
             ]
 
         return CombinedDataImporter(importers)
@@ -145,6 +150,7 @@ def _importer_from_dict(
         config_path: Text,
         domain_path: Optional[Text] = None,
         training_data_paths: Optional[List[Text]] = None,
+        training_type: Optional[TrainingType] = TrainingType.BOTH,
     ) -> Optional["TrainingDataImporter"]:
         from rasa.importers.multi_project import MultiProjectImporter
         from rasa.importers.rasa import RasaFileImporter
@@ -161,9 +167,12 @@ def _importer_from_dict(
                 logging.warning(f"Importer '{module_path}' not found.")
                 return None
 
+        importer_config["training_type"] = training_type
+
         constructor_arguments = common_utils.minimal_kwargs(
             importer_config, importer_class
         )
+
         return importer_class(
             config_path, domain_path, training_data_paths, **constructor_arguments
         )
diff --git a/rasa/importers/rasa.py b/rasa/importers/rasa.py
index 17a524c09e75..5747f42f8a60 100644
--- a/rasa/importers/rasa.py
+++ b/rasa/importers/rasa.py
@@ -6,6 +6,7 @@
 from rasa.core.training.structures import StoryGraph
 from rasa.importers import utils, autoconfig
 from rasa.importers.importer import TrainingDataImporter
+from rasa.importers.autoconfig import TrainingType
 from rasa.nlu.training_data import TrainingData
 from rasa.utils.common import raise_warning
 
@@ -20,6 +21,7 @@ def __init__(
         config_file: Optional[Text] = None,
         domain_path: Optional[Text] = None,
         training_data_paths: Optional[Union[List[Text], Text]] = None,
+        training_type: Optional[TrainingType] = TrainingType.BOTH,
     ):
 
         self._domain_path = domain_path
@@ -27,7 +29,7 @@ def __init__(
         self._nlu_files = data.get_data_files(training_data_paths, data.is_nlu_file)
         self._story_files = data.get_data_files(training_data_paths, data.is_story_file)
 
-        self.config = autoconfig.get_configuration(config_file)
+        self.config = autoconfig.get_configuration(config_file, training_type)
 
     async def get_config(self) -> Dict:
         return self.config
diff --git a/tests/examples/test_example_bots_training_data.py b/tests/examples/test_example_bots_training_data.py
index 4ba07428ab2d..bcf569f558c1 100644
--- a/tests/examples/test_example_bots_training_data.py
+++ b/tests/examples/test_example_bots_training_data.py
@@ -42,7 +42,7 @@ async def test_example_bot_training_data_not_raises(
 ):
 
     importer = TrainingDataImporter.load_from_config(
-        config_file, domain_file, data_folder
+        config_file, domain_file, [data_folder]
     )
 
     with pytest.warns(None) as record:
diff --git a/tests/importers/test_autoconfig.py b/tests/importers/test_autoconfig.py
index cfc7310b1838..ab399b581ed0 100644
--- a/tests/importers/test_autoconfig.py
+++ b/tests/importers/test_autoconfig.py
@@ -180,3 +180,48 @@ def test_dump_config(
             assert k in captured.out
         else:
             assert k not in captured.out
+
+
+@pytest.mark.parametrize(
+    "input_file, expected_file, expected_file_windows, training_type",
+    [
+        (
+            "config_empty_en.yml",
+            "config_empty_en_after_dumping.yml",
+            "config_empty_en_after_dumping_windows.yml",
+            autoconfig.TrainingType.BOTH,
+        ),
+        (
+            "config_empty_en.yml",
+            "config_empty_en_after_dumping_core.yml",
+            "config_empty_en_after_dumping_windows_core.yml",
+            autoconfig.TrainingType.CORE,
+        ),
+        (
+            "config_empty_en.yml",
+            "config_empty_en_after_dumping_nlu.yml",
+            "config_empty_en_after_dumping_windows_nlu.yml",
+            autoconfig.TrainingType.NLU,
+        ),
+    ],
+)
+def test_get_configuration_for_different_training_types(
+    tmp_path: Path,
+    input_file: Text,
+    expected_file: Text,
+    expected_file_windows: Text,
+    training_type: autoconfig.TrainingType,
+):
+    config_file = str(tmp_path / "config.yml")
+    shutil.copyfile(str(CONFIG_FOLDER / input_file), config_file)
+
+    autoconfig.get_configuration(config_file, training_type)
+
+    actual = io_utils.read_file(config_file)
+
+    if sys.platform == "win32":
+        expected = io_utils.read_file(str(CONFIG_FOLDER / expected_file_windows))
+    else:
+        expected = io_utils.read_file(str(CONFIG_FOLDER / expected_file))
+
+    assert actual == expected
diff --git a/tests/test_train.py b/tests/test_train.py
index bd772e40c0c4..5c1420325ae8 100644
--- a/tests/test_train.py
+++ b/tests/test_train.py
@@ -12,6 +12,7 @@
 
 import rasa.model
 import rasa.core
+import rasa.importers.autoconfig as autoconfig
 from rasa.core.interpreter import RasaNLUInterpreter
 
 from rasa.train import train_core, train_nlu, train
@@ -258,3 +259,69 @@ def test_interpreter_from_previous_model_returns_none_for_none():
     from rasa.train import _interpreter_from_previous_model
 
     assert _interpreter_from_previous_model(None) is None
+
+
+def test_train_core_autoconfig(
+    tmp_path: Text,
+    monkeypatch: MonkeyPatch,
+    default_domain_path: Text,
+    default_stories_file: Text,
+    default_stack_config: Text,
+):
+    monkeypatch.setattr(tempfile, "tempdir", tmp_path)
+
+    # mock function that returns configuration
+    mocked_get_configuration = Mock()
+    monkeypatch.setattr(autoconfig, "get_configuration", mocked_get_configuration)
+
+    # skip actual core training
+    _train_core_with_validated_data = Mock()
+    monkeypatch.setattr(
+        sys.modules["rasa.train"],
+        "_train_core_with_validated_data",
+        asyncio.coroutine(_train_core_with_validated_data),
+    )
+
+    # do training
+    train_core(
+        default_domain_path,
+        default_stack_config,
+        default_stories_file,
+        output="test_train_core_temp_files_models",
+    )
+
+    mocked_get_configuration.assert_called_once()
+    _, args, _ = mocked_get_configuration.mock_calls[0]
+    assert args[1] == autoconfig.TrainingType.CORE
+
+
+def test_train_nlu_autoconfig(
+    tmp_path: Text,
+    monkeypatch: MonkeyPatch,
+    default_stack_config: Text,
+    default_nlu_data: Text,
+):
+    monkeypatch.setattr(tempfile, "tempdir", tmp_path)
+
+    # mock function that returns configuration
+    mocked_get_configuration = Mock()
+    monkeypatch.setattr(autoconfig, "get_configuration", mocked_get_configuration)
+
+    # skip actual NLU training
+    _train_nlu_with_validated_data = Mock()
+    monkeypatch.setattr(
+        sys.modules["rasa.train"],
+        "_train_nlu_with_validated_data",
+        asyncio.coroutine(_train_nlu_with_validated_data),
+    )
+
+    # do training
+    train_nlu(
+        default_stack_config,
+        default_nlu_data,
+        output="test_train_nlu_temp_files_models",
+    )
+
+    mocked_get_configuration.assert_called_once()
+    _, args, _ = mocked_get_configuration.mock_calls[0]
+    assert args[1] == autoconfig.TrainingType.NLU


Code review comments:
diff_hunk: @@ -161,9 +167,12 @@ def _importer_from_dict(
                 logging.warning(f"Importer '{module_path}' not found.")
                 return None
 
+        importer_config["training_type"] = training_type
path: rasa/importers/importer.py
text: @user1:
this line updates the `importer_config` dictionary, could it lead to unpredictable consequences, and would something like `importer_config = dict(training_type=training_type, **importer_config)` be more suitable? (or in 2 lines, copy() + update)