Repository: kedro-org/kedro

Issue:
Implement pipeline autodiscovery
Following https://github.com/kedro-org/kedro/issues/1284 and a discussion with Ivan, here's what we're going to do:

### 1. Define a function `find_pipelines`

In `kedro.framework.project.__init__.py`, something like this:
```
def find_pipelines():
    registered_pipelines = {}
    pipeline in Path(f"{PACKAGE_NAME}.pipelines").iterdir():
        importlib.import_module(pipeline)
        if hasattr(pipeline, "create_pipeline"):
            registered_pipelines[pipeline] = pipeline.create_pipeline()
    registered_pipelines["__default__"] = pipeline([])
    return registered_pipelines
```

Name is TBD (note I changed from previously-suggested `autoregister` because it doesn't actually register any more, just finds them). It's also like setup.py's `find_packages` here.

### 2. Call this function from project template and all starters

In pipeline_registry.py:
```
from kedro.framework.project import find_pipelines

def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    MAKE THIS DOCSTRING MORE INFORMATIVE

    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.

    """
    pipelines = find_pipelines()
    pipelines["__default__"] = sum(pipelines.values())
    return pipelines
```

### 3. Add documentation, tests, etc.


Discussion:
@imdoroshenko we should sync up on this and make sure our assumptions still hold 
Thought about this in general quite a bit over the past couple days. Overall, quite aligned. I like the fact that, by just automating discovery, and even so in an optional way, it does one thing well without forcing any workflow.

> Name is TBD (note I changed from previously-suggested `autoregister` because it doesn't actually register any more, just finds them). It's also like setup.py's `find_packages` here.

I like the name. :)

> ```python
>     importlib.import_module(pipeline)
>         if hasattr(pipeline, "create_pipeline"):
>             registered_pipelines[pipeline] = pipeline.create_pipeline()
> ```

Just confirming--this maintains the behavior since 0.17.3 wherein the registry is lazily loaded, right? I assume, since it's in `register_pipelines`, and it's that function whose execution is delayed, this should be fine?

> ```python
> pipelines["__default__"] = sum(pipelines.values())
> ```

Nit: Need a case for no pipelines being defined; guess just wrap this in an `if pipelines` block.
@deepyaman thanks for the comments! I'm much happier about this solution too, and with the definition of `__default__` being project-side it automatically handles the case that we talked about.

Good question about the lazy loading behaviour. This should indeed be maintained since no code in `find_pipelines` will be executed until `register_pipelines` is executed.

And very good point about the case of no pipelines being defined. What do you think of defining `pipelines["__default__"] = pipeline([])` inside `find_pipelines`? This way we keep the "real" definition of `__default__` in `pipeline_registry.py` but no need for an `if` there.

Either that or I would just explicitly do:
```
pipelines["__default__"] = sum(pipelines.values()) if pipelines else pipeline([])
```
> What do you think of defining `pipelines["__default__"] = pipeline([])` inside `find_pipelines`? This way we keep the "real" definition of `__default__` in `pipeline_registry.py` but no need for an `if` there.

Clever; I like it!

Resolving patch:
diff --git a/features/steps/cli_steps.py b/features/steps/cli_steps.py
--- a/features/steps/cli_steps.py
+++ b/features/steps/cli_steps.py
@@ -428,34 +428,6 @@ def is_created(name):
         assert is_created(path)
 
 
-@then("the pipeline should contain no nodes")
-def check_empty_pipeline_exists(context):
-    """Check if the created pipeline in
-    `pipeline_registry.py` contains no nodes.
-    """
-    pipeline_file = (
-        context.root_project_dir
-        / "src"
-        / context.project_name.replace("-", "_")
-        / "pipeline_registry.py"
-    )
-    assert '"__default__": pipeline([])' in pipeline_file.read_text("utf-8")
-
-
-@then("the pipeline should contain nodes")
-def check_pipeline_not_empty(context):
-    """Check if the created pipeline in
-    `pipeline_registry.py` contains nodes.
-    """
-    pipeline_file = (
-        context.root_project_dir
-        / "src"
-        / context.project_name.replace("-", "_")
-        / "pipeline_registry.py"
-    )
-    assert "pipeline = pipeline([])" not in pipeline_file.read_text("utf-8")
-
-
 @then("the logs should show that {number} nodes were run")
 def check_one_node_run(context, number):
     expected_log_line = f"Completed {number} out of {number} tasks"
diff --git a/kedro/framework/project/__init__.py b/kedro/framework/project/__init__.py
--- a/kedro/framework/project/__init__.py
+++ b/kedro/framework/project/__init__.py
@@ -5,19 +5,22 @@
 import logging.config
 import operator
 import sys
+import traceback
+import warnings
 from collections import UserDict
 from collections.abc import MutableMapping
 from pathlib import Path
 from typing import Any, Dict, Optional
 
 import click
+import importlib_resources
 import rich.pretty
 import rich.traceback
 import yaml
 from dynaconf import LazySettings
 from dynaconf.validator import ValidationError, Validator
 
-from kedro.pipeline import Pipeline
+from kedro.pipeline import Pipeline, pipeline
 
 
 def _get_default_class(class_import_path):
@@ -150,7 +153,7 @@ def _get_pipelines_registry_callable(pipelines_module: str):
         return register_pipelines
 
     def _load_data(self):
-        """Lazily read pipelines defined in the pipelines registry module"""
+        """Lazily read pipelines defined in the pipelines registry module."""
 
         # If the pipelines dictionary has not been configured with a pipelines module
         # or if data has been loaded
@@ -167,7 +170,7 @@ def _load_data(self):
 
     def configure(self, pipelines_module: Optional[str] = None) -> None:
         """Configure the pipelines_module to load the pipelines dictionary.
-        Reset the data loading state so that after every `configure` call,
+        Reset the data loading state so that after every ``configure`` call,
         data are reloaded.
         """
         self._pipelines_module = pipelines_module
@@ -204,7 +207,7 @@ def __init__(self):
         rich.pretty.install()
 
     def configure(self, logging_config: Dict[str, Any]) -> None:
-        """Configure project logging using `logging_config` (e.g. from project
+        """Configure project logging using ``logging_config`` (e.g. from project
         logging.yml). We store this in the UserDict data so that it can be reconfigured
         in _bootstrap_subprocess.
         """
@@ -239,7 +242,7 @@ def configure_project(package_name: str):
 
 
 def configure_logging(logging_config: Dict[str, Any]) -> None:
-    """Configure logging according to `logging_config` dictionary."""
+    """Configure logging according to ``logging_config`` dictionary."""
     LOGGING.configure(logging_config)
 
 
@@ -259,3 +262,63 @@ def validate_settings():
         )
 
     importlib.import_module(f"{PACKAGE_NAME}.settings")
+
+
+def find_pipelines() -> Dict[str, Pipeline]:
+    """Automatically find modular pipelines having a ``create_pipeline``
+    function. By default, projects created using Kedro 0.18.3 and higher
+    call this function to autoregister pipelines upon creation/addition.
+
+    Returns:
+        A generated mapping from pipeline names to ``Pipeline`` objects.
+
+    Warns:
+        UserWarning: When a module does not expose a ``create_pipeline``
+            function, the ``create_pipeline`` function does not return a
+            ``Pipeline`` object, or if the module import fails up front.
+    """
+    pipelines_dict = {"__default__": pipeline([])}
+    for pipeline_dir in importlib_resources.files(
+        f"{PACKAGE_NAME}.pipelines"
+    ).iterdir():
+        if not pipeline_dir.is_dir():
+            continue
+
+        pipeline_name = pipeline_dir.name
+        if pipeline_name == "__pycache__":
+            continue
+
+        try:
+            pipeline_module = importlib.import_module(
+                f"{PACKAGE_NAME}.pipelines.{pipeline_name}"
+            )
+        except:  # pylint: disable=bare-except  # noqa: E722
+            warnings.warn(
+                f"An error occurred while importing the "
+                f"'{PACKAGE_NAME}.pipelines.{pipeline_name}' module. "
+                f"Nothing defined therein will be returned by "
+                f"'find_pipelines'.\n\n{traceback.format_exc()}"
+            )
+            continue
+
+        if not hasattr(pipeline_module, "create_pipeline"):
+            warnings.warn(
+                f"The '{pipeline_module.__name__}' module does not "
+                f"expose a 'create_pipeline' function, so no pipelines "
+                f"defined therein will be returned by 'find_pipelines'."
+            )
+            continue
+
+        obj = getattr(pipeline_module, "create_pipeline")()
+        if not isinstance(obj, Pipeline):
+            warnings.warn(
+                f"Expected the 'create_pipeline' function in the "
+                f"'{pipeline_module.__name__}' module to return a "
+                f"'Pipeline' object, got '{type(obj).__name__}' "
+                f"instead. Nothing defined therein will be returned by "
+                f"'find_pipelines'."
+            )
+            continue
+
+        pipelines_dict[pipeline_name] = obj
+    return pipelines_dict
diff --git a/kedro/templates/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py b/kedro/templates/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
--- a/kedro/templates/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py	
+++ b/kedro/templates/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py	
@@ -1,13 +1,20 @@
 """Project pipelines."""
 from typing import Dict
 
+from kedro.framework.project import find_pipelines
 from kedro.pipeline import Pipeline, pipeline
 
 
 def register_pipelines() -> Dict[str, Pipeline]:
     """Register the project's pipelines.
 
+    Since Kedro 0.18.3, projects can use the ``find_pipelines`` function
+    to autodiscover pipelines. However, projects that require more fine-
+    grained control can still construct the pipeline mapping without it.
+
     Returns:
-        A mapping from a pipeline name to a ``Pipeline`` object.
+        A mapping from pipeline names to ``Pipeline`` objects.
     """
-    return {"__default__": pipeline([])}
+    pipelines = find_pipelines()
+    pipelines["__default__"] = sum(pipelines.values())
+    return pipelines


Test patch:
diff --git a/features/steps/e2e_test_cli_config.yml b/features/steps/e2e_test_cli_config.yml
--- a/features/steps/e2e_test_cli_config.yml
+++ b/features/steps/e2e_test_cli_config.yml
@@ -1,3 +1,3 @@
 run:
-  pipeline: de
+  pipeline: data_engineering
   env: base
diff --git a/features/steps/test_starter/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py b/features/steps/test_starter/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py
--- a/features/steps/test_starter/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py	
+++ b/features/steps/test_starter/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipeline_registry.py	
@@ -1,27 +1,23 @@
 """Project pipelines."""
 from typing import Dict
 
+from kedro.framework.project import find_pipelines
 from kedro.pipeline import Pipeline, pipeline
 
-from {{cookiecutter.python_package}}.pipelines import data_engineering as de
-from {{cookiecutter.python_package}}.pipelines import data_science as ds
-
 
 def register_pipelines() -> Dict[str, Pipeline]:
     """Register the project's pipelines.
 
+    Since Kedro 0.18.3, projects can use the ``find_pipelines`` function
+    to autodiscover pipelines. However, projects that require more fine-
+    grained control can still construct the pipeline mapping without it.
+
     Returns:
-        A mapping from a pipeline name to a ``Pipeline`` object.
+        A mapping from pipeline names to ``Pipeline`` objects.
     """
-    data_engineering_pipeline = de.create_pipeline()
-    data_processing_pipeline = pipeline(
-        de.create_pipeline(), namespace="data_processing"
+    pipelines = find_pipelines()
+    pipelines["__default__"] = sum(pipelines.values())
+    pipelines["data_processing"] = pipeline(
+        pipelines["data_engineering"], namespace="data_processing"
     )
-    data_science_pipeline = ds.create_pipeline()
-
-    return {
-        "de": data_engineering_pipeline,
-        "ds": data_science_pipeline,
-        "dp": data_processing_pipeline,
-        "__default__": data_engineering_pipeline + data_science_pipeline,
-    }
+    return pipelines
diff --git a/tests/framework/cli/test_catalog.py b/tests/framework/cli/test_catalog.py
--- a/tests/framework/cli/test_catalog.py
+++ b/tests/framework/cli/test_catalog.py
@@ -153,7 +153,7 @@ def identity(data):
 
 @pytest.mark.usefixtures("chdir_to_dummy_project")
 class TestCatalogCreateCommand:
-    PIPELINE_NAME = "de"
+    PIPELINE_NAME = "data_engineering"
 
     @staticmethod
     @pytest.fixture(params=["base"])
diff --git a/tests/framework/cli/test_registry.py b/tests/framework/cli/test_registry.py
--- a/tests/framework/cli/test_registry.py
+++ b/tests/framework/cli/test_registry.py
@@ -10,15 +10,15 @@ def yaml_dump_mock(mocker):
 @pytest.fixture
 def pipelines_dict():
     pipelines = {
-        "de": ["split_data (split_data)"],
-        "ds": [
+        "data_engineering": ["split_data (split_data)"],
+        "data_science": [
             "train_model (train_model)",
             "predict (predict)",
             "report_accuracy (report_accuracy)",
         ],
-        "dp": ["data_processing.split_data (split_data)"],
+        "data_processing": ["data_processing.split_data (split_data)"],
     }
-    pipelines["__default__"] = pipelines["de"] + pipelines["ds"]
+    pipelines["__default__"] = pipelines["data_engineering"] + pipelines["data_science"]
     return pipelines
 
 
@@ -36,7 +36,10 @@ def test_list_registered_pipelines(
 
 @pytest.mark.usefixtures("chdir_to_dummy_project")
 class TestRegistryDescribeCommand:
-    @pytest.mark.parametrize("pipeline_name", ["de", "ds", "dp", "__default__"])
+    @pytest.mark.parametrize(
+        "pipeline_name",
+        ["data_engineering", "data_science", "data_processing", "__default__"],
+    )
     def test_describe_registered_pipeline(
         self,
         fake_project_cli,
@@ -63,7 +66,7 @@ def test_registered_pipeline_not_found(self, fake_project_cli, fake_metadata):
         assert result.exit_code
         expected_output = (
             "Error: 'missing' pipeline not found. Existing pipelines: "
-            "[__default__, de, dp, ds]\n"
+            "[__default__, data_engineering, data_processing, data_science]\n"
         )
         assert expected_output in result.output
 
diff --git a/tests/framework/project/test_logging.py b/tests/framework/project/test_logging.py
--- a/tests/framework/project/test_logging.py
+++ b/tests/framework/project/test_logging.py
@@ -2,7 +2,7 @@
 
 import pytest
 
-from kedro.framework.project import LOGGING, configure_logging  # noqa
+from kedro.framework.project import LOGGING, configure_logging
 
 default_logging_config = {
     "version": 1,
diff --git a/tests/framework/project/test_pipeline_discovery.py b/tests/framework/project/test_pipeline_discovery.py
new file mode 100644
--- /dev/null
+++ b/tests/framework/project/test_pipeline_discovery.py
@@ -0,0 +1,170 @@
+import sys
+import textwrap
+import warnings
+from pathlib import Path
+
+import pytest
+
+from kedro.framework.project import configure_project, find_pipelines
+
+
+@pytest.fixture
+def mock_package_name_with_pipelines(tmp_path, request):
+    package_name = "test_package"
+    pipelines_dir = tmp_path / package_name / "pipelines"
+    pipelines_dir.mkdir(parents=True)
+    (pipelines_dir / "__init__.py").touch()
+    for pipeline_name in request.param:
+        pipeline_dir = pipelines_dir / pipeline_name
+        pipeline_dir.mkdir()
+        (pipeline_dir / "__init__.py").write_text(
+            textwrap.dedent(
+                f"""
+                from kedro.pipeline import Pipeline, node, pipeline
+
+
+                def create_pipeline(**kwargs) -> Pipeline:
+                    return pipeline([node(lambda: 1, None, "{pipeline_name}")])
+                """
+            )
+        )
+    sys.path.insert(0, str(tmp_path))
+    yield package_name
+    sys.path.pop(0)
+
+    # Make sure that the `importlib_resources.files` in `find_pipelines`
+    # will point to the correct `test_package.pipelines` not from cache.
+    del sys.modules[f"{package_name}.pipelines"]
+
+
+@pytest.fixture
+def pipeline_names(request):
+    return request.param
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines(
+    mock_package_name_with_pipelines,
+    pipeline_names,
+):
+    configure_project(mock_package_name_with_pipelines)
+    pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"good_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_skips_modules_without_create_pipelines_function(
+    mock_package_name_with_pipelines,
+    pipeline_names,
+):
+    # Create a module without `create_pipelines` in the `pipelines` dir.
+    pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
+    pipeline_dir = pipelines_dir / "bad_touch"
+    pipeline_dir.mkdir()
+    (pipeline_dir / "__init__.py").touch()
+
+    configure_project(mock_package_name_with_pipelines)
+    with pytest.warns(
+        UserWarning, match="module does not expose a 'create_pipeline' function"
+    ):
+        pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_skips_modules_with_unexpected_return_value_type(
+    mock_package_name_with_pipelines,
+    pipeline_names,
+):
+    # Define `create_pipelines` so that it does not return a `Pipeline`.
+    pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
+    pipeline_dir = pipelines_dir / "not_my_pipeline"
+    pipeline_dir.mkdir()
+    (pipeline_dir / "__init__.py").write_text(
+        textwrap.dedent(
+            """
+            from typing import Dict
+
+            from kedro.pipeline import Pipeline, node, pipeline
+
+
+            def create_pipeline(**kwargs) -> Dict[str, Pipeline]:
+                return {
+                    "pipe1": pipeline([node(lambda: 1, None, "pipe1")]),
+                    "pipe2": pipeline([node(lambda: 2, None, "pipe2")]),
+                }
+            """
+        )
+    )
+
+    configure_project(mock_package_name_with_pipelines)
+    with pytest.warns(
+        UserWarning,
+        match=(
+            r"Expected the 'create_pipeline' function in the '\S+' "
+            r"module to return a 'Pipeline' object, got 'dict' instead."
+        ),
+    ):
+        pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_skips_regular_files_within_the_pipelines_folder(
+    mock_package_name_with_pipelines,
+    pipeline_names,
+):
+    # Create a regular file (not a subdirectory) in the `pipelines` dir.
+    pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
+    (pipelines_dir / "not_my_pipeline.py").touch()
+
+    configure_project(mock_package_name_with_pipelines)
+    with warnings.catch_warnings():
+        warnings.filterwarnings("error", category=UserWarning)
+        pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_skips_modules_that_cause_exceptions_upon_import(
+    mock_package_name_with_pipelines,
+    pipeline_names,
+):
+    # Create a module that will result in errors when we try to load it.
+    pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
+    pipeline_dir = pipelines_dir / "boulevard_of_broken_pipelines"
+    pipeline_dir.mkdir()
+    (pipeline_dir / "__init__.py").write_text("I walk a lonely road...")
+
+    configure_project(mock_package_name_with_pipelines)
+    with pytest.warns(
+        UserWarning,
+        match=r"An error occurred while importing the '\S+' module.",
+    ):
+        pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
diff --git a/tests/framework/project/test_pipeline_registry.py b/tests/framework/project/test_pipeline_registry.py
--- a/tests/framework/project/test_pipeline_registry.py
+++ b/tests/framework/project/test_pipeline_registry.py
@@ -25,7 +25,15 @@ def register_pipelines():
     sys.path.pop(0)
 
 
-def test_pipelines_without_configure_project_is_empty():
+def test_pipelines_without_configure_project_is_empty(
+    mock_package_name_with_pipelines_file,  # pylint: disable=unused-argument
+):
+    # Reimport `pipelines` from `kedro.framework.project` to ensure that
+    # it was not set by a pior call to the `configure_project` function.
+    del sys.modules["kedro.framework.project"]
+    # pylint: disable=reimported, import-outside-toplevel
+    from kedro.framework.project import pipelines
+
     assert pipelines == {}