Repository: kedro-org/kedro

Issue:
Consider whether `find_pipelines` should handle a simplified project structure
Following a conversation in https://github.com/kedro-org/kedro/pull/1706. Just creating the issue while I remember. I'll write more details when I get the chance.

1. Should we support the "simplified" project structure? I think probably yes. If no, we should change the starters that use this.
2. If yes, should `find_pipelines` support this simplified project structure? I think probably yes. 
3. Should `find_pipelines` support both the simplified and extended project structure in a single project? Probably not very important either way.


Discussion:
Dropping my view here:
 * I think it should be default for all starters, so users don't have to discover the `find_pipelines` feature, including `kedro new` with the default project template.
 * I think it should support the simplified case, simply because a unified workflow would be much nicer.

I could be biased because I almost always use `kedro new -s pandas-iris` and delete the unnecessary bit, because `spaceflights` has too much things to delete and the empty `template` has too little thing, I have to find all the right `import` which I end up just searching the `pandas-iris` template anyway.
> 3\. Should `find_pipelines` support both the simplified and extended project structure in a single project? Probably not very important either way.

If there is a `src/<package_name>/pipeline.py` file that exposes `create_pipeline()`, it can't be a named pipeline, so I was going to initialize `__default__` to that pipeline, rather than an empty pipeline (i.e. essentially in https://github.com/deepyaman/kedro/blob/1c1883ac1c595929171a8b2af666b7fe7d514a16/kedro/framework/project/__init__.py#L280).

Then, if somebody creates new modular pipelines with `create_pipeline()` methods, the login in `register_pipelines()` would add them on top of the `pipeline.py` default. How does that sound?
Yeah, I like it 👍 The important thing is that `pipeline_registry.py` file should look exactly the same in both the simplified project and the multi-pipeline project cases. So long as that holds, I'm not too worried about whether/how we support the "`pipeline.py` + `pipelines` folder in one project" case. But this way definitely makes sense and I think is in keeping with what @noklam suggested on the PR.

One thing we should probably do (not necessarily now) is describe somewhere what these two acceptable project structures are. The simplified view is what we use in https://kedro.readthedocs.io/en/stable/get_started/example_project.html (since it's based on `pandas-iris`).
> > 3\. Should `find_pipelines` support both the simplified and extended project structure in a single project? Probably not very important either way.
> 
> If there is a `src/<package_name>/pipeline.py` file that exposes `create_pipeline()`, it can't be a named pipeline, so I was going to initialize `__default__` to that pipeline, rather than an empty pipeline (i.e. essentially in https://github.com/deepyaman/kedro/blob/1c1883ac1c595929171a8b2af666b7fe7d514a16/kedro/framework/project/__init__.py#L280).
> 
> Then, if somebody creates new modular pipelines with `create_pipeline()` methods, the login in `register_pipelines()` would add them on top of the `pipeline.py` default. How does that sound?

This sounds very reasonable 👍🏼

Resolving patch:
diff --git a/kedro/framework/project/__init__.py b/kedro/framework/project/__init__.py
--- a/kedro/framework/project/__init__.py
+++ b/kedro/framework/project/__init__.py
@@ -7,6 +7,7 @@
 import os
 import sys
 import traceback
+import types
 import warnings
 from collections import UserDict
 from collections.abc import MutableMapping
@@ -23,6 +24,11 @@
 
 from kedro.pipeline import Pipeline, pipeline
 
+IMPORT_ERROR_MESSAGE = (
+    "An error occurred while importing the '{module}' module. Nothing "
+    "defined therein will be returned by 'find_pipelines'.\n\n{tb_exc}"
+)
+
 
 def _get_default_class(class_import_path):
     module, _, class_name = class_import_path.rpartition(".")
@@ -270,6 +276,29 @@ def validate_settings():
     importlib.import_module(f"{PACKAGE_NAME}.settings")
 
 
+def _create_pipeline(pipeline_module: types.ModuleType) -> Optional[Pipeline]:
+    if not hasattr(pipeline_module, "create_pipeline"):
+        warnings.warn(
+            f"The '{pipeline_module.__name__}' module does not "
+            f"expose a 'create_pipeline' function, so no pipelines "
+            f"defined therein will be returned by 'find_pipelines'."
+        )
+        return None
+
+    obj = getattr(pipeline_module, "create_pipeline")()
+    if not isinstance(obj, Pipeline):
+        warnings.warn(
+            f"Expected the 'create_pipeline' function in the "
+            f"'{pipeline_module.__name__}' module to return a "
+            f"'Pipeline' object, got '{type(obj).__name__}' "
+            f"instead. Nothing defined therein will be returned by "
+            f"'find_pipelines'."
+        )
+        return None
+
+    return obj
+
+
 def find_pipelines() -> Dict[str, Pipeline]:
     """Automatically find modular pipelines having a ``create_pipeline``
     function. By default, projects created using Kedro 0.18.3 and higher
@@ -290,10 +319,32 @@ def find_pipelines() -> Dict[str, Pipeline]:
             function, the ``create_pipeline`` function does not return a
             ``Pipeline`` object, or if the module import fails up front.
     """
-    pipelines_dict = {"__default__": pipeline([])}
-    for pipeline_dir in importlib_resources.files(
-        f"{PACKAGE_NAME}.pipelines"
-    ).iterdir():
+    pipeline_obj = None
+
+    # Handle the simplified project structure found in several starters.
+    pipeline_module_name = f"{PACKAGE_NAME}.pipeline"
+    try:
+        pipeline_module = importlib.import_module(pipeline_module_name)
+    except Exception as exc:  # pylint: disable=broad-except
+        if str(exc) != f"No module named '{pipeline_module_name}'":
+            warnings.warn(
+                IMPORT_ERROR_MESSAGE.format(
+                    module=pipeline_module_name, tb_exc=traceback.format_exc()
+                )
+            )
+    else:
+        pipeline_obj = _create_pipeline(pipeline_module)
+
+    pipelines_dict = {"__default__": pipeline_obj or pipeline([])}
+
+    # Handle the case that a project doesn't have a pipelines directory.
+    try:
+        pipelines_package = importlib_resources.files(f"{PACKAGE_NAME}.pipelines")
+    except ModuleNotFoundError as exc:
+        if str(exc) == f"No module named '{PACKAGE_NAME}.pipelines'":
+            return pipelines_dict
+
+    for pipeline_dir in pipelines_package.iterdir():
         if not pipeline_dir.is_dir():
             continue
 
@@ -301,37 +352,18 @@ def find_pipelines() -> Dict[str, Pipeline]:
         if pipeline_name == "__pycache__":
             continue
 
+        pipeline_module_name = f"{PACKAGE_NAME}.pipelines.{pipeline_name}"
         try:
-            pipeline_module = importlib.import_module(
-                f"{PACKAGE_NAME}.pipelines.{pipeline_name}"
-            )
+            pipeline_module = importlib.import_module(pipeline_module_name)
         except:  # pylint: disable=bare-except  # noqa: E722
             warnings.warn(
-                f"An error occurred while importing the "
-                f"'{PACKAGE_NAME}.pipelines.{pipeline_name}' module. "
-                f"Nothing defined therein will be returned by "
-                f"'find_pipelines'.\n\n{traceback.format_exc()}"
-            )
-            continue
-
-        if not hasattr(pipeline_module, "create_pipeline"):
-            warnings.warn(
-                f"The '{pipeline_module.__name__}' module does not "
-                f"expose a 'create_pipeline' function, so no pipelines "
-                f"defined therein will be returned by 'find_pipelines'."
-            )
-            continue
-
-        obj = getattr(pipeline_module, "create_pipeline")()
-        if not isinstance(obj, Pipeline):
-            warnings.warn(
-                f"Expected the 'create_pipeline' function in the "
-                f"'{pipeline_module.__name__}' module to return a "
-                f"'Pipeline' object, got '{type(obj).__name__}' "
-                f"instead. Nothing defined therein will be returned by "
-                f"'find_pipelines'."
+                IMPORT_ERROR_MESSAGE.format(
+                    module=pipeline_module_name, tb_exc=traceback.format_exc()
+                )
             )
             continue
 
-        pipelines_dict[pipeline_name] = obj
+        pipeline_obj = _create_pipeline(pipeline_module)
+        if pipeline_obj is not None:
+            pipelines_dict[pipeline_name] = pipeline_obj
     return pipelines_dict


Test patch:
diff --git a/tests/framework/project/test_pipeline_discovery.py b/tests/framework/project/test_pipeline_discovery.py
--- a/tests/framework/project/test_pipeline_discovery.py
+++ b/tests/framework/project/test_pipeline_discovery.py
@@ -1,3 +1,4 @@
+import shutil
 import sys
 import textwrap
 import warnings
@@ -32,9 +33,14 @@ def create_pipeline(**kwargs) -> Pipeline:
     yield package_name
     sys.path.pop(0)
 
+    # Make sure that any new `test_package.pipeline` module gets loaded.
+    if f"{package_name}.pipeline" in sys.modules:
+        del sys.modules[f"{package_name}.pipeline"]
+
     # Make sure that the `importlib_resources.files` in `find_pipelines`
     # will point to the correct `test_package.pipelines` not from cache.
-    del sys.modules[f"{package_name}.pipelines"]
+    if f"{package_name}.pipelines" in sys.modules:
+        del sys.modules[f"{package_name}.pipelines"]
 
 
 @pytest.fixture
@@ -47,10 +53,7 @@ def pipeline_names(request):
     [(x, x) for x in [set(), {"my_pipeline"}]],
     indirect=True,
 )
-def test_find_pipelines(
-    mock_package_name_with_pipelines,
-    pipeline_names,
-):
+def test_find_pipelines(mock_package_name_with_pipelines, pipeline_names):
     configure_project(mock_package_name_with_pipelines)
     pipelines = find_pipelines()
     assert set(pipelines) == pipeline_names | {"__default__"}
@@ -63,8 +66,7 @@ def test_find_pipelines(
     indirect=True,
 )
 def test_find_pipelines_skips_modules_without_create_pipelines_function(
-    mock_package_name_with_pipelines,
-    pipeline_names,
+    mock_package_name_with_pipelines, pipeline_names
 ):
     # Create a module without `create_pipelines` in the `pipelines` dir.
     pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
@@ -87,8 +89,7 @@ def test_find_pipelines_skips_modules_without_create_pipelines_function(
     indirect=True,
 )
 def test_find_pipelines_skips_modules_with_unexpected_return_value_type(
-    mock_package_name_with_pipelines,
-    pipeline_names,
+    mock_package_name_with_pipelines, pipeline_names
 ):
     # Define `create_pipelines` so that it does not return a `Pipeline`.
     pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
@@ -130,8 +131,7 @@ def create_pipeline(**kwargs) -> Dict[str, Pipeline]:
     indirect=True,
 )
 def test_find_pipelines_skips_regular_files_within_the_pipelines_folder(
-    mock_package_name_with_pipelines,
-    pipeline_names,
+    mock_package_name_with_pipelines, pipeline_names
 ):
     # Create a regular file (not a subdirectory) in the `pipelines` dir.
     pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
@@ -151,8 +151,7 @@ def test_find_pipelines_skips_regular_files_within_the_pipelines_folder(
     indirect=True,
 )
 def test_find_pipelines_skips_modules_that_cause_exceptions_upon_import(
-    mock_package_name_with_pipelines,
-    pipeline_names,
+    mock_package_name_with_pipelines, pipeline_names
 ):
     # Create a module that will result in errors when we try to load it.
     pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
@@ -162,9 +161,100 @@ def test_find_pipelines_skips_modules_that_cause_exceptions_upon_import(
 
     configure_project(mock_package_name_with_pipelines)
     with pytest.warns(
-        UserWarning,
-        match=r"An error occurred while importing the '\S+' module.",
+        UserWarning, match=r"An error occurred while importing the '\S+' module."
+    ):
+        pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_handles_simplified_project_structure(
+    mock_package_name_with_pipelines, pipeline_names
+):
+    (Path(sys.path[0]) / mock_package_name_with_pipelines / "pipeline.py").write_text(
+        textwrap.dedent(
+            """
+            from kedro.pipeline import Pipeline, node, pipeline
+
+
+            def create_pipeline(**kwargs) -> Pipeline:
+                return pipeline([node(lambda: 1, None, "simple_pipeline")])
+            """
+        )
+    )
+
+    configure_project(mock_package_name_with_pipelines)
+    pipelines = find_pipelines()
+    assert set(pipelines) == pipeline_names | {"__default__"}
+    assert sum(pipelines.values()).outputs() == pipeline_names | {"simple_pipeline"}
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,pipeline_names",
+    [(x, x) for x in [set(), {"my_pipeline"}]],
+    indirect=True,
+)
+def test_find_pipelines_skips_unimportable_pipeline_module(
+    mock_package_name_with_pipelines, pipeline_names
+):
+    (Path(sys.path[0]) / mock_package_name_with_pipelines / "pipeline.py").write_text(
+        textwrap.dedent(
+            f"""
+            import {"".join(pipeline_names)}
+
+            from kedro.pipeline import Pipeline, node, pipeline
+
+
+            def create_pipeline(**kwargs) -> Pipeline:
+                return pipeline([node(lambda: 1, None, "simple_pipeline")])
+            """
+        )
+    )
+
+    configure_project(mock_package_name_with_pipelines)
+    with pytest.warns(
+        UserWarning, match=r"An error occurred while importing the '\S+' module."
     ):
         pipelines = find_pipelines()
     assert set(pipelines) == pipeline_names | {"__default__"}
     assert sum(pipelines.values()).outputs() == pipeline_names
+
+
+@pytest.mark.parametrize(
+    "mock_package_name_with_pipelines,simplified",
+    [(set(), False), (set(), True)],
+    indirect=["mock_package_name_with_pipelines"],
+)
+def test_find_pipelines_handles_project_structure_without_pipelines_dir(
+    mock_package_name_with_pipelines, simplified
+):
+    # Delete the `pipelines` directory to simulate a project without it.
+    pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
+    shutil.rmtree(pipelines_dir)
+
+    if simplified:
+        (
+            Path(sys.path[0]) / mock_package_name_with_pipelines / "pipeline.py"
+        ).write_text(
+            textwrap.dedent(
+                """
+                from kedro.pipeline import Pipeline, node, pipeline
+
+
+                def create_pipeline(**kwargs) -> Pipeline:
+                    return pipeline([node(lambda: 1, None, "simple_pipeline")])
+                """
+            )
+        )
+
+    configure_project(mock_package_name_with_pipelines)
+    pipelines = find_pipelines()
+    assert set(pipelines) == {"__default__"}
+    assert sum(pipelines.values()).outputs() == (
+        {"simple_pipeline"} if simplified else set()
+    )