Repository: kedro-org/kedro

Issue:
Provide option to raise error explicitly for `find_pipelines`
## Description
<!-- Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..." -->

`find_pipelines` is added since 0.18.3 as a pipeline autodiscovery feature. While this is useful, it is sometimes hard to debug because it raises warning but not error so the program will keep running. Cc @deepyaman 


## Context
<!-- Why is this change important to you? How would you use it? How can it benefit other users? -->
For library developers, they may need strict control and want the pipeline fails explicitly. This features was designed for development, so any half-developed pipeline won't prevent other pipeline from running (incomplete pipeline, not installed library etc)

Some quotes from users:
User 1 (library developer)
> We recently have users reporting issues when running tensorflow on M1. The error was hidden in some kedro pipelines and it took a while to dig out the root cause from the user. In general, it can be difficult to predict when and where we will encounter these issue. Therefore, we are planning to build some CI workflows for checking our packages and pipelines for M1 machines in general.

User 2 (general user)
> I'm noticing that warnings like SyntaxErrors and type errors are considered "warnings" by Kedro, which continues to try to run the pipeline.  Is there a setting to escalate these to Errors so they can abort the pipeline run?
  [15 days ago](https://kedro-org.slack.com/archives/C03RKP2LW64/p1690226861464369?thread_ts=1690222660.057049&cid=C03RKP2LW64)
Is there a way to mark a pipeline as a work in progress and then tell find_pipelines to error on warnings?  Or is the answer to just not use pipeline autodiscovery

## Possible Implementation
<!-- (Optional) Suggest an idea for implementing the addition or change. -->
`find_pipelines(raise_errors=False)` - default unchanged, optionally fail eagerly.
## Possible Alternatives
<!-- (Optional) Describe any alternative solutions or features you've considered. -->



Discussion:
Yes! I've heard this too!

Resolving patch:
diff --git a/kedro/framework/project/__init__.py b/kedro/framework/project/__init__.py
--- a/kedro/framework/project/__init__.py
+++ b/kedro/framework/project/__init__.py
@@ -347,7 +347,7 @@ def _create_pipeline(pipeline_module: types.ModuleType) -> Pipeline | None:
     return obj
 
 
-def find_pipelines() -> dict[str, Pipeline]:  # noqa: PLR0912
+def find_pipelines(raise_errors: bool = False) -> dict[str, Pipeline]:  # noqa: PLR0912
     """Automatically find modular pipelines having a ``create_pipeline``
     function. By default, projects created using Kedro 0.18.3 and higher
     call this function to autoregister pipelines upon creation/addition.
@@ -359,13 +359,23 @@ def find_pipelines() -> dict[str, Pipeline]:  # noqa: PLR0912
     For more information on the pipeline registry and autodiscovery, see
     https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/pipeline_registry.html
 
+    Args:
+        raise_errors: If ``True``, raise an error upon failed discovery.
+
     Returns:
         A generated mapping from pipeline names to ``Pipeline`` objects.
 
+    Raises:
+        ImportError: When a module does not expose a ``create_pipeline``
+            function, the ``create_pipeline`` function does not return a
+            ``Pipeline`` object, or if the module import fails up front.
+            If ``raise_errors`` is ``False``, see Warns section instead.
+
     Warns:
         UserWarning: When a module does not expose a ``create_pipeline``
             function, the ``create_pipeline`` function does not return a
             ``Pipeline`` object, or if the module import fails up front.
+            If ``raise_errors`` is ``True``, see Raises section instead.
     """
     pipeline_obj = None
 
@@ -375,6 +385,12 @@ def find_pipelines() -> dict[str, Pipeline]:  # noqa: PLR0912
         pipeline_module = importlib.import_module(pipeline_module_name)
     except Exception as exc:
         if str(exc) != f"No module named '{pipeline_module_name}'":
+            if raise_errors:
+                raise ImportError(
+                    f"An error occurred while importing the "
+                    f"'{pipeline_module_name}' module."
+                ) from exc
+
             warnings.warn(
                 IMPORT_ERROR_MESSAGE.format(
                     module=pipeline_module_name, tb_exc=traceback.format_exc()
@@ -406,7 +422,13 @@ def find_pipelines() -> dict[str, Pipeline]:  # noqa: PLR0912
         pipeline_module_name = f"{PACKAGE_NAME}.pipelines.{pipeline_name}"
         try:
             pipeline_module = importlib.import_module(pipeline_module_name)
-        except:  # noqa: E722
+        except Exception as exc:
+            if raise_errors:
+                raise ImportError(
+                    f"An error occurred while importing the "
+                    f"'{pipeline_module_name}' module."
+                ) from exc
+
             warnings.warn(
                 IMPORT_ERROR_MESSAGE.format(
                     module=pipeline_module_name, tb_exc=traceback.format_exc()
diff --git a/kedro/pipeline/modular_pipeline.py b/kedro/pipeline/modular_pipeline.py
--- a/kedro/pipeline/modular_pipeline.py
+++ b/kedro/pipeline/modular_pipeline.py
@@ -99,8 +99,10 @@ def _get_dataset_names_mapping(
             the same as they are named in the provided pipeline.
             When dict[str, str] is provided, current names will be
             mapped to new names in the resultant pipeline.
+
     Returns:
         A dictionary that maps the old dataset names to the provided ones.
+
     Examples:
         >>> _get_dataset_names_mapping("dataset_name")
         {"dataset_name": "dataset_name"}  # a str name will stay the same
@@ -138,8 +140,10 @@ def _get_param_names_mapping(
             the same as they are named in the provided pipeline.
             When dict[str, str] is provided, current names will be
             mapped to new names in the resultant pipeline.
+
     Returns:
         A dictionary that maps the old parameter names to the provided ones.
+
     Examples:
         >>> _get_param_names_mapping("param_name")
         {"params:param_name": "params:param_name"}  # a str name will stay the same


Test patch:
diff --git a/tests/framework/project/test_pipeline_discovery.py b/tests/framework/project/test_pipeline_discovery.py
--- a/tests/framework/project/test_pipeline_discovery.py
+++ b/tests/framework/project/test_pipeline_discovery.py
@@ -177,12 +177,16 @@ def test_find_pipelines_skips_regular_files_within_the_pipelines_folder(
 
 
 @pytest.mark.parametrize(
-    "mock_package_name_with_pipelines,pipeline_names",
-    [(x, x) for x in [set(), {"my_pipeline"}]],
-    indirect=True,
+    "mock_package_name_with_pipelines,pipeline_names,raise_errors",
+    [
+        (x, x, raise_errors)
+        for x in [set(), {"my_pipeline"}]
+        for raise_errors in [True, False]
+    ],
+    indirect=["mock_package_name_with_pipelines", "pipeline_names"],
 )
 def test_find_pipelines_skips_modules_that_cause_exceptions_upon_import(
-    mock_package_name_with_pipelines, pipeline_names
+    mock_package_name_with_pipelines, pipeline_names, raise_errors
 ):
     # Create a module that will result in errors when we try to load it.
     pipelines_dir = Path(sys.path[0]) / mock_package_name_with_pipelines / "pipelines"
@@ -191,12 +195,14 @@ def test_find_pipelines_skips_modules_that_cause_exceptions_upon_import(
     (pipeline_dir / "__init__.py").write_text("I walk a lonely road...")
 
     configure_project(mock_package_name_with_pipelines)
-    with pytest.warns(
-        UserWarning, match=r"An error occurred while importing the '\S+' module."
+    with getattr(pytest, "raises" if raise_errors else "warns")(
+        ImportError if raise_errors else UserWarning,
+        match=r"An error occurred while importing the '\S+' module.",
     ):
-        pipelines = find_pipelines()
-    assert set(pipelines) == pipeline_names | {"__default__"}
-    assert sum(pipelines.values()).outputs() == pipeline_names
+        pipelines = find_pipelines(raise_errors=raise_errors)
+    if not raise_errors:
+        assert set(pipelines) == pipeline_names | {"__default__"}
+        assert sum(pipelines.values()).outputs() == pipeline_names
 
 
 @pytest.mark.parametrize(
@@ -226,12 +232,16 @@ def create_pipeline(**kwargs) -> Pipeline:
 
 
 @pytest.mark.parametrize(
-    "mock_package_name_with_pipelines,pipeline_names",
-    [(x, x) for x in [set(), {"my_pipeline"}]],
-    indirect=True,
+    "mock_package_name_with_pipelines,pipeline_names,raise_errors",
+    [
+        (x, x, raise_errors)
+        for x in [set(), {"my_pipeline"}]
+        for raise_errors in [True, False]
+    ],
+    indirect=["mock_package_name_with_pipelines", "pipeline_names"],
 )
 def test_find_pipelines_skips_unimportable_pipeline_module(
-    mock_package_name_with_pipelines, pipeline_names
+    mock_package_name_with_pipelines, pipeline_names, raise_errors
 ):
     (Path(sys.path[0]) / mock_package_name_with_pipelines / "pipeline.py").write_text(
         textwrap.dedent(
@@ -248,12 +258,14 @@ def create_pipeline(**kwargs) -> Pipeline:
     )
 
     configure_project(mock_package_name_with_pipelines)
-    with pytest.warns(
-        UserWarning, match=r"An error occurred while importing the '\S+' module."
+    with getattr(pytest, "raises" if raise_errors else "warns")(
+        ImportError if raise_errors else UserWarning,
+        match=r"An error occurred while importing the '\S+' module.",
     ):
-        pipelines = find_pipelines()
-    assert set(pipelines) == pipeline_names | {"__default__"}
-    assert sum(pipelines.values()).outputs() == pipeline_names
+        pipelines = find_pipelines(raise_errors=raise_errors)
+    if not raise_errors:
+        assert set(pipelines) == pipeline_names | {"__default__"}
+        assert sum(pipelines.values()).outputs() == pipeline_names
 
 
 @pytest.mark.parametrize(