Repository: kedro-org/kedro
Pull request: Improve error handling for Custom Datasets
## Description
Resolves https://github.com/kedro-org/kedro/issues/4577

Before the fix; if there were errors such as import errors in your custom dataset, you got the following error message

```
DatasetError: An exception occurred when parsing config for dataset 'reporting.cancellation_policy_grid':
Class 'demo_project.extras.datasets.image_dataset.ImageDataset' not found, is this a typo?
```
After the fix; you get the following error message 
```
DatasetError: An exception occurred when parsing config for dataset 'reporting.cancellation_policy_grid':
No module named 'hdfs'
```

## Development notes

Previously, `_load_obj() `ignored warnings and did not pass them back to parse_dataset_definition(). As a result, when `_load_obj()` failed to load a dataset class, we had no way of knowing what went wrong. Instead of surfacing the actual issue, parse_dataset_definition() misleadingly raised:

"Class '{dataset_type}' not found, is this a typo?"

However, the real issue was not always a typo—it could have been a ModuleNotFoundError, ValueError, or AttributeError, indicating missing dependencies or misconfigurations. Because `_load_obj()` failed silently, we lacked visibility into the real cause of failure.

in this PR, we capture and return warnings from _load_obj()

`_load_obj()` now returns both the class object and any warnings/errors encountered during loading.
This ensures that we do not lose useful debugging information.
Raise the correct error in parse_dataset_definition()

Instead of assuming a typo, we now check for warnings from `_load_obj()` and raise them if they exist.
If no specific warning is found, we fall back to the generic error message.

 


## Developer Certificate of Origin
We need all contributions to comply with the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). All commits must be signed off by including a `Signed-off-by` line in the commit message. [See our wiki for guidance](https://github.com/kedro-org/kedro/wiki/Guidelines-for-contributing-developers/).

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

## Checklist

- [ ] Read the [contributing](https://github.com/kedro-org/kedro/blob/main/CONTRIBUTING.md) guidelines
- [ ] Signed off each commit with a [Developer Certificate of Origin (DCO)](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-the-commit-signoff-policy-for-your-repository)
- [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [ ] Updated the documentation to reflect the code changes
- [ ] Added a description of this change in the [`RELEASE.md`](https://github.com/kedro-org/kedro/blob/main/RELEASE.md) file
- [ ] Added tests to cover my changes
- [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team


Problem statement:
Importing custom kedro catalog with missing dependency suppreses missing library import error traceback
## Description
When importing catalog class (e.g., `MyCustomDataset`) kedro suppresses import traceback in case the class dependencies are missing. If the user has the mentioned class but it relies on some other libs that were not installed, the user will never know but will get the misleading error (something like `MyCustomDataset` not found, is this a typo?) instead.

## Context
I've spent a lot of time understanding why my custom Spark dataset wasn't loaded and the reason was I didn't have `hdfs` installed on the worker node. 

That has to do with the way the import is handled now in `parse_dataset_definition` -> `_load_obj`. We should carefully check if the dataset exists and if it exists, we should try to load that propagating any issues during its import. Now we try to import and if `ModuleNotFound` occurs, we don't propagate the underlying traceback.

## Steps to Reproduce
1. Create a custom dataset (e.g., `MyCustomDataset`)
2. Import uninstalled lib in it (e.g., `hdfs`)
3. Mention that class in the catalog config
4. Run a session in which the catalog item will be used

## Expected Result
Full traceback with the one mentioning missing lib (`hdfs` in my example)

## Actual Result
Reduced traceback like "`MyCustomDataset` not found, is this a typo?"

## Your Environment
* kero 0.19.3
* python 3.10

Hints from issue discussion:
As I look at the implementation, this comment in `core:_load_obj` doesn't seem valid

```
# ModuleNotFoundError: When `load_obj` can't find `mod_path` (e.g `kedro.io.pandas`)
#                      this is because we try a combination of all prefixes.
# AttributeError: When `load_obj` manages to load `mod_path` but it doesn't have an
#                 `__all__` attribute -- either because it's a custom or a kedro.io dataset
```
@star-yar, I confirm that I can reproduce this error and am currently working on a fix. Thank you for reporting it.
I've added the test of the expected behaviour https://github.com/star-yar/kedro

Maybe that can be usefull

Commit message: fix tests

Patch to review:
diff --git a/.secrets.baseline b/.secrets.baseline
index cbacd9f01c..43586daf06 100644
--- a/.secrets.baseline
+++ b/.secrets.baseline
@@ -206,7 +206,7 @@
         "filename": "tests/io/test_data_catalog.py",
         "hashed_secret": "15dd2c9ccec914f1470b4dccb45789844e49cf70",
         "is_verified": false,
-        "line_number": 529
+        "line_number": 530
       }
     ],
     "tests/io/test_kedro_data_catalog.py": [
@@ -219,5 +219,5 @@
       }
     ]
   },
-  "generated_at": "2025-02-06T19:46:16Z"
+  "generated_at": "2025-03-17T17:50:02Z"
 }
diff --git a/kedro/io/core.py b/kedro/io/core.py
index 03e0d4d853..7c26116031 100644
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -578,6 +578,7 @@ def parse_dataset_definition(
 
     dataset_type = config.pop(TYPE_KEY)
     class_obj = None
+    warning = None
     if isinstance(dataset_type, str):
         if len(dataset_type.strip(".")) != len(dataset_type):
             raise DatasetError(
@@ -587,21 +588,28 @@ def parse_dataset_definition(
         class_paths = (prefix + dataset_type for prefix in _DEFAULT_PACKAGES)
 
         for class_path in class_paths:
-            tmp = _load_obj(class_path)
+            tmp, warning = _load_obj(
+                class_path
+            )  # Load dataset class, capture the warning
+
             if tmp is not None:
                 class_obj = tmp
                 break
-        else:
+
+        if class_obj is None:  # If no valid class was found, raise an error
+            error_msg = (
+                warning
+                if warning
+                else f"Class '{dataset_type}' not found, is this a typo?"
+            )
             hint = (
-                "Hint: If you are trying to use a dataset from `kedro-datasets`, "
+                "\nHint: If you are trying to use a dataset from `kedro-datasets`, "
                 "make sure that the package is installed in your current environment. "
                 "You can do so by running `pip install kedro-datasets` or "
                 "`pip install kedro-datasets[<dataset-group>]` to install `kedro-datasets` along with "
                 "related dependencies for the specific dataset group."
             )
-            raise DatasetError(
-                f"Class '{dataset_type}' not found, is this a typo?" f"\n{hint}"
-            )
+            raise DatasetError(f"{error_msg}{hint}")
 
     if not class_obj:
         class_obj = dataset_type
@@ -654,9 +662,9 @@ def _load_obj(class_path: str) -> Any | None:
                 f"https://docs.kedro.org/en/stable/kedro_project_setup/"
                 f"dependencies.html#install-dependencies-related-to-the-data-catalog"
             ) from exc
-        return None
+        return None, str(exc)
 
-    return class_obj
+    return class_obj, None
 
 
 def _local_exists(filepath: str) -> bool:  # SKIP_IF_NO_SPARK
diff --git a/tests/io/test_core.py b/tests/io/test_core.py
index f3d92ca8b5..1ebed84deb 100644
--- a/tests/io/test_core.py
+++ b/tests/io/test_core.py
@@ -328,31 +328,27 @@ def test_dataset_name_typo(self, mocker):
         ):
             parse_dataset_definition({"type": dataset_name})
 
+    def test_parse_dataset_definition(self):
+        config = {"type": "LambdaDataset"}
+        dataset, _ = parse_dataset_definition(config)
+        assert dataset is LambdaDataset
+
+    def test_parse_dataset_definition_with_python_class_type(self):
+        config = {"type": MyDataset}
+        parse_dataset_definition(config)
+
     def test_dataset_missing_dependencies(self, mocker):
-        # If the module is found but import the dataset trigger ModuleNotFoundError
         dataset_name = "LambdaDataset"
 
         def side_effect_function(value):
-            if "__all__" in value:
-                return [dataset_name]
-            else:
-                raise ModuleNotFoundError
+            import random_package  # noqa: F401
 
         mocker.patch("kedro.io.core.load_obj", side_effect=side_effect_function)
 
-        pattern = "Please see the documentation on how to install relevant dependencies"
+        pattern = "No module named 'random_package'.*"
         with pytest.raises(DatasetError, match=pattern):
             parse_dataset_definition({"type": dataset_name})
 
-    def test_parse_dataset_definition(self):
-        config = {"type": "LambdaDataset"}
-        dataset, _ = parse_dataset_definition(config)
-        assert dataset is LambdaDataset
-
-    def test_parse_dataset_definition_with_python_class_type(self):
-        config = {"type": MyDataset}
-        parse_dataset_definition(config)
-
     def test_load_and_save_are_wrapped_once(self):
         assert not getattr(
             MyOtherVersionedDataset.load.__wrapped__, "__loadwrapped__", False
diff --git a/tests/io/test_data_catalog.py b/tests/io/test_data_catalog.py
index 180919e309..b378610056 100644
--- a/tests/io/test_data_catalog.py
+++ b/tests/io/test_data_catalog.py
@@ -421,7 +421,7 @@ def test_config_invalid_module(self, correct_config):
             "kedro.invalid_module_name.io.CSVDataset"
         )
 
-        error_msg = "Class 'kedro.invalid_module_name.io.CSVDataset' not found"
+        error_msg = "No module named 'kedro.invalid_module_name'"
         with pytest.raises(DatasetError, match=re.escape(error_msg)):
             DataCatalog.from_config(**correct_config)
 
@@ -457,10 +457,11 @@ def test_config_missing_class(self, correct_config):
         correct_config["catalog"]["boats"]["type"] = "kedro.io.CSVDatasetInvalid"
 
         pattern = (
-            "An exception occurred when parsing config for dataset 'boats':\n"
-            "Class 'kedro.io.CSVDatasetInvalid' not found, is this a typo?"
+            r"An exception occurred when parsing config for dataset 'boats':\n"
+            r"module 'kedro.io' has no attribute 'CSVDatasetInvalid'.*"
         )
-        with pytest.raises(DatasetError, match=re.escape(pattern)):
+
+        with pytest.raises(DatasetError, match=pattern):
             DataCatalog.from_config(**correct_config)
 
     def test_config_invalid_dataset(self, correct_config):
diff --git a/tests/io/test_kedro_data_catalog.py b/tests/io/test_kedro_data_catalog.py
index 4873e4a38b..ed40875419 100644
--- a/tests/io/test_kedro_data_catalog.py
+++ b/tests/io/test_kedro_data_catalog.py
@@ -458,7 +458,7 @@ def test_config_invalid_module(self, correct_config):
                 "kedro.invalid_module_name.io.CSVDataset"
             )
 
-            error_msg = "Class 'kedro.invalid_module_name.io.CSVDataset' not found"
+            error_msg = "No module named 'kedro.invalid_module_name'"
             with pytest.raises(DatasetError, match=re.escape(error_msg)):
                 KedroDataCatalog.from_config(**correct_config).get("boats")
 
@@ -486,10 +486,10 @@ def test_config_missing_class(self, correct_config):
             correct_config["catalog"]["boats"]["type"] = "kedro.io.CSVDatasetInvalid"
 
             pattern = (
-                "An exception occurred when parsing config for dataset 'boats':\n"
-                "Class 'kedro.io.CSVDatasetInvalid' not found, is this a typo?"
+                r"An exception occurred when parsing config for dataset 'boats':\n"
+                r"module 'kedro.io' has no attribute 'CSVDatasetInvalid'.*"
             )
-            with pytest.raises(DatasetError, match=re.escape(pattern)):
+            with pytest.raises(DatasetError, match=pattern):
                 KedroDataCatalog.from_config(**correct_config).get("boats")
 
         def test_config_invalid_dataset(self, correct_config):


Code review comments:
diff_hunk: @@ -587,21 +588,28 @@ def parse_dataset_definition(
         class_paths = (prefix + dataset_type for prefix in _DEFAULT_PACKAGES)
 
         for class_path in class_paths:
-            tmp = _load_obj(class_path)
+            tmp, warning = _load_obj(
+                class_path
+            )  # Load dataset class, capture the warning
+
             if tmp is not None:
                 class_obj = tmp
                 break
-        else:
+
+        if class_obj is None:  # If no valid class was found, raise an error
+            error_msg = (
+                warning
+                if warning
+                else f"Class '{dataset_type}' not found, is this a typo?"
path: kedro/io/core.py
text: @user1:
I'm not sure this error will now ever be reached. I played around with it and if I have a class called `MyCSVDataset` but try to import `YourCSVDataset` in the catalog I get:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
module 'wizard.my_dataset' has no attribute 'YourCSVDataset'
```




@user1:
I think it's still better in terms of overall experience compared to before, but maybe it's still possible to catch the class error properly?