Repository: kedro-org/kedro

Issue:
Make import failures in `kedro-datasets` clearer
## Description
Disambiguate the "module does not exist" error from the `ImportError` in messages like:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.
```

This task will require investigation into how the error messages are swallowed, which will inform the proper implementation. Probably change: https://github.com/kedro-org/kedro/blob/4194fbd16a992af0320a395c0060aaaea356efb2/kedro/io/core.py#L433

## Context
This week I've been battling some puzzling documentation errors and after a while I noticed that, if the dependencies of a particular dataset are not present, the `ImportError` is swallowed silently. Examples:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/biosequence/__init__.py#L7-L8

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/networkx/__init__.py#L8-L15

This was done in https://github.com/quantumblacklabs/private-kedro/pull/575 to solve https://github.com/quantumblacklabs/private-kedro/issues/563 at the same time dependencies were moved to `extras_require`.

I see how _not_ suppressing these errors could be extremely annoying back then, because `kedro.io` used to re-export all the datasets in its `__init__.py`:

https://github.com/quantumblacklabs/private-kedro/blob/f7dd2478aec4de1b46afbaded9bce3c69bff6304/kedro/io/__init__.py#L29-L47

```python
# kedro/io/__init__.py

"""``kedro.io`` provides functionality to read and write to a
number of data sets. At core of the library is ``AbstractDataSet``
which allows implementation of various ``AbstractDataSet``s.
"""

from .cached_dataset import CachedDataSet  # NOQA
from .core import AbstractDataSet  # NOQA
from .core import AbstractVersionedDataSet  # NOQA
from .core import DataSetAlreadyExistsError  # NOQA
from .core import DataSetError  # NOQA
from .core import DataSetNotFoundError  # NOQA
from .core import Version  # NOQA
from .data_catalog import DataCatalog  # NOQA
from .data_catalog_with_default import DataCatalogWithDefault  # NOQA
from .lambda_data_set import LambdaDataSet  # NOQA
from .memory_data_set import MemoryDataSet  # NOQA
from .partitioned_data_set import IncrementalDataSet  # NOQA
from .partitioned_data_set import PartitionedDataSet  # NOQA
from .transformers import AbstractTransformer  # NOQA
```

However, now our `__init__.py` is empty and datasets are meant to be imported separately:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/__init__.py#L1-L3

So I think it would be much better if we did _not_ silence those import errors.

## More context
If one dependency is missing, the user would get an unhelpful "module X has no attribute Y" when trying to import a dataset rather than an actual error:

```
> pip uninstall biopython                          (kedro-dev) 
Found existing installation: biopython 1.81
Uninstalling biopython-1.81:
  Would remove:
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/Bio/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/BioSQL/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/biopython-1.81.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled biopython-1.81
> python                                           (kedro-dev) 
Python 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kedro_datasets.biosequence import BioSequenceDataSet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'BioSequenceDataSet' from 'kedro_datasets.biosequence' (/Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/kedro_datasets/biosequence/__init__.py)
```


Comments:

Patch:
diff --git a/RELEASE.md b/RELEASE.md
index 90b8adf1..33dc1c28 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -6,6 +6,7 @@
 * The new spaceflights starters, `spaceflights-pandas`, `spaceflights-pandas-viz`, `spaceflights-pyspark`, and `spaceflights-pyspark-viz` can be used with the `kedro new` command with the `--starter` flag.
 * Added the `--conf-source` option to `%reload_kedro`, allowing users to specify a source for project configuration.
 * Added the functionality to choose a merging strategy for config files loaded with `OmegaConfigLoader`.
+* Modified the mechanism of importing datasets, raise more explicit error when dependencies are missing.
 
 
 ## Bug fixes and other changes
diff --git a/kedro/io/core.py b/kedro/io/core.py
index e620d15f..f605c272 100644
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -376,14 +376,14 @@ def parse_dataset_definition(
     if "type" not in config:
         raise DatasetError("'type' is missing from dataset catalog configuration")
 
-    class_obj = config.pop("type")
-    if isinstance(class_obj, str):
-        if len(class_obj.strip(".")) != len(class_obj):
+    dataset_type = config.pop("type")
+    if isinstance(dataset_type, str):
+        if len(dataset_type.strip(".")) != len(dataset_type):
             raise DatasetError(
                 "'type' class path does not support relative "
                 "paths or paths ending with a dot."
             )
-        class_paths = (prefix + class_obj for prefix in _DEFAULT_PACKAGES)
+        class_paths = (prefix + dataset_type for prefix in _DEFAULT_PACKAGES)
 
         for class_path in class_paths:
             tmp = _load_obj(class_path)
@@ -391,10 +391,7 @@ def parse_dataset_definition(
                 class_obj = tmp
                 break
         else:
-            raise DatasetError(
-                f"Class '{class_obj}' not found or one of its dependencies "
-                f"has not been installed."
-            )
+            raise DatasetError(f"Class '{dataset_type}' not found, is this a typo?")
 
     if not issubclass(class_obj, AbstractDataset):
         raise DatasetError(
@@ -422,8 +419,9 @@ def parse_dataset_definition(
     return class_obj, config
 
 
-def _load_obj(class_path: str) -> object | None:
+def _load_obj(class_path: str) -> Any | None:
     mod_path, _, class_name = class_path.rpartition(".")
+    # Check if the module exists
     try:
         available_classes = load_obj(f"{mod_path}.__all__")
     # ModuleNotFoundError: When `load_obj` can't find `mod_path` (e.g `kedro.io.pandas`)
@@ -432,18 +430,16 @@ def _load_obj(class_path: str) -> object | None:
     #                 `__all__` attribute -- either because it's a custom or a kedro.io dataset
     except (ModuleNotFoundError, AttributeError, ValueError):
         available_classes = None
-
     try:
         class_obj = load_obj(class_path)
-    except (ModuleNotFoundError, ValueError):
-        return None
-    except AttributeError as exc:
+    except (ModuleNotFoundError, ValueError, AttributeError) as exc:
+        # If it's available, module exist but dependencies are missing
         if available_classes and class_name in available_classes:
             raise DatasetError(
-                f"{exc} Please see the documentation on how to "
+                f"{exc}. Please see the documentation on how to "
                 f"install relevant dependencies for {class_path}:\n"
-                f"https://kedro.readthedocs.io/en/stable/"
-                f"kedro_project_setup/dependencies.html"
+                f"https://docs.kedro.org/en/stable/kedro_project_setup/"
+                f"dependencies.html#install-dependencies-related-to-the-data-catalog"
             ) from exc
         return None
 
diff --git a/kedro/utils.py b/kedro/utils.py
index 6067d96b..f527b909 100644
--- a/kedro/utils.py
+++ b/kedro/utils.py
@@ -23,6 +23,4 @@ def load_obj(obj_path: str, default_obj_path: str = "") -> Any:
     obj_path = obj_path_list.pop(0) if len(obj_path_list) > 1 else default_obj_path
     obj_name = obj_path_list[0]
     module_obj = importlib.import_module(obj_path)
-    if not hasattr(module_obj, obj_name):
-        raise AttributeError(f"Object '{obj_name}' cannot be loaded from '{obj_path}'.")
     return getattr(module_obj, obj_name)


Test patch:
diff --git a/tests/io/test_core.py b/tests/io/test_core.py
index dcf2f30a..1cbea798 100644
--- a/tests/io/test_core.py
+++ b/tests/io/test_core.py
@@ -19,6 +19,7 @@ from kedro.io.core import (
     generate_timestamp,
     get_filepath_str,
     get_protocol_and_path,
+    parse_dataset_definition,
     validate_on_forbidden_chars,
 )
 
@@ -265,6 +266,32 @@ class TestCoreFunctions:
         with pytest.raises(DatasetError, match=expected_error_message):
             validate_on_forbidden_chars(**input)
 
+    def test_dataset_name_typo(self, mocker):
+        # If the module doesn't exist, it return None instead ModuleNotFoundError
+        mocker.patch("kedro.io.core.load_obj", return_value=None)
+        dataset_name = "lAmbDaDaTAsET"
+
+        with pytest.raises(
+            DatasetError, match=f"Class '{dataset_name}' not found, is this a typo?"
+        ):
+            parse_dataset_definition({"type": dataset_name})
+
+    def test_dataset_missing_dependencies(self, mocker):
+        # If the module is found but import the dataset trigger ModuleNotFoundError
+        dataset_name = "LambdaDataset"
+
+        def side_effect_function(value):
+            if "__all__" in value:
+                return [dataset_name]
+            else:
+                raise ModuleNotFoundError
+
+        mocker.patch("kedro.io.core.load_obj", side_effect=side_effect_function)
+
+        pattern = "Please see the documentation on how to install relevant dependencies"
+        with pytest.raises(DatasetError, match=pattern):
+            parse_dataset_definition({"type": dataset_name})
+
 
 class TestAbstractVersionedDataset:
     def test_version_str_repr(self, load_version, save_version):
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 1ca93067..34704513 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -18,12 +18,6 @@ class TestExtractObject:
         extracted_obj = load_obj("DummyClass", "tests.test_utils")
         assert extracted_obj is DummyClass
 
-    def test_load_obj_invalid_attribute(self):
-        with pytest.raises(
-            AttributeError, match=r"Object 'InvalidClass' cannot be loaded"
-        ):
-            load_obj("InvalidClass", "tests.test_utils")
-
     def test_load_obj_invalid_module(self):
         with pytest.raises(ImportError, match=r"No module named 'missing_path'"):
             load_obj("InvalidClass", "missing_path")