Repository: kedro-org/kedro

Issue:
Make import failures in `kedro-datasets` clearer
## Description
> [!NOTE]  
> See https://github.com/kedro-org/kedro/issues/2943#issuecomment-1944348860 for current status

Disambiguate the "module does not exist" error from the `ImportError` in messages like:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.
```

This task will require investigation into how the error messages are swallowed, which will inform the proper implementation. Probably change: https://github.com/kedro-org/kedro/blob/4194fbd16a992af0320a395c0060aaaea356efb2/kedro/io/core.py#L433

## Context
This week I've been battling some puzzling documentation errors and after a while I noticed that, if the dependencies of a particular dataset are not present, the `ImportError` is swallowed silently. Examples:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/biosequence/__init__.py#L7-L8

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/networkx/__init__.py#L8-L15

This was done in https://github.com/quantumblacklabs/private-kedro/pull/575 to solve https://github.com/quantumblacklabs/private-kedro/issues/563 at the same time dependencies were moved to `extras_require`.

I see how _not_ suppressing these errors could be extremely annoying back then, because `kedro.io` used to re-export all the datasets in its `__init__.py`:

https://github.com/quantumblacklabs/private-kedro/blob/f7dd2478aec4de1b46afbaded9bce3c69bff6304/kedro/io/__init__.py#L29-L47

```python
# kedro/io/__init__.py

"""``kedro.io`` provides functionality to read and write to a
number of data sets. At core of the library is ``AbstractDataSet``
which allows implementation of various ``AbstractDataSet``s.
"""

from .cached_dataset import CachedDataSet  # NOQA
from .core import AbstractDataSet  # NOQA
from .core import AbstractVersionedDataSet  # NOQA
from .core import DataSetAlreadyExistsError  # NOQA
from .core import DataSetError  # NOQA
from .core import DataSetNotFoundError  # NOQA
from .core import Version  # NOQA
from .data_catalog import DataCatalog  # NOQA
from .data_catalog_with_default import DataCatalogWithDefault  # NOQA
from .lambda_data_set import LambdaDataSet  # NOQA
from .memory_data_set import MemoryDataSet  # NOQA
from .partitioned_data_set import IncrementalDataSet  # NOQA
from .partitioned_data_set import PartitionedDataSet  # NOQA
from .transformers import AbstractTransformer  # NOQA
```

However, now our `__init__.py` is empty and datasets are meant to be imported separately:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/__init__.py#L1-L3

So I think it would be much better if we did _not_ silence those import errors.

## More context
If one dependency is missing, the user would get an unhelpful "module X has no attribute Y" when trying to import a dataset rather than an actual error:

```
> pip uninstall biopython                          (kedro-dev) 
Found existing installation: biopython 1.81
Uninstalling biopython-1.81:
  Would remove:
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/Bio/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/BioSQL/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/biopython-1.81.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled biopython-1.81
> python                                           (kedro-dev) 
Python 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kedro_datasets.biosequence import BioSequenceDataSet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'BioSequenceDataSet' from 'kedro_datasets.biosequence' (/Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/kedro_datasets/biosequence/__init__.py)
```



Discussion:
Last time I saw this I was thinking the same, it turns out to be not trivial because of this. Not sure if things are still the same but worth mentioning.

https://github.com/kedro-org/kedro/blob/38dfe6f5e9d2e8758a732044267769aafb1ff465/kedro/io/core.py#L381-L389
Thanks @noklam - I see that an error saying "some of the dependencies have not been installed" is raised if the desired class is the list of`_DEFAULT_PACKAGES` (which includes `""`, hence allowing the user to type the full import path) is exhausted:

https://github.com/kedro-org/kedro/blob/38dfe6f5e9d2e8758a732044267769aafb1ff465/kedro/io/core.py#L392-L396

I see how we would not like to bubble all `ImportError`s to the user because in most cases, only 1 of those prefixes will have the desired class. Maybe we'd want to bubble up `ModuleNotFoundError` errors, but definitely I'd have to think this through.
@astrojuanlu I think I come to similar conclusion, actually I am not sure why the issue disappear I thought I created it but I couldn't find it... 
I would push for this if it's possible. Definitely one of the most annoying thing and as a beginner you will hit this issue for sure (almost as annoying as the DataSet vs Dataset typo issue)
Updated the title of this issue and adding more evidence:

https://linen-slack.kedro.org/t/14153932/hey-team-i-m-getting-the-following-exception-an-exception-oc#8b6debb1-ed6f-4a16-9440-5a03d0370b07

> Hey team, I'm getting the following exception:
> ```
> An exception occurred when parsing config for dataset 'raw_web@delta':
> Object 'DeltaTableDataSet' cannot be loaded from 'kedro.extras.datasets.spark'. Please see the documentation on how to install relevant dependencies for kedro.extras.datasets.spark.DeltaTableDataSet:
> <https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html>
> ```
> If I go to the link, it doesn't tell me what I should be installing.
> I already have the following in the requirements.txt: kedro-datasets[spark,delta]

Indeed, `https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html` contains a lot and is difficult to parse.

I believe the same thing happened just this morning to @NeroOkwa.

Maybe it would be better if the underlying `ImportError` could be shown.

In any case, this is something that should be changed on Kedro, so I'm moving the issue.
> Remove all the with suppress(ImportError):.

Without reading the full context, are you aware of this is merged already? I think the error is surfaced now already (with kedro-datasets). I haven't fully tested it tho.
- https://github.com/kedro-org/kedro-plugins/pull/277
This is what I see now:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.
```

doesn't tell me whether I made a typo, or what dependency I need to install. Am I missing something? This is with kedro-datasets 1.5.2
Will revisit that when we fixed the dataset installation issue. If this is not the case yet, I think we are in a good position to solve this finally. 
I want to have more clarification of this ticket. Is this a `kedro` or `kedro-datasets` ticket? The description and discussion is slightly outdated because https://github.com/kedro-org/kedro-plugins/pull/277 change how we import library.

The original error mentioned in the description is something in `kedro` rather than `kedro-datasets`
> DatasetError: An exception occurred when parsing config for dataset 'companies':
> Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.

```
from kedro_datasets.biosequence import BioSequenceDataSet
```
> ModuleNotFoundError: No module named 'Bio'

@astrojuanlu Would you be able to give me an fail cases so I can investigate this?

Resolving patch:
diff --git a/kedro/io/core.py b/kedro/io/core.py
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -376,14 +376,14 @@ def parse_dataset_definition(
     if "type" not in config:
         raise DatasetError("'type' is missing from dataset catalog configuration")
 
-    class_obj = config.pop("type")
-    if isinstance(class_obj, str):
-        if len(class_obj.strip(".")) != len(class_obj):
+    dataset_type = config.pop("type")
+    if isinstance(dataset_type, str):
+        if len(dataset_type.strip(".")) != len(dataset_type):
             raise DatasetError(
                 "'type' class path does not support relative "
                 "paths or paths ending with a dot."
             )
-        class_paths = (prefix + class_obj for prefix in _DEFAULT_PACKAGES)
+        class_paths = (prefix + dataset_type for prefix in _DEFAULT_PACKAGES)
 
         for class_path in class_paths:
             tmp = _load_obj(class_path)
@@ -391,10 +391,7 @@ def parse_dataset_definition(
                 class_obj = tmp
                 break
         else:
-            raise DatasetError(
-                f"Class '{class_obj}' not found or one of its dependencies "
-                f"has not been installed."
-            )
+            raise DatasetError(f"Class '{dataset_type}' not found, is this a typo?")
 
     if not issubclass(class_obj, AbstractDataset):
         raise DatasetError(
@@ -422,8 +419,9 @@ def parse_dataset_definition(
     return class_obj, config
 
 
-def _load_obj(class_path: str) -> object | None:
+def _load_obj(class_path: str) -> Any | None:
     mod_path, _, class_name = class_path.rpartition(".")
+    # Check if the module exists
     try:
         available_classes = load_obj(f"{mod_path}.__all__")
     # ModuleNotFoundError: When `load_obj` can't find `mod_path` (e.g `kedro.io.pandas`)
@@ -432,18 +430,16 @@ def _load_obj(class_path: str) -> object | None:
     #                 `__all__` attribute -- either because it's a custom or a kedro.io dataset
     except (ModuleNotFoundError, AttributeError, ValueError):
         available_classes = None
-
     try:
         class_obj = load_obj(class_path)
-    except (ModuleNotFoundError, ValueError):
-        return None
-    except AttributeError as exc:
+    except (ModuleNotFoundError, ValueError, AttributeError) as exc:
+        # If it's available, module exist but dependencies are missing
         if available_classes and class_name in available_classes:
             raise DatasetError(
-                f"{exc} Please see the documentation on how to "
+                f"{exc}. Please see the documentation on how to "
                 f"install relevant dependencies for {class_path}:\n"
-                f"https://kedro.readthedocs.io/en/stable/"
-                f"kedro_project_setup/dependencies.html"
+                f"https://docs.kedro.org/en/stable/kedro_project_setup/"
+                f"dependencies.html#install-dependencies-related-to-the-data-catalog"
             ) from exc
         return None
 
diff --git a/kedro/utils.py b/kedro/utils.py
--- a/kedro/utils.py
+++ b/kedro/utils.py
@@ -23,6 +23,4 @@ def load_obj(obj_path: str, default_obj_path: str = "") -> Any:
     obj_path = obj_path_list.pop(0) if len(obj_path_list) > 1 else default_obj_path
     obj_name = obj_path_list[0]
     module_obj = importlib.import_module(obj_path)
-    if not hasattr(module_obj, obj_name):
-        raise AttributeError(f"Object '{obj_name}' cannot be loaded from '{obj_path}'.")
     return getattr(module_obj, obj_name)


Test patch:
diff --git a/tests/io/test_core.py b/tests/io/test_core.py
--- a/tests/io/test_core.py
+++ b/tests/io/test_core.py
@@ -19,6 +19,7 @@
     generate_timestamp,
     get_filepath_str,
     get_protocol_and_path,
+    parse_dataset_definition,
     validate_on_forbidden_chars,
 )
 
@@ -265,6 +266,32 @@ def test_validate_forbidden_chars(self, input):
         with pytest.raises(DatasetError, match=expected_error_message):
             validate_on_forbidden_chars(**input)
 
+    def test_dataset_name_typo(self, mocker):
+        # If the module doesn't exist, it return None instead ModuleNotFoundError
+        mocker.patch("kedro.io.core.load_obj", return_value=None)
+        dataset_name = "lAmbDaDaTAsET"
+
+        with pytest.raises(
+            DatasetError, match=f"Class '{dataset_name}' not found, is this a typo?"
+        ):
+            parse_dataset_definition({"type": dataset_name})
+
+    def test_dataset_missing_dependencies(self, mocker):
+        # If the module is found but import the dataset trigger ModuleNotFoundError
+        dataset_name = "LambdaDataset"
+
+        def side_effect_function(value):
+            if "__all__" in value:
+                return [dataset_name]
+            else:
+                raise ModuleNotFoundError
+
+        mocker.patch("kedro.io.core.load_obj", side_effect=side_effect_function)
+
+        pattern = "Please see the documentation on how to install relevant dependencies"
+        with pytest.raises(DatasetError, match=pattern):
+            parse_dataset_definition({"type": dataset_name})
+
 
 class TestAbstractVersionedDataset:
     def test_version_str_repr(self, load_version, save_version):
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -18,12 +18,6 @@ def test_load_obj_default_path(self):
         extracted_obj = load_obj("DummyClass", "tests.test_utils")
         assert extracted_obj is DummyClass
 
-    def test_load_obj_invalid_attribute(self):
-        with pytest.raises(
-            AttributeError, match=r"Object 'InvalidClass' cannot be loaded"
-        ):
-            load_obj("InvalidClass", "tests.test_utils")
-
     def test_load_obj_invalid_module(self):
         with pytest.raises(ImportError, match=r"No module named 'missing_path'"):
             load_obj("InvalidClass", "missing_path")