Repository: kedro-org/kedro

Issue:
Make import failures in `kedro-datasets` clearer
## Description
> [!NOTE]  
> See https://github.com/kedro-org/kedro/issues/2943#issuecomment-1944348860 for current status

Disambiguate the "module does not exist" error from the `ImportError` in messages like:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.
```

This task will require investigation into how the error messages are swallowed, which will inform the proper implementation. Probably change: https://github.com/kedro-org/kedro/blob/4194fbd16a992af0320a395c0060aaaea356efb2/kedro/io/core.py#L433

## Context
This week I've been battling some puzzling documentation errors and after a while I noticed that, if the dependencies of a particular dataset are not present, the `ImportError` is swallowed silently. Examples:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/biosequence/__init__.py#L7-L8

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/networkx/__init__.py#L8-L15

This was done in https://github.com/quantumblacklabs/private-kedro/pull/575 to solve https://github.com/quantumblacklabs/private-kedro/issues/563 at the same time dependencies were moved to `extras_require`.

I see how _not_ suppressing these errors could be extremely annoying back then, because `kedro.io` used to re-export all the datasets in its `__init__.py`:

https://github.com/quantumblacklabs/private-kedro/blob/f7dd2478aec4de1b46afbaded9bce3c69bff6304/kedro/io/__init__.py#L29-L47

```python
# kedro/io/__init__.py

"""``kedro.io`` provides functionality to read and write to a
number of data sets. At core of the library is ``AbstractDataSet``
which allows implementation of various ``AbstractDataSet``s.
"""

from .cached_dataset import CachedDataSet  # NOQA
from .core import AbstractDataSet  # NOQA
from .core import AbstractVersionedDataSet  # NOQA
from .core import DataSetAlreadyExistsError  # NOQA
from .core import DataSetError  # NOQA
from .core import DataSetNotFoundError  # NOQA
from .core import Version  # NOQA
from .data_catalog import DataCatalog  # NOQA
from .data_catalog_with_default import DataCatalogWithDefault  # NOQA
from .lambda_data_set import LambdaDataSet  # NOQA
from .memory_data_set import MemoryDataSet  # NOQA
from .partitioned_data_set import IncrementalDataSet  # NOQA
from .partitioned_data_set import PartitionedDataSet  # NOQA
from .transformers import AbstractTransformer  # NOQA
```

However, now our `__init__.py` is empty and datasets are meant to be imported separately:

https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/__init__.py#L1-L3

So I think it would be much better if we did _not_ silence those import errors.

## More context
If one dependency is missing, the user would get an unhelpful "module X has no attribute Y" when trying to import a dataset rather than an actual error:

```
> pip uninstall biopython                          (kedro-dev) 
Found existing installation: biopython 1.81
Uninstalling biopython-1.81:
  Would remove:
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/Bio/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/BioSQL/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/biopython-1.81.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled biopython-1.81
> python                                           (kedro-dev) 
Python 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kedro_datasets.biosequence import BioSequenceDataSet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'BioSequenceDataSet' from 'kedro_datasets.biosequence' (/Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/kedro_datasets/biosequence/__init__.py)
```



Discussion:
Last time I saw this I was thinking the same, it turns out to be not trivial because of this. Not sure if things are still the same but worth mentioning.

https://github.com/kedro-org/kedro/blob/38dfe6f5e9d2e8758a732044267769aafb1ff465/kedro/io/core.py#L381-L389
Thanks @noklam - I see that an error saying "some of the dependencies have not been installed" is raised if the desired class is the list of`_DEFAULT_PACKAGES` (which includes `""`, hence allowing the user to type the full import path) is exhausted:

https://github.com/kedro-org/kedro/blob/38dfe6f5e9d2e8758a732044267769aafb1ff465/kedro/io/core.py#L392-L396

I see how we would not like to bubble all `ImportError`s to the user because in most cases, only 1 of those prefixes will have the desired class. Maybe we'd want to bubble up `ModuleNotFoundError` errors, but definitely I'd have to think this through.
@astrojuanlu I think I come to similar conclusion, actually I am not sure why the issue disappear I thought I created it but I couldn't find it... 
I would push for this if it's possible. Definitely one of the most annoying thing and as a beginner you will hit this issue for sure (almost as annoying as the DataSet vs Dataset typo issue)
Updated the title of this issue and adding more evidence:

https://linen-slack.kedro.org/t/14153932/hey-team-i-m-getting-the-following-exception-an-exception-oc#8b6debb1-ed6f-4a16-9440-5a03d0370b07

> Hey team, I'm getting the following exception:
> ```
> An exception occurred when parsing config for dataset 'raw_web@delta':
> Object 'DeltaTableDataSet' cannot be loaded from 'kedro.extras.datasets.spark'. Please see the documentation on how to install relevant dependencies for kedro.extras.datasets.spark.DeltaTableDataSet:
> <https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html>
> ```
> If I go to the link, it doesn't tell me what I should be installing.
> I already have the following in the requirements.txt: kedro-datasets[spark,delta]

Indeed, `https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html` contains a lot and is difficult to parse.

I believe the same thing happened just this morning to @NeroOkwa.

Maybe it would be better if the underlying `ImportError` could be shown.

In any case, this is something that should be changed on Kedro, so I'm moving the issue.
> Remove all the with suppress(ImportError):.

Without reading the full context, are you aware of this is merged already? I think the error is surfaced now already (with kedro-datasets). I haven't fully tested it tho.
- https://github.com/kedro-org/kedro-plugins/pull/277
This is what I see now:

```
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.
```

doesn't tell me whether I made a typo, or what dependency I need to install. Am I missing something? This is with kedro-datasets 1.5.2
Will revisit that when we fixed the dataset installation issue. If this is not the case yet, I think we are in a good position to solve this finally. 
I want to have more clarification of this ticket. Is this a `kedro` or `kedro-datasets` ticket? The description and discussion is slightly outdated because https://github.com/kedro-org/kedro-plugins/pull/277 change how we import library.

The original error mentioned in the description is something in `kedro` rather than `kedro-datasets`
> DatasetError: An exception occurred when parsing config for dataset 'companies':
> Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.

```
from kedro_datasets.biosequence import BioSequenceDataSet
```
> ModuleNotFoundError: No module named 'Bio'

@astrojuanlu Would you be able to give me an fail cases so I can investigate this?

Resolving patch:
diff --git a/kedro/io/core.py b/kedro/io/core.py
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -396,14 +396,16 @@ def parse_dataset_definition(
             )
         class_paths = (prefix + class_obj for prefix in _DEFAULT_PACKAGES)
 
-        trials = (_load_obj(class_path) for class_path in class_paths)
-        try:
-            class_obj = next(obj for obj in trials if obj is not None)
-        except StopIteration as exc:
+        for class_path in class_paths:
+            tmp = _load_obj(class_path)
+            if tmp is not None:
+                class_obj = tmp
+                break
+        else:
             raise DatasetError(
                 f"Class '{class_obj}' not found or one of its dependencies "
                 f"has not been installed."
-            ) from exc
+            )
 
     if not issubclass(class_obj, AbstractDataset):
         raise DatasetError(


Test patch: