Repository: kedro-org/kedro

Issue #2943: Make import failures in `kedro-datasets` clearer
Created: 2023-03-22T15:45:09Z
Description
NoteSee #2943 (comment) for current status

Disambiguate the "module does not exist" error from the ImportError in messages like:
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'polars.CSVDataSet' not found or one of its dependencies has not been installed.

This task will require investigation into how the error messages are swallowed, which will inform the proper implementation. Probably change: 
  
    
      kedro/kedro/io/core.py
    
    
         Line 433
      in
      4194fbd
    
  
  
    

        
          
           def _load_obj(class_path: str) -> object | None: 
        
    
  


Context
This week I've been battling some puzzling documentation errors and after a while I noticed that, if the dependencies of a particular dataset are not present, the ImportError is swallowed silently. Examples:
https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/biosequence/__init__.py#L7-L8
https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/networkx/__init__.py#L8-L15
This was done in McK-Private/private-kedro#575 to solve McK-Private/private-kedro#563 at the same time dependencies were moved to extras_require.
I see how not suppressing these errors could be extremely annoying back then, because kedro.io used to re-export all the datasets in its __init__.py:
https://github.com/quantumblacklabs/private-kedro/blob/f7dd2478aec4de1b46afbaded9bce3c69bff6304/kedro/io/__init__.py#L29-L47
# kedro/io/__init__.py

"""``kedro.io`` provides functionality to read and write to a
number of data sets. At core of the library is ``AbstractDataSet``
which allows implementation of various ``AbstractDataSet``s.
"""

from .cached_dataset import CachedDataSet  # NOQA
from .core import AbstractDataSet  # NOQA
from .core import AbstractVersionedDataSet  # NOQA
from .core import DataSetAlreadyExistsError  # NOQA
from .core import DataSetError  # NOQA
from .core import DataSetNotFoundError  # NOQA
from .core import Version  # NOQA
from .data_catalog import DataCatalog  # NOQA
from .data_catalog_with_default import DataCatalogWithDefault  # NOQA
from .lambda_data_set import LambdaDataSet  # NOQA
from .memory_data_set import MemoryDataSet  # NOQA
from .partitioned_data_set import IncrementalDataSet  # NOQA
from .partitioned_data_set import PartitionedDataSet  # NOQA
from .transformers import AbstractTransformer  # NOQA
However, now our __init__.py is empty and datasets are meant to be imported separately:
https://github.com/kedro-org/kedro-plugins/blob/b8881d113f8082ff03e0233db3ae4557a4c32547/kedro-datasets/kedro_datasets/__init__.py#L1-L3
So I think it would be much better if we did not silence those import errors.
More context
If one dependency is missing, the user would get an unhelpful "module X has no attribute Y" when trying to import a dataset rather than an actual error:
> pip uninstall biopython                          (kedro-dev) 
Found existing installation: biopython 1.81
Uninstalling biopython-1.81:
  Would remove:
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/Bio/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/BioSQL/*
    /Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/biopython-1.81.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled biopython-1.81
> python                                           (kedro-dev) 
Python 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from kedro_datasets.biosequence import BioSequenceDataSet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'BioSequenceDataSet' from 'kedro_datasets.biosequence' (/Users/juan_cano/.micromamba/envs/kedro-dev/lib/python3.10/site-packages/kedro_datasets/biosequence/__init__.py)

Duplicate issue #3911: [DataCatalog]: Error message is confusing when kedro-dataset is not installed
Created: 2024-06-03T11:00:26Z
Description
When kedro-datasets is not installed the error message one gets is not informative.
We propose enhancing the error message to provide a clear message on the root cause of the failure - when dataset dependencies are missing.
Relates to #2401
Context
Currently, users are required to install all dependencies even for unused datasets (in case you want to run pipeline partially or do not load some datasets when standalone catalog usage). The error message generated when some datasets are not installed is unclear, making it difficult for users to understand why the pipeline fails.
Example of the current error message:
DatasetError: An exception occurred when parsing config for dataset 'companies':
Class 'pandas.CSVDataset' not found, is this a typo?

This error occurs during dataset configuration parsing and lacks clarity, not suggesting the straightforward solution of installing the necessary package. This can lead to confusion and delays as users may not immediately realize that the issue is due to missing software rather than a typo in their configuration.

Closing comment (2024-06-06T06:58:45Z): Related to #2943