Repository: kedro-org/kedro

Issue #400: Allow new attributes to be added to DataSets
Created: 2020-06-02T15:30:27Z
Description
I have certain attributes to track within my datasets and have created custom DataSets to get around this issue.  Now that hooks are out most of my reasons for custom DataSets are gone, and I can achieve the same thing with an after_node_run hook, but I still cannot attach custom attributes to datasets.
Use Case 1 (can I share this dataset)
I would like to attach things like confidentiality to the dataset so that team members can easily know who they can share a dataset with by looking at an attribute on the dataset.  Ideally, I would like to add these to the catalog.
Use Case 2 (can I delete this sub_pipeliene)
I would also like to be able to check the pipeline health in CI,  one thing that I would like to look for is dangling edges that are useless.  Sometimes during refactoring we switch to a new section of the pipeline, the old one gets disconnected, never removed, and now we wonder if anyone is using that output.  It would have been nice to have CI tell us that we need to mark that dataset as a final output or remove the section of pipeline.
Possible Implementation
cars:
  type: pandas.CSVDataSet
  filepath: data/01_raw/company/cars.csv
  attributes: # 👈 this is the proposed feature, not currently in the framework
    is_output: true
    confidentiality: public
The AbstractDataset's would need to accept the attributes keyword, then attach the attributes to each instance.

Duplicate issue #324: Optional tags for DataSets and accessing through the catalog
Created: 2020-04-13T01:59:08Z
Description
It would be nice to be able to add optional tags to datasets in the catalog, and list/access them via tags, similar to what is currently implemented in node
Context
The DataSet/Catalog system and the python interface are awesome. As the number of datasets defined in the catalog grows it would be helpful to be able to list/access specific subsets based on common tags.
Possible Implementation
I'm imagining adding an optional tags attribute in the catalog yml like so:
...
bikes:
  type: pandas.CSVDataSet
  filepath: data/01_raw/bikes.csv
  tags: 
    - transportation
    - has_wheels

trains:
  type: pandas.CSVDataSet
  filepath: data/01_raw/bikes.csv
  tags: 
    - transportation
...
The the ability to view them in the data calog from python based on the tags where catalog.list might have the following api:
catalog.list(tags=['transportation'])
which might return
['bikes', 'trains', ...]
If I wanted to do something specifically with just transportation data I can easily loop through this list and load these datasources with catalog.load without having to explicitly list them in my python code. This helps keep the catalog as the one source of truth.
The following simple (but hacky) modifications are working so far. When a DataSet is created there could be an instance variable _tags and then the list method in DataCatalog could look like this:
def list(self, tags: List[str] = None) -> List[str]:
    """List of ``DataSet`` names registered in the catalog.
    Args:
        tags: list of tags that will subset the list of returned datasets
            to those which match on at least one tag.
            
    Returns:
        A List of ``DataSet`` names, corresponding to the entries that are
        registered in the current catalog object.
    """
    if tags is not None:
        matched_data_sets = []
        for key, ds in self._data_sets.items():
            if set(tags).intersection(set(ds._tags)):
                matched_data_sets.append(key)
        
        return matched_data_sets

    return list(self._data_sets.keys())
And the following modification to AbstractDataSet is not ideal but seems to work:
    @classmethod
    def from_config(
        cls: Type,
        name: str,
        config: Dict[str, Any],
        load_version: str = None,
        save_version: str = None,
    ) -> "AbstractDataSet":
        """Create a data set instance using the configuration provided.

        Args:
            name: Data set name.
            config: Data set config dictionary.
            load_version: Version string to be used for ``load`` operation if
                the data set is versioned. Has no effect on the data set
                if versioning was not enabled.
            save_version: Version string to be used for ``save`` operation if
                the data set is versioned. Has no effect on the data set
                if versioning was not enabled.

        Returns:
            An instance of an ``AbstractDataSet`` subclass.

        Raises:
            DataSetError: When the function fails to create the data set
                from its config.

        """
        try:
            class_obj, config = parse_dataset_definition(
                config, load_version, save_version
            )
        except Exception as ex:
            raise DataSetError(
                "An exception occurred when parsing config "
                "for DataSet `{}`:\n{}".format(name, str(ex))
            )

        tags = None
        if 'tags' in list(config.keys()):
            tags = config['tags']
            del config['tags']

        try:
            data_set = class_obj(**config)  # type: ignore
        except TypeError as err:
            raise DataSetError(
                "\n{}.\nDataSet '{}' must only contain "
                "arguments valid for the constructor "
                "of `{}.{}`.".format(
                    str(err), name, class_obj.__module__, class_obj.__qualname__
                )
            )
        except Exception as err:
            raise DataSetError(
                "\n{}.\nFailed to instantiate DataSet "
                "'{}' of type `{}.{}`.".format(
                    str(err), name, class_obj.__module__, class_obj.__qualname__
                )
            )
        
        if tags is not None:
            data_set._tags = tags

        return data_set
Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.

Closing comment (2020-10-05T14:28:12Z): Closing this as duplicate of #400