Package: project-data-catalog
Data CatalogA catalog to define, create, store, and access datasets.This python package aims to streamline data engineering and data analysis during data science projects:organize all datasets used or created by your project,define datasets as a transformation of others,easily propagate updates when datasets are updated,avoid boilerplate code,access datasets from anywhere in your code, without having to remember file paths,share dataset definitions within a project team,document datasets,and enable smooth transitions from exploration to deployment.Many data cataloging python packages exist (kedro, prefect, ...) for slightly different use cases. This package is tailored for managing datasets during data science projects. The emphasis is on minimal boilerplate, easy data access, and no-effort updates.InstallationUse a python environment compatible with this project, e.g. with conda:conda create -n my_env python "pandas>=0.19" "dask>=0.2.0" "s3fs>=0.2.0" "pytz>=2011k" pytest pyarrowInstall this package:pip install git+https://github.com/numerical-io/data_catalog.git@mainExample 1: a catalog of datasetsA data catalog is defined in a python module or package. The following catalog defines three classes, each representing a dataset:DatasetA, defined by code (createfunction),DatasetB, defined as a transformation of DatasetA (createfunction andparentsattribute),DatasetC, defined and read from a CSV file (relative_pathattribute).Each class inherits from a class defining the data format on disk. This example uses CSV and parquet files.# example_catalog.pyimportpandasaspdfromdata_catalog.datasetsimportCsvDataset,ParquetDatasetclassDatasetA(CsvDataset):"""A dataset defined in code, and saved as a CSV file."""defcreate(self):df_a=pd.DataFrame({"col1":[1,2,3]})returndf_aread_kwargs={"index_col":0}classDatasetB(ParquetDataset):"""A dataset defined from another dataset, and saved as a Parquet file."""parents=[DatasetA]defcreate(self,df_a):df_b=2*df_areturndf_bclassDatasetC(CsvDataset):"""A dataset defined in a CSV file."""relative_path="dataset_c.csv"This catalog definition contains all that is needed. The datasets defined in code can be generated by a succession of tasks, that we encode in a task graph. The task graph follows the Dask DAG format, and we execute it with Dask. (The graph itself is otherwise independent from Dask, and could be run by an engine of your choice.)fromdata_catalog.taskgraphimportcreate_task_graphfromdask.threadedimportgetfromexample_catalogimportDatasetA,DatasetB# Define a context. The context is necessary to instanciate datasets.# It contains an URI indicating where to save all the catalog's datasets.context={"catalog_uri":"file:///path/to/data/folder"}# Generate a task graph to create datasets, resolving dependencies between them.datasets=[DatasetA,DatasetB]# leave out DatasetC unless you provide a file dataset_c.csvtaskgraph,targets=create_task_graph(datasets,context)# Let Dask generate all datasets on disk_=get(taskgraph,targets)Once the files are created, you can access datasets from anywhere in your project.dataset_b=DatasetB(context)# Load into a dataframedf=dataset_b.read()# View its descriptiondataset_b.description()# View the file pathdataset_b.path()Example 2: a catalog with collections of datasetsSometimes data is available as a collection of identically formatted files. Collections of datasets are available to handle this case.Collections can be defined in a catalog as follows:# example_catalog.pyimportpandasaspdfromdata_catalog.datasetsimportParquetDatasetfromdata_catalog.collectionsimportFileCollection,same_key_inclassCollectionA(FileCollection):"""A collection of datasets saved as Parquet files."""defkeys(self):return["file_1","file_2","file_3"]classItem(ParquetDataset):defcreate(self):df=pd.DataFrame({"col1":[1,2,3]})returndfclassCollectionB(FileCollection):"""A collection defined from CollectionA.Each item corresponds to one item in CollectionA."""defkeys(self):return["file_1","file_2","file_3"]classItem(ParquetDataset):parents=[same_key_in(CollectionA)]defcreate(self,df):return2*dfclassDatasetD(ParquetDataset):"""A dataset concatenating all items from CollectionA."""parents=[CollectionA]defcreate(self,collection_a):df=pd.concat(collection_a)returndfThe generation of files is identical as in the previous example:fromdata_catalog.taskgraphimportcreate_task_graphfromdask.threadedimportgetfromexample_catalogimportCollectionA,CollectionB,DatasetD# Define the catalog's contextcontext={"catalog_uri":"file:///path/to/data/folder"}# Generate the task graph and run it with Dasktaskgraph,targets=create_task_graph([CollectionA,CollectionB,DatasetD],context)_=get(taskgraph,targets)You can then access data anywhere in your project.# Load a collectionCollectionA(context).read()# Get a single dataset from a collection.item_2=CollectionA.get("file_2")# item_2 is a usual dataset objectdf=item_2(context).read()The task graph only includes necessary updates. If all files exist and parents have older update times than their children, no task will be executed. If however you modify a file, the task graph will contain tasks to update all its descendants. When modifying the code of a dataset, remove the corresponding file to trigger its re-creation, and the updates of all its descendants.Dataset attributesWhen defining a dataset class, you can set the following attributes:parents: A list of dataset/collection classes from which this dataset is defined.create: A method to create the dataset. It takes as inputs, aside fromself, the data loaded from all classes inparents. The number of input arguments (not countingself) must therefore be equal to the length ofparents. The method must return the created data.relative_path: The file path, relative to the catalog URI.file_extension: The file extension.is_binary_file: A boolean indicating whether the file is a text or binary file.read_kwargs: A dict of keyword arguments for reading the dataset.write_kwargs: A dict of keyword arguments for writing the dataset.All these attributes are optional, and have default values if omitted.When relative_path is missing, it is inferred from the class name and path in the package. For instance, a CSV datasetSomeDatasetdefined in the submoduleexample_catalog.part_onewill have a relative path set topart_one/SomeDataset.csv.If a docstring is set, it becomes the dataset description available through thedescription()method.Datasets must inherit from a subclass ofAbstractDataset. The data catalog provides a few such classes for common cases:CsvDataset,ParquetDataset,PickleDataset,ExcelDataset, andYamlDataset.Collection attributesA collection can have the following attributes:Item: A nested class defining a dataset in the collection. It is a template for each item in the collection.keys: A method returning a list of keys. Each key maps to a collection item. Files in the collection are named after keys, and conversely.relative_path: If set, this path refers to the directory containing collection data files. This value is used to define therelative_pathfor eachItem.Collections inherit fromFileCollection.Collection have a class methodgetthat returns dataset classes for given keys.Managing the catalogThe data files reside at the URI set in thecontextvariable, used for instanciating all objects. The catalog supports, as of now, URI's pointing to local files (file://) or to S3 (s3://). Note that the catalog itself is defined independently of its location; only data instances are dependent on the context. This facilitates the creation of several copies, e.g. for sharing between different users or versioning datasets.To view all datasets and collections defined in a catalog, use the following functions:fromdata_catalog.utilsimportdescribe_catalog,list_catalogimportexample_catalog# Get dataset names and descriptions, in a dict.describe_catalog(example_catalog)# List all classes representing datasets and collections.# If example_catalog is a package, the list will contain# classes from all _imported_ submodules.list_catalog(example_catalog)When running the task graph, each task logs messages to a logger named data_catalog. This logger configuration will show messages on sys.stderr:importlogginglogger=logging.getLogger("data_catalog")logger.setLevel(logging.INFO)log_formatter=logging.Formatter("%(asctime)s-%(levelname)s-%(message)s")log_handler=logging.StreamHandler()log_handler.setLevel(logging.INFO)log_handler.setFormatter(log_formatter)logger.addHandler(log_handler)