Package: tdfs4ds
tdfs4ds : A Feature Store Library for Data Scientists working with Clearscape AnalyticsThe tdfs library is a Python package designed for managing and utilizing Feature Stores in a Teradata Database. With a set of easy-to-use functions, tdfs enables the efficient creation, registration, and storage of features. It also simplifies the process of preparing feature data for ingestion, building datasets for data analysis, and obtaining already existing features.Getting StartedInstall the tdfs package via pip:pipinstalltdfs4dsTo utilize the functionality of the tdfs4ds package, import it in your Python script:importtdfs4dsIt is recommended to import it after creating the context with teradataml in order to use the connection parameters to get the feature store database as the default database. Otherwise you can specify it as follows:tdfs4ds.SCHEMA=<yourdatabase>Getting startedThetdfs4dspackage aims to be very simple and straightforward to start your feature store in a Vantage system and especially in your datalab. To start, you only need to master a restricted number of functions.tdfs4ds.setup(database, if_exists='fail'): Creates a feature catalog table and a process catalog rable in the Teradata database you specify.tdfs4ds.upload_features(df, entity_id, feature_names, metadata = {}):: Ingests the features calculated with the teradata dataframe df. You have to specify the entity_id, meaning the columns that define the unique ID in the result set. You have to specify the data type. Hence the entity_id is a dictionary with column name as key, and data type as value. e.g. {'ID': 'BIGINT'}. feature_names is the list of column name corresponding to the features you want to ingest in the feature store. Do not hesitate to use the metadata argument to document your features.tdfs4ds.build_dataset(entity_id, selected_features, view_name,comment = 'dataset'): Create a dataset view from the feature store. entity_id is the list of column names defining the entity, selected_features is the dictionary with feature names as key and feature version (meaning the process id that calculates these features) are values. The view_name is the name of the dataset view we want. Do not hesitate to use the comment argument to comment the view in the database.These three functions is the core of the package. They manage for you the registering of entities or features when needed. They also manage the process catalog to simplify the operationalization of your feature engineering process.I forgot to mention that the feature catalog, process catalog and feature stores are all temporal. Meaning that you can time travel by changing thetdfs4ds.FEATURE_STORE_TIMEvariable (the format is '9999-01-01 00:00:00' or None for current time).Finally, it manages data_domain to avoid conflicts in feature and entity names across multiple use cases. The active data domain is stored intdfs4ds.DATA_DOMAIN.ExampleImagine you already have a feature engineering process implemented as a view in Teradata Vantage. If you do not have a feature store, the first step is to set up one:Step 1: setup a feature storeAfter having created a context with teradataml, just type:importtdfs4dstdfs4ds.setup(database=my_database)this will create for you the feature and process catalogs in the database named my_database.Step 2: connect to your feature storeNow we speficy the active database and the data domain our feature is dealing with.tdfs4ds.DATA_DOMAIN='DATA_QUALITY'tdfs4ds.SCHEMA=my_databaseThese two parameters are initialized with the default database when the tdfs4ds package is imported after the create_context call of teradataml that establish the connection with Vantage.Step 3: feature engineeringIn Vantage, almost any feature engineering process, being in SQL or involving external languages or engines can be implemented in a SQL view. A view can be handled as a teradata dataframe (a cousin of pandas dataframe).df=tdml.DataFrame(tdml.in_schema(my_database,my_view))assuming tdml is the alias of the teradataml import, and my_view is the view that implements the feature engineering process. If you apply additional transformation with teradataml, please usetdfs4ds.utils.lineage.crystallize_viewto make permanent the views generated by teradataml.Step 4: upload & operationaliseWe define among the output columns of the view the columns dealing with the entity description, and the features to be ingested. We also add some metadata to describe our project.fromtdfs4ds.feature_storeimportupload_features# Specify the entity ids in the view columns (+ data types)entity_id=['EVENT_DT','ID']# Specify the columns that contains the actual featuresfeature_names=['KPI1','KPI2]# attach informative metadata to document your processmetadata={'project':'data quality'}# upload & operationaliseupload_features(df=df,entity_id=entity_id,feature_names=feature_names,metadata={'project':'data quality'})Here we go! This command will register the entities, register the features in the data domain if they are not yet registered, register a feature engineering process in the process catalog and register the features in the feature store, maintaining the lineage.This function also outputs a teradata dataframe corresponding to the dataset you have just registered, i.e. the results of my_view.You also get a process id. This process id can also be retrieved with:fromtdfs4ds.process_store.process_query_administrationimportlist_processeslist_processes()You can also run the process again and ingest new features in the feature store by using this process id as follows:fromtdfs4dsimportrunrun(process_id)So no need to get the code that builds the feature engineering process, the process id is all what you need.No worries if you are computing features that are already present in the feature store, the feature store is temporal so it will avoid any duplication of the feature and version the feature values when needed.Building a new dataset with existing featuresNow your feature store is populated, you can build any dataset knowing the entity_ids, the features and the process_id (or feature version) of your choice.fromtdfs4dsimportbuild_datasetmydataset=build_dataset(entity_id=['customer_id'],selected_features=selected_features,view_name='mydataset',comment='dataset for CHURN')selected_feature is a dictionary where keys are the feature name, and value the process id corresponding to the process used for the computation of the feature value.if you do not know the registered entities, features and corresponding feature versions you can use the functionsget_list_entity,get_list_features,get_available_featuresandget_feature_versionsintdfs4ds.feature_store.feature_query_retrieval.Here is the structure of the package:.tdfs4ds
    ├── datasets.py
    │   └── Function: outstanding_amounts_dataset
    │   └── Function: upload_outstanding_amounts_dataset
    ├── __init__.py
    │   └── Function: build_dataset
    │   └── Function: build_dataset_time_series
    │   └── Function: roll_out
    │   └── Function: run
    │   └── Function: setup
    │   └── Function: upload_features
    │   └── Function: upload_tdstone2_scores
    └── data
    └── feature_store
        ├── entity_management.py
        │   └── Function: register_entity
        │   └── Function: tdstone2_entity_id
        │   └── Function: remove_entity        
        ├── feature_data_processing.py
        │   └── Function: prepare_feature_ingestion
        │   └── Function: prepare_feature_ingestion_tdstone2
        │   └── Function: store_feature
        ├── feature_query_retrieval.py
        │   └── Function: get_available_features
        │   └── Function: get_entity_tables
        │   └── Function: get_feature_store_content
        │   └── Function: get_feature_store_table_name
        │   └── Function: get_feature_versions
        │   └── Function: get_list_entity
        │   └── Function: get_list_features
        │   └── Function: list_features
        ├── feature_store_management.py
        │   └── Function: GetAlreadyExistingFeatureNames
        │   └── Function: GetTheLargestFeatureID
        │   └── Function: Gettdtypes
        │   └── Function: delete_feature
        │   └── Function: feature_store_catalog_creation
        │   └── Function: feature_store_table_creation
        │   └── Function: register_features
        │   └── Function: remove_feature
        │   └── Function: tdstone2_Gettdtypes
        ├── __init__.py
    └── process_store
        ├── process_query_administration.py
        │   └── Function: get_process_id
        │   └── Function: list_processes
        │   └── Function: remove_process
        ├── process_registration_management.py
        │   └── Function: register_process_tdstone
        │   └── Function: register_process_view
        ├── process_store_catalog_management.py
        │   └── Function: process_store_catalog_creation
        ├── __init__.py
    └── utils
        ├── info.py
        │   └── Function: get_column_types
        │   └── Function: get_column_types_simple
        ├── lineage.py
        │   └── Function: _analyze_sql_query
        │   └── Function: analyze_sql_query
        │   └── Function: crystallize_view
        │   └── Function: generate_view_dependency_network
        │   └── Function: generate_view_dependency_network_fs
        │   └── Function: get_ddl
        ├── query_management.py
        │   └── Function: execute_query
        │   └── Function: execute_query_wrapper
        │   └── Function: is_version_greater_than
        ├── time_management.py
        │   └── Class: TimeManager
        ├── visualization.py
        │   └── Function: display_table
        │   └── Function: linear_depth_layout
        │   └── Function: plot_graph
        │   └── Function: prepare_plotly_traces
        │   └── Function: radial_layout
        │   └── Function: segmented_linear_layout
        │   └── Function: visualize_graph
        ├── __init__.py