Repository: Teradata/feast-teradata (language: python)

Issue:
publish package

Pytest (Testing feast using framework)

Pytest (Testing feast using framework)

Pytest (Testing feast using framework)

Pytest (Testing feast using framework)



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/cd-release.yml b/.github/workflows/cd-release.yml
new file mode 100644
index 0000000..b7f70b0
--- /dev/null
+++ b/.github/workflows/cd-release.yml
@@ -0,0 +1,53 @@
+name: CD Release
+on:
+  workflow_dispatch:
+    inputs:
+      releaseVersion:
+        description: 'Semantic version of the release'
+        required: true
+
+jobs:
+  cdRelease:
+    runs-on: ubuntu-latest
+    name: Release
+    steps:
+      - uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements_dev.txt
+      - name: Bump version
+        shell: bash
+        run: |
+          git checkout -b releases/$RELEASE_VERSION
+          git config --global user.email "developers@teradata.com"
+          git config --global user.name "Teradata GitHub Actions"
+          bumpversion --tag --commit --new-version $RELEASE_VERSION num
+          git push --set-upstream origin releases/$RELEASE_VERSION
+          git push --tags
+        env:
+          RELEASE_VERSION: ${{ github.event.inputs.releaseVersion }}
+      - name: Upload to PyPi
+        shell: bash
+        run: |
+          rm -fr dist/*
+          python setup.py sdist bdist_wheel
+          twine upload dist/*
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      - name: Create Release
+        id: create_release
+        uses: actions/create-release@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
+        with:
+          tag_name: v${{ github.event.inputs.releaseVersion }}
+          release_name: feast-teradata v${{ github.event.inputs.releaseVersion }}
+          body_path: ./CHANGELOG.md
+          draft: false
+          prerelease: false
diff --git a/LICENSE.txt b/LICENSE
similarity index 100%
rename from LICENSE.txt
rename to LICENSE
diff --git a/README.md b/README.md
index 18cef98..30a37a3 100644
--- a/README.md
+++ b/README.md
@@ -1,225 +1,47 @@
 # Feast Teradata Connector
-[![test-offline-store](https://github.com/feast-dev/feast-custom-offline-store-demo/actions/workflows/test_custom_offline_store.yml/badge.svg?branch=main)](https://github.com/feast-dev/feast-custom-offline-store-demo/actions/workflows/test_custom_offline_store.yml)
+[![feast-teradata tests](https://github.com/Teradata/feast-teradata/actions/workflows/ci-integeration-tests.yml/badge.svg?branch=master)](https://github.com/Teradata/feast-teradata/actions/workflows/ci-integeration-tests.yml)
 
-## Offline Store
+## Overview
 
-### Overview
+We recommend you familiarize yourself with the terminology and concepts of feast by reading the official [feast documentation](https://docs.feast.dev/). 
 
-This document demonstrates how developers can integrate `Teradata's offline store` for Feast.
-Teradata's offline stores allow users to use any underlying data store as their offline feature store. Features can be retrieved from the offline store for model training, and can be materialized into the online feature store for use during model inference. 
+The `feast-teradata` library adds support for Teradata as 
+- OfflineStore 
+- OnlineStore
 
+Additional, the using Teradata as the registry is already supported via the `registry_type: sql` and included in our examples. This means that everything is located in Teradata. However, depending on the requirements, installation, etc, this can be mixed and matched with other systems as appropriate.  
 
-### Why create an offline store?
+## Getting Started
 
-Feast uses an offline store as the source of truth for features. These features can be retrieved from the offline store for model training. Typically, scalable data warehouses are used for this purpose.
- 
-Feast also materializes features from offline stores to an online store for low-latency lookup at model inference time. 
-
-Feast comes with some offline stores built in, e.g, Parquet file, Redshift and Bigquery. However, users can develop their own offline stores by creating a class that implements the contract in the [OfflineStore class](https://github.com/feast-dev/feast/blob/5e61a6f17c3b52f20b449214a4bb56bafa5cfcbc/sdk/python/feast/infra/offline_stores/offline_store.py#L41).
-
-### How to configure feature_store.yaml?
-
-```yaml
-project: <name of project>
-registry: <registry>
-provider: local
-offline_store:
-   type: feast_teradata.offline.teradata.TeradataOfflineStore
-   host: <db host>
-   database: <db name>
-   user: <username>
-   password: <password>
-   log_mech: <connection mechanism>
-```
-
-### Example Feature Definitions
-
-```python
-import yaml
-import pandas as pd
-from datetime import timedelta
-from feast import Entity, Field, FeatureView,FeatureService, PushSource,RequestSource
-from feast.on_demand_feature_view import on_demand_feature_view
-from feast.types import Float64, Int64, Float32
-from feast_teradata.offline.teradata_source import TeradataSource
-
-driver = Entity(name="driver", join_keys=["driver_id"])
-project_name = yaml.safe_load(open("feature_store.yaml"))["project"]
-
-driver_stats_source = TeradataSource(
-    database=yaml.safe_load(open("feature_store.yaml"))["offline_store"]["database"],
-    table=f"{project_name}_feast_driver_hourly_stats",
-    timestamp_field="event_timestamp",
-    created_timestamp_column="created",
-)
-
-driver_stats_fv = FeatureView(
-    name="driver_hourly_stats",
-    entities=[driver],
-    ttl=timedelta(weeks=52 * 10),
-    schema=[
-        Field(name="driver_id", dtype=Int64),
-        Field(name="conv_rate", dtype=Float32),
-        Field(name="acc_rate", dtype=Float32),
-        Field(name="avg_daily_trips", dtype=Int64),
-    ],
-    source=driver_stats_source,
-    tags={"team": "driver_performance"},
-)
+To get started, install the `feast-teradata` library
 
+```bash
+pip install feast-teradata
 ```
 
-Now to explain the different components:
-
-* `TeradataSource`:  Data Source for features stored in Teradata (Enterprise or Lake) or accessible via a Foreign Table from Teradata (NOS, QueryGrid)
-* `Entity`: A collection of semantically related features
-* `Feature View`: A feature view is a group of feature data from a specific data source. Feature views allow you to consistently define features and their data sources, enabling the reuse of feature groups across a project
-
-### Example OfflineStore usage
-
-There are two different ways to test your offline store as explained below. 
-But first there are a few mandatory steps to follow:
+Let's create a simple feast setup with Teradata using the standard drivers dataset. Note that you cannot use `feast init` as this command only works for templates which are part of the core feast library. We intend on getting this library merged into feast core eventually but for now, you will need to use the following cli command for this specific task. All other `feast` cli commands work as expected. 
 
-Importing the relevant entities and feature views
-```python
-from <path_to_repository.py> import <name_of_entity>, <name_of_feature_view>
+```bash
+feast-td init-repo
 ```
 
-Initialize the feature store:
-```python
-fs = FeatureStore("<path_to_feature_repository>/")
-```
+This will then prompt you for the required information for the Teradata system and upload the example dataset. Let's assume you used the repo name `demo` when running the above command. You can find the repository files along with a file called `test_workflow.py`. Running this `test_workflow.py` will execute a complete workflow for feast with Teradata as the Registry, OfflineStore and OnlineStore. 
 
-Applying the feature view and entity on the feature store
-```python
-fs.apply([<name_of_entity>, <name_of_feature_view>])
 ```
-
-
-```python
-        entity_sql = f"""
-            SELECT
-                "driver_id",
-                "event_timestamp"
-            FROM {fs.get_data_source(table_name).get_table_query_string()}
-            WHERE "event_timestamp" BETWEEN '{start_date}' AND '{end_date}'
-        """
-
-    training_df = fs.get_historical_features(
-        entity_df=entity_sql,
-        features=[
-            "driver_hourly_stats:conv_rate",
-            "driver_hourly_stats:acc_rate",
-            "driver_hourly_stats:avg_daily_trips",
-        ],
-    ).to_df()
+demo/
+    feature_repo/
+        driver_repo.py
+        feature_store.yml
+    test_workflow.py
 ```
 
-## Online Store
-
-### Overview
-
-This repository demonstrates how developers can create their own custom online stores for Feast. 
-Custom online stores allow users to use any underlying data store to store features for low-latency retrieval, typically needed during model inference.
-
-### Why create an online store?
-
-Feast materializes data to online stores for low-latency lookup at model inference time. Typically, key-value stores are used for the online stores, however relational databases can be used for this purpose as well.
-
-Users can develop their own online stores by creating a class that implements the contract in the OnlineStore class.
-
-### How to configure feature_store.yaml?
-
-```yaml
-project: <name of project>
-registry: <registry>
-provider: local
-offline_store:
-   type: feast_teradata.offline.teradata.TeradataOfflineStore
-   host: <db host>
-   database: <db name>
-   user: <username>
-   password: <password>
-   log_mech: <connection mechanism>
-```
-
-### Example OnlineStore usage
-
-There are a few mandatory steps to follow before we can test the online store:
-
-Importing the relevant entities and feature views
-```python
-from <path_to_repository.py> import <name_of_entity>, <name_of_feature_view>
-```
-
-Initialize the feature store:
-```python
-fs = FeatureStore("<path_to_feature_repository>/")
-```
-
-Applying the feature view and entity on the feature store
-```python
-fs.apply([<name_of_entity>, <name_of_feature_view>])
-```
-
-```python
-def fetch_online_features(store, source: str = ""):
-    entity_rows = [
-        # {join_key: entity_value}
-        {
-            "driver_id": 1001,
-            "val_to_add": 1000,
-            "val_to_add_2": 2000,
-        },
-        {
-            "driver_id": 1002,
-            "val_to_add": 1001,
-            "val_to_add_2": 2002,
-        },
-    ]
-    if source == "feature_service":
-        features_to_fetch = store.get_feature_service("driver_activity_v1")
-    elif source == "push":
-        features_to_fetch = store.get_feature_service("driver_activity_v3")
-    else:
-        features_to_fetch = [
-            "driver_hourly_stats:acc_rate",
-            "transformed_conv_rate:conv_rate_plus_val1",
-            "transformed_conv_rate:conv_rate_plus_val2",
-        ]
-    returned_features = store.get_online_features(
-        features=features_to_fetch,
-        entity_rows=entity_rows,
-    ).to_dict()
-    for key, value in sorted(returned_features.items()):
-        print(key, " : ", value)
-```
-
-The command below is used to incrementally materialize features in the online store. 
-If there are no new features to be added, this command will essentially not be doing
-anything. With feast `materialize_incremental`, the start time is either now — ttl 
-(the ttl that we defined in our feature views) or the time of the most recent 
-materialization. If you’ve materialized features at least once, then subsequent 
-materializations will only fetch features that weren’t present in the store at 
-the time of the previous materializations
-
-```python
-fs.materialize_incremental(end_date=datetime.now())
+```bash
+python demo/test_workflow.py
 ```
 
-Next, while fetching the online features, we have two parameters `features` and
-`entity_rows`. The `features` parameter is a list and can take any number of features
-that are present in the `df_feature_view`. The example above shows all 4 features present
-but these can be less than 4 as well. Secondly, the `entity_rows` parameter is also
-a list and takes a dictionary of the form {feature_identifier_column: value_to_be_fetched}.
-In our case, the column `driver_id` is used to uniquely identify the different rows
-of the entity driver. We are currently fetching values of the features where driver_id
-is equal to 5. We can also fetch multiple such rows using the format: 
-`[{driver_id: val_1}, {driver_id: val_2}, .., {driver_id: val_n}]`
-`[{driver_id: val_1}, {driver_id: val_2}, .., {driver_id: val_n}]`
 
 ## Release Notes
 
-### 0.1
+### 1.0.0
 
 - Feature: Initial implementation of feast-teradata library
diff --git a/feast b/feast
index 9d5c0af..5ca1dd8 160000
--- a/feast
+++ b/feast
@@ -1,1 +1,1 @@
-Subproject commit 9d5c0afbb07369ecc6062ab2c1a0ccab24f78e8d
+Subproject commit 5ca1dd88e6c8fc7404313521370f155a7739154e
diff --git a/feast_teradata/cli/__init__.py b/feast_teradata/cli/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/create_repo.py b/feast_teradata/cli/cli.py
similarity index 80%
rename from create_repo.py
rename to feast_teradata/cli/cli.py
index 6160408..9bc3252 100644
--- a/create_repo.py
+++ b/feast_teradata/cli/cli.py
@@ -1,7 +1,6 @@
 import sys
 import click
 import re
-import os
 
 from importlib.abc import Loader
 from importlib.machinery import ModuleSpec
@@ -14,11 +13,17 @@ def is_valid_name(name: str) -> bool:
     return not name.startswith("_") and re.compile(r"\W+").search(name) is None
 
 
+@click.group()
+def cli():
+    pass
+
+
+@cli.command()
 def init_repo():
     import os
+    import importlib.util
     from distutils.dir_util import copy_tree
     from pathlib import Path
-
     from colorama import Fore, Style
 
     # check if running in interactive mode or via CI
@@ -31,8 +36,6 @@ def init_repo():
     else:
         repo_name = os.environ.get("CI_FEAST_REPO_NAME")
 
-    print(repo_name)
-
     if not is_valid_name(repo_name):
         raise BadParameter(
             message="Name should be alphanumeric values and underscores but not start with an underscore",
@@ -54,21 +57,20 @@ def init_repo():
 
     # Copy template directory
     template_path = str(Path(Path(__file__).parent / "template" / template).absolute())
-    # x
     copy_tree(template_path, str(repo_path))
 
     # Seed the repository
     bootstrap_path = repo_path / "bootstrap.py"
-    if os.path.exists(bootstrap_path):
-        import importlib.util
+    if not os.path.exists(bootstrap_path):
+        raise Exception(f"boostrap.py not found at {bootstrap_path}")
 
-        spec = importlib.util.spec_from_file_location("bootstrap", str(bootstrap_path))
-        assert isinstance(spec, ModuleSpec)
-        bootstrap = importlib.util.module_from_spec(spec)
-        assert isinstance(spec.loader, Loader)
-        spec.loader.exec_module(bootstrap)
-        bootstrap.bootstrap()  # type: ignore
-        os.remove(bootstrap_path)
+    spec = importlib.util.spec_from_file_location("bootstrap", str(bootstrap_path))
+    assert isinstance(spec, ModuleSpec)
+    bootstrap = importlib.util.module_from_spec(spec)
+    assert isinstance(spec.loader, Loader)
+    spec.loader.exec_module(bootstrap)
+    bootstrap.bootstrap()  # type: ignore
+    os.remove(bootstrap_path)
 
     # template the feature_store.yaml file
     feature_store_yaml_path = repo_path / "feature_repo" / "feature_store.yaml"
@@ -87,4 +89,6 @@ def init_repo():
     click.echo()
 
 
-init_repo()
+if __name__ == '__main__':
+    cli()
+
diff --git a/template/teradata/bootstrap.py b/feast_teradata/cli/template/teradata/bootstrap.py
similarity index 98%
rename from template/teradata/bootstrap.py
rename to feast_teradata/cli/template/teradata/bootstrap.py
index 8e20270..7fc9b6c 100644
--- a/template/teradata/bootstrap.py
+++ b/feast_teradata/cli/template/teradata/bootstrap.py
@@ -2,7 +2,6 @@
 import os
 
 from feast.file_utils import replace_str_in_file
-
 from teradataml import (
     create_context,
     get_context
@@ -75,7 +74,3 @@ def bootstrap():
         with teradata_conn as conn:
             driver_df.to_sql(name=f"{project_name}_feast_driver_hourly_stats", con=conn, if_exists="replace",
                              index=False)
-
-
-if __name__ == "__main__":
-    bootstrap()
\ No newline at end of file
diff --git a/template/teradata/feature_repo/driver_repo.py b/feast_teradata/cli/template/teradata/feature_repo/driver_repo.py
similarity index 100%
rename from template/teradata/feature_repo/driver_repo.py
rename to feast_teradata/cli/template/teradata/feature_repo/driver_repo.py
diff --git a/template/teradata/feature_repo/feature_store.yaml b/feast_teradata/cli/template/teradata/feature_repo/feature_store.yaml
similarity index 100%
rename from template/teradata/feature_repo/feature_store.yaml
rename to feast_teradata/cli/template/teradata/feature_repo/feature_store.yaml
diff --git a/setup.py b/setup.py
index 9041ed1..8489336 100644
--- a/setup.py
+++ b/setup.py
@@ -3,15 +3,29 @@
 with open('requirements.txt') as f:
     required = f.read().splitlines()
 
+with open("README.md") as f:
+    long_description = f.read()
+
 setup(
     name="feast-teradata",
-    version="0.1",
-    author="Teradata",
-    long_description=open("README.md").read(),
+    version="1.0.0",
+    author="Teradata Corporation",
+    author_email="developers@teradata.com",
+    url="https://github.com/Teradata/feast-teradata",
+    long_description=long_description,
     long_description_content_type="text/markdown",
     python_requires=">=3.7",
     packages=find_packages(exclude=('tests',)),
+    package_data={
+        'feast_teradata.cli': [
+            'template/teradata/**',
+            'template/teradata/**/**',
+        ],
+    },
     install_requires=required,
     tests_require=['pytest==6.2.4'],
-    license_files=['LICENSE.txt', 'LICENSE-3RD-PARTY.txt'],
+    license_files=['LICENSE', 'LICENSE-3RD-PARTY.txt'],
+    entry_points={'console_scripts': [
+        'feast-td=feast_teradata.cli.cli:cli',
+    ]},
 )


Test patch:
diff --git a/.github/workflows/ci-integeration-tests.yml b/.github/workflows/ci-integeration-tests.yml
index 7230015..3431c28 100644
--- a/.github/workflows/ci-integeration-tests.yml
+++ b/.github/workflows/ci-integeration-tests.yml
@@ -4,7 +4,7 @@ on:
   pull_request:
   push:
     branches:
-      - 'main'
+      - 'master'
 
 jobs:
   test:
@@ -80,11 +80,14 @@ jobs:
         shell: bash
         run: ./.github/workflows/scripts/createDatabase.sh
 
-      - name: Run test workflow
+      - name: Build and install package
         run: |
           python setup.py sdist bdist_wheel
           pip install dist/*.whl
-          python create_repo.py
+
+      - name: Run test workflow
+        run: |
+          feast-td init-repo
           cd test_repo && python test_workflow.py
         env:
           CI_RUNNING: "true"
diff --git a/template/teradata/test_workflow.py b/feast_teradata/cli/template/teradata/test_workflow.py
similarity index 99%
rename from template/teradata/test_workflow.py
rename to feast_teradata/cli/template/teradata/test_workflow.py
index c0b42ed..8b2dc4d 100644
--- a/template/teradata/test_workflow.py
+++ b/feast_teradata/cli/template/teradata/test_workflow.py
@@ -11,7 +11,7 @@
 
 
 def run_demo():
-    store = FeatureStore(repo_path="./feature_repo")
+    store = FeatureStore(repo_path="feature_repo")
     print("\n--- Run feast apply to setup feature store on Teradata ---")
     command = "cd feature_repo; feast apply"
     subprocess.run(command, shell=True)