Package: airflow-tecton
[preview] airflow-tectonContains an Apache Airflow provider that allows you to author Tecton workflows inside Airflow.Two basic capabilities are supported:Submitting materialization jobsWaiting for Feature View/Feature Service data to materialize.Changelog0.1.0 Added 2 new operators to support triggering Feature Table ingestion jobs0.0.3 Added support forallow_overwritesetting in the operators0.0.2 Removed type annotations that caused compatibility issues with Airflow versions below 2.4.0.0.1 Initial releaseInstallation and ConfigurationInstallationYou can install this package viapip install airflow-tecton. Note that this requiresapache-airflow>=2.0.For a deployed environment, addairflow-tectonto yourrequirements.txtor wherever you configure installed packages.You can confirm a successful installation by runningairflow providers list, which should showairflow-tectonin the list.ConfigurationThis provider uses operators that interface with Tecton's API, thus it requires you set up Airflow to connect to Tecton.You can add a new connection by going toConnectionsunder theAdmintab in the Airflow UI. From there, hit the+button and selectTectonin the connection type dropdown. From there, you can configure the connection name, Tecton URL, and Tecton API key. Note that the default connection name istecton_default, so we recommend starting with this as a connection name to minimize configuration.UsageConfiguring a Feature View for manual triggeringABatchFeatureViewand aStreamFeatureViewcan be configured for manual triggering only. To do so, setbatch_trigger=BatchTriggerType.MANUAL. When set to manual, Tecton will not automatically create any batch materialization jobs for the Feature View. As of Tecton 0.6, any FeatureView can be manually triggered, but this is recommended mostly for manual usage.For aStreamFeatureView, only batch materialization job scheduling will be impacted by thebatch_triggersetting. Streaming materialization job scheduling will still be managed by Tecton.Here’s an example of aBatchFeatureViewconfigured for manual triggering.fromtectonimportbatch_feature_view,FilteredSource,Aggregation,BatchTriggerTypefromfraud.entitiesimportuserfromfraud.data_sources.transactionsimporttransactions_batchfromdatetimeimportdatetime,timedelta@batch_feature_view(sources=[FilteredSource(transactions_batch)],entities=[user],mode='spark_sql',aggregation_interval=timedelta(days=1),aggregations=[Aggregation(column='transaction',function='count',time_window=timedelta(days=1)),Aggregation(column='transaction',function='count',time_window=timedelta(days=30)),Aggregation(column='transaction',function='count',time_window=timedelta(days=90))],online=False,offline=True,feature_start_time=datetime(2022,5,1),tags={'release':'production'},owner='matt@tecton.ai',description='User transaction totals over a series of time windows, updated daily.',batch_trigger=BatchTriggerType.MANUAL# Use manual triggers)defuser_transaction_counts(transactions):returnf'''SELECTuser_id,1 as transaction,timestampFROM{transactions}'''If a Data Source input to the Feature View hasdata_delayset, then that delay will still be factored in to constructing training data sets but does not impact when the job can be triggered with the materialization API.Materialization Job SubmissionThere are two methods available to submit materialization jobs:TectonTriggerOperator: This triggers a materialization job for a Feature View. Tecton will retry any failing jobs automatically. Note that completion of this operator only means submission succeeded. To wait for completion, you must combine this withTectonSensor.TectonJobOperator: This triggers a materialization job for a Feature View with no retries. Additionally, when this operator is terminated, it will make a best effort to clean up the execution of the materialization job. Using this operator allows you to use standard Airflow keyword arguments to configure retry behavior. Additionally, this operator is synchronous, meaning that when the operator has succeeded, the underlying job has succeeded.Both of these require the following arguments:workspace - the workspace name of the Feature View you intend to materializefeature_view - the name of the Feature View you intend to materializeonline - whether the job should materialize to the online store. This requires that your FeatureView also has online materialization enabled.offline - whether the job should materialize to the offline store. This requires that your FeatureView also has offline materialization enabled.The time interval of the materialization job is configured automatically using Airflow templates. By default, it is from thedata_interval_startto thedata_interval_endof your DAG run. These can overridden if necessary.Example Usagefromairflow_tectonimportTectonJobOperator,TectonTriggerOperatorTectonJobOperator(task_id="tecton_job",workspace="my_workspace",feature_view="my_fv",online=False,offline=True,retries=3,)TectonTriggerOperator(task_id="trigger_tecton",workspace="my_workspace",feature_view="my_fv",online=True,offline=True,)Waiting For MaterializationTectonSensorThis enables you to wait for Materialization to complete for both Feature Views and Feature Services. Common uses are for monitoring as well as kicking off a training job after daily materialization completes.Example Usagefromairflow_tectonimportTectonSensorTectonSensor(task_id="wait_for_fs_online",workspace="my_workspace",feature_service="my_fs",online=True,offline=False,)TectonSensor(task_id="wait_for_fv",workspace="my_workspace",feature_view="my_fv",online=True,offline=True,)ExamplesSeeexample dags here.DevelopmentPre-commitThis repo uses pre-commit. Runpre-commit installin the repo root to configure pre-commit hooks. Pre-commit hooks take care of running unit tests as well as linting files.Run unit tests manuallyRunpython -m pytest tests/in the repo root.LicenseThis is licensed with the Apache 2.0 License.