Package: airflow-provider-fivetran-async
Fivetran Async Provider for Apache AirflowThis package provides an async operator, sensor and hook that integratesFivetraninto Apache Airflow.FivetranSensorallows you to monitor a Fivetran sync job for completion before running downstream processes.FivetranOperatorsubmits a Fivetran sync job and polls for its status on the triggerer.
Since an async sensor or operator frees up worker slot while polling is happening on the triggerer,
they consume less resources when compared to traditional "sync" sensors and operators.Fivetran automates your data pipeline, and Airflow automates your data processing.InstallationPrerequisites: An environment runningapache-airflow.pip install airflow-provider-fivetran-asyncConfigurationIn the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:Conn Id:fivetranConn Type:FivetranLogin: Fivetran API KeyPassword: Fivetran API SecretFind the Fivetran API Key and Secret in theFivetran Account Settings, under theAPI Configsection. See our documentation for more information onFivetran API Authentication.The sensor assumes theConn Idis set tofivetran, however if you are managing multiple Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a customConn Id.ModulesFivetran Operator Asyncfromfivetran_provider_async.operatorsimportFivetranOperatorFivetranOperatorsubmits a Fivetran sync job and monitors it on trigger for completion.FivetranOperatorrequires that you specify theconnector_idof the Fivetran connector you wish to trigger. You can findconnector_idin the Settings page of the connector you configured in theFivetran dashboard.TheFivetranOperatorwill wait for the sync to complete so long aswait_for_completion=True(this is the default). It is recommended that
you run in deferrable mode (this is also the default). Ifwait_for_completion=False, the operator will return the timestamp for the last sync.Import into your DAG via:Fivetran Sensor Asyncfromfivetran_provider_async.sensorsimportFivetranSensorFivetranSensormonitors a Fivetran sync job for completion.
Monitoring withFivetranSensorallows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency.FivetranSensorrequires that you specify theconnector_idof the Fivetran connector you want to wait for. You can findconnector_idin the Settings page of the connector you configured in theFivetran dashboard.You can use multiple instances ofFivetranSensorto monitor multiple Fivetran connectors.FivetranSensoris most commonly useful in two scenarios:Fivetran is using a separate scheduler than the Airflow scheduler.You setwait_for_completion=Falsein theFivetranOperator, and you need to await theFivetranOperatortask later. (You may want to do this if you want to arrange your DAG such that some tasks are dependent onstartinga sync and other tasks are dependent oncompletinga sync).If you are doing the 1st pattern, you may find it useful to set thecompleted_after_timetodata_interval_end, ordata_interval_endwith some buffer:fivetran_sensor=FivetranSensor(task_id="wait_for_fivetran_externally_scheduled_sync",connector_id="bronzing_largely",poke_interval=5,completed_after_time="{{ data_interval_end + macros.timedelta(minutes=1) }}",)If you are doing the 2nd pattern, you can use XComs to pass the target completed time to the sensor:fivetran_op=FivetranOperator(task_id="fivetran_sync_my_db",connector_id="bronzing_largely",wait_for_completion=False,)fivetran_sensor=FivetranSensor(task_id="wait_for_fivetran_db_sync",connector_id="bronzing_largely",poke_interval=5,completed_after_time="{{ task_instance.xcom_pull('fivetran_sync_op', key='return_value') }}",)fivetran_op>>fivetran_sensorYou may also specify theFivetranSensorwithout acompleted_after_time.
In this case, the sensor will make note of when the last completed time was, and will wait for a new completed time.ExamplesSee theexamplesdirectory for an example DAG.IssuesPlease submitissuesandpull requestsin our official repo:https://github.com/astronomer/airflow-provider-fivetran-asyncWe are happy to hear from you. Please email any feedback to the authors athumans@astronomer.io.