Package: prefect-airbyte
prefect-airbyteWelcome!prefect-airbyteis a collection of prebuilt Prefect tasks and flows that can be used to quickly construct Prefect flows to interact withAirbyte.Getting StartedPython setupRequires an installation of Python 3.7+We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.Airbyte setupSeethe airbyte documentionon how to get your own instance.InstallationInstallprefect-airbytepipinstallprefect-airbyteA list of available blocks inprefect-airbyteand their setup instructions can be foundhere.ExamplesCreate anAirbyteServerblock and save itfromprefect_airbyte.serverimportAirbyteServer# running airbyte locally at http://localhost:8000 with default authlocal_airbyte_server=AirbyteServer()# running airbyte remotely at http://<someIP>:<somePort> as user `Marvin`remote_airbyte_server=AirbyteServer(username="Marvin",password="DontPanic42",server_host="42.42.42.42",server_port="4242")local_airbyte_server.save("my-local-airbyte-server")remote_airbyte_server.save("my-remote-airbyte-server")Trigger a defined connection syncfromprefectimportflowfromprefect_airbyte.serverimportAirbyteServerfromprefect_airbyte.connectionsimportAirbyteConnectionfromprefect_airbyte.flowsimportrun_connection_syncserver=AirbyteServer(server_host="localhost",server_port=8000)connection=AirbyteConnection(airbyte_server=server,connection_id="e1b2078f-882a-4f50-9942-cfe34b2d825b",status_updates=True,)@flowdefairbyte_syncs():# do some setupsync_result=run_connection_sync(airbyte_connection=connection,)# do some other things, like trigger DBT based on number of records syncedprint(f'Number of Records Synced:{sync_result.records_synced}')❯ python airbyte_syncs.py03:46:03 | prefect.engine - Created flow run 'thick-seahorse' for flow 'example_trigger_sync_flow'03:46:03 | Flow run 'thick-seahorse' - Using task runner 'ConcurrentTaskRunner'03:46:03 | Flow run 'thick-seahorse' - Created task run 'trigger_sync-35f0e9c2-0' for task 'trigger_sync'03:46:03 | prefect - trigger airbyte connection: e1b2078f-882a-4f50-9942-cfe34b2d825b, poll interval 3 seconds03:46:03 | prefect - pending03:46:06 | prefect - running03:46:09 | prefect - running03:46:12 | prefect - running03:46:16 | prefect - running03:46:19 | prefect - running03:46:22 | prefect - Job 26 succeeded.03:46:22 | Task run 'trigger_sync-35f0e9c2-0' - Finished in state Completed(None)03:46:22 | Flow run 'thick-seahorse' - Finished in state Completed('All states completed.')Export an Airbyte instance's configurationNOTE: The API endpoint corresponding to this task is no longer supported by open-source Airbyte versions as of v0.40.7. Check out theOctavia CLI docsfor more info.importgzipfromprefectimportflow,taskfromprefect_airbyte.configurationimportexport_configurationfromprefect_airbyte.serverimportAirbyteServer@taskdefzip_and_write_somewhere(airbyte_config:bytearray,somewhere:str,):withgzip.open(somewhere,'wb')asf:f.write(airbyte_config)@flowdefexample_export_configuration_flow(filepath:str):# Run other tasks and subflows hereairbyte_config=export_configuration(airbyte_server=AirbyteServer.load("my-airbyte-server-block"))zip_and_write_somewhere(somewhere=filepath,airbyte_config=airbyte_config)if__name__=="__main__":example_export_configuration_flow('*://**/my_destination.gz')Usewith_optionsto customize options on any existing task or flowfromprefectimportflowfromprefect_airbyte.connectionsimportAirbyteConnectionfromprefect_airbyte.flowsimportrun_connection_synccustom_run_connection_sync=run_connection_sync.with_options(name="Custom Airbyte Sync Flow",retries=2,retry_delay_seconds=10,)@flowdefsome_airbyte_flow():custom_run_connection_sync(airbyte_connection=AirbyteConnection.load("my-airbyte-connection-block"))some_airbyte_flow()For more tips on how to use tasks and flows in a Collection, check outUsing Collections!ResourcesIf you encounter and bugs while usingprefect-airbyte, feel free to open an issue in theprefect-airbyterepository.If you have any questions or issues while usingprefect-airbyte, you can find help in either thePrefect Discourse forumor thePrefect Slack communityFeel free to star or watchprefect-airbytefor updates too!ContributeIf you'd like to help contribute to fix an issue or add a feature toprefect-airbyte, pleasepropose changes through a pull request from a fork of the repository.Contribution Steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changes.Add tests.Insert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request.