Package: valmi-airbyte-cdk
Connector Development Kit (Python)The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:HTTP APIs (REST APIs, GraphQL, etc..)Singer TapsGeneric Python sources (anything not covered by the above)The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.This document is a general introduction to the CDK. Readers should have basic familiarity with theAirbyte Specificationbefore proceeding.Getting StartedGenerate an empty connector using the code generator. First clone the Airbyte repository then from the repository root runcd airbyte-integrations/connector-templates/generator
./generate.shthen follow the interactive prompt. Next, find allTODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.Additionally, you can followthis tutorialfor a complete walkthrough of creating an HTTP connector using the Airbyte CDK.Concepts & DocumentationSee theconcepts docsfor a tour through what the API offers.Example ConnectorsHTTP Connectors:Exchangerates APIStripeSlackSinger connectors:SalesforceGithubSimple Python connectors using the barebonesSourceabstraction:Google SheetsMailchimpContributingFirst time setupWe assumepythonpoints to python >=3.8.Setup a virtual env:python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependenciesIterationIterate on the code locallyRun tests viapython -m pytest -s unit_testsPerform static type checks usingmypy airbyte_cdk.MyPyconfiguration is in.mypy.ini.Thetype_check_and_test.shscript bundles both type checking and testing in one convenient command. Feel free to use it!Autogenerated filesIf the iteration you are working on includes changes to the models, you might want to regenerate them. In order to do that, you can run:SUB_BUILD=CDK ./gradlew formatThis will generate the files based on the schemas, add the license information and format the code. If you want to only do the former and rely on
pre-commit to the others, you can run the appropriate generation command i.e../gradlew generateComponentManifestClassFiles.TestingAll tests are located in theunit_testsdirectory. Runpython -m pytest --cov=airbyte_cdk unit_tests/to run them. This also presents a test coverage report.Building and testing a connector with your local CDKWhen developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:Running a connector locallyBuilding and running a source via DockerInstalling your local CDK into a local Python connectorIn order to get a local Python connector running your local CDK, do the following.First, make sure you have your connector's virtual environment active:# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directorysource.venv/bin/activate# if you haven't installed dependencies for your connector alreadypipinstall-e.Then, navigate to the CDK and install it in editable mode:cd../../../airbyte-cdk/python
pipinstall-e.You should see thatpiphas uninstalled the version ofairbyte-cdkdefined by your connector'ssetup.pyand installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.Building a Python connector in Docker with your local CDK installedYou can build your connector image with the local CDK using# from the airbytehq/airbyte base directoryCONNECTOR_TAG=<TAG_NAME>CONNECTOR_NAME=<CONNECTOR_NAME>shairbyte-integrations/scripts/build-connector-image-with-local-cdk.shNote that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.Running Connector Acceptance Tests for a single connector in Docker with your local CDK installedTo run acceptance tests for a single connectors using the local CDK, from the connector directory, runLOCAL_CDK=1shacceptance-test-docker.shTo additionally fetch secrets required by CATs, set theFETCH_SECRETSenvironment variable. This requires you to have a Google Service Account, and the GCP_GSM_CREDENTIALS environment variable to be set, per the instructionshere.Running Connector Acceptance Tests for multiple connectors in Docker with your local CDK installedTo run acceptance tests for multiple connectors using the local CDK, from the root of theairbyterepo, run./airbyte-cdk/python/bin/run-cats-with-local-cdk.sh-c<connector1>,<connector2>,...Publishing a new version to PyPiOpen a PROnce it is approved andmerged, an Airbyte member must run thePublish CDK Manuallyworkflow from master usingrelease-type=major|manor|patchand setting the changelog message.Coming SoonFull OAuth 2.0 support (including refresh token issuing flow via UI or CLI)Airbyte Java HTTP CDKCDK for Async HTTP endpoints (request-poll-wait style endpoints)CDK for other protocolsDon't see a feature you need?Create an issue and let us know how we can help!