Package: airbyte-lib
airbyte-libairbyte-lib is a library that allows to run Airbyte syncs embedded into any Python application, without the need to run Airbyte server.DevelopmentMake surePoetry is installed.Runpoetry installFor examples, check out theexamplesfolder. They can be run viapoetry run python examples/<example file>Unit tests and type checks can be run viapoetry run pytestReleaseIn your PR:Bump the version inpyproject.tomlAdd a changelog entry to the table belowOnce the PR is merged, go to Github and trigger thePublish AirbyteLib Manuallyworkflow. This will publish the new version to PyPI.Secrets ManagementAirbyteLib can auto-import secrets from the following sources:Environment variables.Google Colab secrets.Manual entry viagetpass.Note: Additional secret store options may be supported in the future.More info here.Retrieving Secretsfromairbyte_libimportget_secret,SecretSourcesource=get_connection("source-github")source.set_config("credentials":{"personal_access_token":get_secret("GITHUB_PERSONAL_ACCESS_TOKEN"),})Theget_secret()function accepts an optionalsourceargument of enum typeSecretSource. If omitted or set toSecretSource.ANY, AirbyteLib will search all available secrets sources. Ifsourceis set to a specific source, then only that source will be checked. If a list ofSecretSourceentries is passed, then the sources will be checked using the provided ordering.By default, AirbyteLib will prompt the user for any requested secrets that are not provided via other secret managers. You can disable this prompt by passingprompt=Falsetoget_secret().VersioningVersioning followsSemantic Versioning. For new features, bump the minor version. For bug fixes, bump the patch version. For pre-releases, appenddev.Nto the version. For example,0.1.0dev.1is the first pre-release of the0.1.0version.DocumentationRegular documentation lives in the/docsfolder. Based on the doc strings of public methods, we generate API documentation usingpdoc. To generate the documentation, runpoetry run generate-docs. The documentation will be generated in thedocs/generatefolder. This needs to be done manually when changing the public interface of the library.A unit test validates the documentation is up to date.Connector compatibilityTo make a connector compatible with airbyte-lib, the following requirements must be met:The connector must be a Python package, with apyproject.tomlor asetup.pyfile.In the package, there must be arun.pyfile that contains arunmethod. This method should read arguments from the command line, and run the connector with them, outputting messages to stdout.Thepyproject.tomlorsetup.pyfile must specify a command line entry point for therunmethod calledsource-<connector name>. This is usually done by adding aconsole_scriptssection to thepyproject.tomlfile, or aentry_pointssection to thesetup.pyfile. For example:[tool.poetry.scripts]source-my-connector="my_connector.run:run"setup(...entry_points={'console_scripts':['source-my-connector = my_connector.run:run',],},...)To publish a connector to PyPI, specify thepypisection in themetadata.yamlfile. For example:data:# ...remoteRegistries:pypi:enabled:truepackageName:"airbyte-source-my-connector"Validating source connectorsTo validate a source connector for compliance, theairbyte-lib-validate-sourcescript can be used. It can be used like this:airbyte-lib-validate-source—connector-dir.-—sample-configsecrets/config.jsonThe script will install the python package in the provided directory, and run the connector against the provided config. The config should be a valid JSON file, with the same structure as the one that would be provided to the connector in Airbyte. The script will exit with a non-zero exit code if the connector fails to run.For a more lightweight check, the--validate-install-onlyflag can be used. This will only check that the connector can be installed and returns a spec, no sample config required.ChangelogVersionPRDescription0.1.0#35184Beta Release 0.1.00.1.0dev.2#34111Initial publish - add publish workflow