Package: airbyte-serverless
Airbyte made simple🔍️ What is AirbyteServerless?AirbyteServerless is a simple tool tomanage Airbyte connectors, run themlocallyor deploy them inserverlessmode.💡  Why AirbyteServerless?Airbyteis a must-have in your data-stack with itscatalog of open-source connectors to move your data from any source to your data-warehouse.To manage these connectors, Airbyte offersAirbyte-Open-Source-Platformwhich includes a server, workers, database, UI, orchestrator, connectors, secret manager, logs manager, etc.AirbyteServerless aims at offeringa lightweight alternativeto Airbyte-Open-Source-Platform to simplify connectors management.📝 Comparing Airbyte-Open-Source-Platform & AirbyteServerlessAirbyte-Open-Source-PlatformAirbyteServerlessHas a UIHas NO UIConnections configurations are managed by documented yaml filesHas a databaseHas NO database- Configurations files are versioned in git- The destination stores thestate(thecheckpointof where sync stops) andlogswhich can then be visualized with your preferred BI toolHas a transform layerAirbyte loads your data in a raw format but then enables you to perform basic transform such as replace, upsert,schema normalizationHas NO transform layer- Data is appended in your destination in raw format.-airbyte_serverlessis dedicated to do one thing and do it well:Extract-Load.NOT Serverless- Can be deployed on a VM or Kubernetes Cluster.- The platform is made of tens of dependent containers that you CANNOT deploy with serverlessServerless- An Airbyte source docker image is upgraded with a destination connector- The upgraded docker image can then be deployed as an isolatedCloud Run Job(orCloud Run Service)- Cloud Run is natively monitored with metrics, dashboards, logs, error reporting, alerting, etc- It can be scheduled or triggered by eventsIs scalable with conditionsScalable if deployed on autoscaled Kubernetes Cluster and if you are skilled enough.👉Check that you are skilled enough with Kubernetes by watchingthis video😁.Is scalableEach connector is deployed independently of each other. You can have as many as you want.💥 Getting Started withabsCLIabsis the CLI (command-line-interface) of AirbyteServerless which facilitates connectors management.Installabs🛠️pipinstallairbyte-serverlessCreate your first Connection 👨‍💻abscreatemy_first_connection--source="airbyte/source-faker:0.1.4"--destination="bigquery"--remote-runner"cloud_run_job"Docker is required. Make sure you have it installed.sourceparam can be any Public Docker Airbyte Source (hereis the list). We recomend that you use faker source to get started.destinationparam must be one of the following:print(default value if not set)bigquerycontributions are welcome to offer more destinations🤗remote-runnerparam must becloud_run_job. More integrations will come in the future. This remote-runner is only used if you want to run the connection on a remote runner and schedule it.The command will create a configuration file./connections/my_first_connection.yamlwith initialized configuration.Update this configuration file to suit your needs.Run it! ⚡absrunmy_first_connectionThis will launch an Extract-Load Job from the source to the destination.Theruncommmand will only work if you have correctly edited./connections/my_first_connection.yamlconfiguration file.If you chosebigquerydestination, you must:havegcloudinstalled on your machine with default credentials initialized with the commandgcloud auth application-default login.have correctly edited thedestinationsection of./connections/my_first_connection.yamlconfiguration file. You must havedataEditorpermission on the chosen BigQuery dataset.Data is always appended at destination (not replaced nor upserted). It will be in raw format.If the connector supports incremental extract (extract only new or recently modified data) then this mode is chosen.Select only some streams 🧛🏼You may not want to copy all the data that the source can get. To see all availablestreamsrun:abslist-available-streamsmy_first_connectionIf you want to configure your connection with only some of these streams, run:absset-streamsmy_first_connection"stream1,stream2"Nextrunexecutions will extract selected streams only.Handle Secrets 🔒For security reasons, you do NOT want to store secrets such as api tokens in your yaml files. Instead, add your secrets in Google Secret Manager by followingthis documentation. Then you can add the secret resource name in the yaml file such as below:source:docker_image:"..."config:api_token:GCP_SECRET({SECRET_RESOURCE_NAME})Replace{SECRET_RESOURCE_NAME}by your secret resource name which must have the format:projects/{PROJECT_ID}/secrets/{SECRET_ID}/versions/{SECRET_VERSION}. To get this path:Go to theSecret Manager pagein the Google Cloud console.Go to the Secret Manager pageOn the Secret Manager page, click on the Name of a secret.On the Secret details page, in the Versions table, locate a secret version to access.In the Actions column, click on the three dots.Click on 'Copy Resource Name' from the menu.Run from the Remote Runner 🚀absremote-runmy_first_connectionTheremote-runcommmand will only work if you have correctly edited./connections/my_first_connection.yamlconfiguration file including theremote_runnerpart.This command will launch an Extract-Load Job like theabs runcommand. The main difference is that the command will be run on a remote deployed container (we use Cloud Run Job as the only container runner for now).If you chosebigquerydestination, the service account you put inservice_accountfield ofremote_runnersection of the yaml must bebigquery.dataEditoron the target dataset and have permission to create some BigQuery jobs in the project.If your yaml config contains some Google Secrets, the service account you put inservice_accountfield ofremote_runnersection of the yaml must has read access to the secrets.Schedule the run from the Remote Runner ⏱️absschedule-remote-runmy_first_connection"0 * * * *"⚠️ THIS IS NOT IMPLEMENTED YETGet help 📙$abs--help
Usage:abs[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:createCreateCONNECTIONlistListcreatedconnectionslist-available-streamsListavailablestreamsofCONNECTIONremote-runRunCONNECTIONExtract-LoadJobfromremoterunnerrunRunCONNECTIONExtract-LoadJobrun-env-varsRunExtract-LoadJobconfiguredbyenvironment...set-streamsSetSTREAMStoretrieveforCONNECTION(STREAMS...Keep in touch 🧑‍💻Join our Slackfor any question, to get help for getting started, to speak about a bug, to suggest improvements, or simply if you want to have a chat 🙂.👋 ContributeAny contribution is more than welcome 🤗!Add a ⭐ on the repo to show your supportJoin our Slackand talk with usRaise an issue to raise a bug or suggest improvementsOpen a PR! Below are some suggestions of work to be done:implements a schedulerimplement theget_logsmethod ofBigQueryDestinationenable updating cloud run job instead of deleting/creating when it already existsadd a new destination connector (Cloud Storage?)add more remote runners such compute instances.implements vpc accessimplement optional post-processing (replace, upsert data at destination instead of append?)🏆 CreditsBig kudos to Airbyte for all the hard work on connectors!The generation of the sample connector configuration in yaml is heavily inspired from the code ofoctaviaCLI developed by airbyte.