Package: airlift
AirliftIntroductionAirlift is a Command Line Interface (CLI) tool designed to provide a local development environment for Apache Airflow with a simple but flexible interface.
It is built on top of the officialAirflow Helm Chart.RequirementsAirlift requires the following software to be installed on your system:HelmDockerKindBelow are the installation instructions for each of these tools on MacOS and Linux distributions.It is also recommended to allocate at least 4GB of RAM for Docker to run this service.Install HomebrewHomebrew is a package manager that we will use to install the necessary software. If you don't have Homebrew installed, you can install it by following these instructions:/bin/bash-c"$(curl-fsSLhttps://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"Install SoftwareWith Homebrew installed, you can now install Helm, Docker, and Kind.HelmbrewinstallhelmDockerbrewinstall--caskdocker-OR- use a tool likeDocker DesktoporRancher DesktopKindbrewinstallkindNote: This software was tested and validated working with kind v0.17.There are known issues with kind v0.20 and Rancher. If you are experiencing issues, please downgrade
your kind installationby installing from the source/release binaries.InstallationAirlift can be installed using pip:pipinstallairliftUsageThe general syntax for using the Airlift CLI tool is:airlift[subcommand][options]Subcommands and Options1. startStarts the Airflow service.airliftstart-d/path/to/dags-p/path/to/plugins-r/path/to/requirements.txtNote: The DAG and Plugins folders are mounted directly to the airflow service for hot-reloading. When you make a change locally, it should automatically appear in the Airflow UI.Note: Start times for Airflow can be upwards of 5 minutes, due to the bootstrapping, installation of required PyPi packages, and the creation of the Postgres database.
This all depends on your local machines power & the complexity of your Airflow setup.2. checkChecks if all pre-requisite software is installed.airliftcheck3. pausePauses the Airflow service.airliftpause4. unpauseUnpauses the Airflow service.airliftunpause5. removeRemoves all containers/clusters related to theairliftservice.airliftremove6. statusChecks the status of the service and whether or not it is reachable.airliftstatus-P80807. import_variablesImports avariables.jsonfile to a running Airflow instance.airliftimport_variables-P8080-V/path/to/variables.json8. run_dagRuns a DAG given an ID.airliftrun_dag-P8080-Dexample_dag_idConfiguration filesHelm values.yamlThis file provides the configuration for the Airflow Helm chart. This can be used for things such as:Setting the Secrets Backend to AWS Secrets ManagerAdding custom environment variables (such as connections)Changing the executorModifying the memory allocation for the webserver/scheduler/workersUpdating anyairflow.cfgvalue.Here's an example:executor:"CeleryExecutor"config:core:load_examples:'False'executor:CeleryExecutorcolored_console_log:'False'# Airflow scheduler settingsscheduler:#  hostAliases for the scheduler podhostAliases:[]#  - ip: "127.0.0.1"#    hostnames:#      - "foo.local"#  - ip: "10.1.2.3"#    hostnames:#      - "foo.remote"# If the scheduler stops heartbeating for 5 minutes (5*60s) kill the# scheduler and let Kubernetes restart itlivenessProbe:initialDelaySeconds:10timeoutSeconds:20failureThreshold:5periodSeconds:60command:~# Airflow 2.0 allows users to run multiple schedulers,# However this feature is only recommended for MySQL 8+ and Postgresreplicas:1# Max number of old replicasets to retainrevisionHistoryLimit:~# Command to use when running the Airflow scheduler (templated).command:~# Args to use when running the Airflow scheduler (templated).args:["bash","-c","execairflowscheduler"]You can find all the possible configuration overrides here:https://artifacthub.io/packages/helm/apache-airflow/airflow?modal=valuesNote: By default, we disable thelivenessProbechecks for the scheduler & triggerer due to conflicts with Kind. See./src/airlift/config/helm/values.yamlfor the exact config valuesAirlift ConfigurationThe Airlift configuration file overrides all flag values to simplify starting the service.For example,$HOME/.config/airlift/config.yaml:# config.yamldag_path:/path/to/dagsplugin_path:/path/to/pluginsrequirements_file:/path/to/requirements.txthelm_values_file:/path/to/values.yamlextra_volume_mounts:-hostPath=/my/cool/path,containerPath=/my/mounted/path,name=a_unique_namecluster_config_file:/path/to/cluster/config.yamlimage:'apache/airflow:2.6.0'helm_chart_version:'1.0.0'port:8080post_start_dag_id:'example_dag_id'In this example,dag_pathin the yaml file overrides the-dsetting,plugin_pathoverrides the-psetting, and so forth.Using this configuration, you can now start the service using:airliftstart-c$HOME/.config/airlift/config.yamlExamplesSee here for examples with common configuration modifications.FAQSee here for Frequently Asked QuestionsMotivationThe motivation behind the creation of Airlift is to simplify the process of setting up a local development environment for Apache Airflow. It aims to be a flexible tool that allows developers to easily configure and manage their Airflow instances with unlimited flexibility.Support and ContributionIf you encounter any issues or have suggestions for improvements, feel free to open an issue on the GitHub repository. Contributions to the project are also welcome.ContactIf you have questions or feedback about Airlift, please reach out by opening an issue on the GitHub repository.