Package: airflowctl
airflowctlairflowctlis a command-line tool for managing Apache Airflow™ projects.
It provides a set of commands to initialize, build, start, stop, and manage Airflow projects.
Withairflowctl, you can easily set up and manage your Airflow projects, install
specific versions of Apache Airflow, and manage virtual environments.The main goal ofairflowctlis for first-time Airflow users to install and setup Airflow using a single command and
for existing Airflow users to manage multiple Airflow projects with different Airflow versions on the same machine.FeaturesProject Initialization with Connections & Variables:Initialize a new Airflow project with customizable project name, Apache Airflow
version, and Python version. It also allows you to manage Airflow connections and variables.Automatic Virtual Environment Management:Automatically create and manage virtual environments for
your Airflow projects, even for Python versions that are not installed on your system.Airflow Version Management:Install and manage specific versions of Apache Airflow.Background Process Management:Start and stop Airflow in the background with process management capabilities.Live Logs Display:Continuously display live logs of background Airflow processes with optional log filtering.Table of ContentsInstallationQuickstartUsageStep 1: Initialize a New ProjectStep 2: Build the ProjectStep 3: Start AirflowStep 4: Monitor LogsStep 5: Stop AirflowStep 6: List Airflow ProjectsStep 7: Show Project InfoStep 8: Running Airflow CommandsStep 9: Changing Airflow configurationUsing with other Airflow toolsAstro CLIInstallationpipinstallairflowctlQuickstartTo initialize a new Airflow project with the latest airflow version, build a Virtual environment
and run the project, run the following command:airflowctlinitmy_airflow_project--build-startThis will start Airflow and display the logs in the terminal. You can
access the Airflow UI athttp://localhost:8080. To stop Airflow, pressCtrl+C.UsageStep 1: Initialize a New ProjectTo create a new Apache Airflow project, use the init command.
This command sets up the basic project structure, including configuration files,
directories, and sample DAGs.airflowctlinit<project_name>--airflow-version<version>--python-version<version>Example:airflowctlinitmy_airflow_project--airflow-version2.6.3--python-version3.8This creates a new project directory with the following structure:my_airflow_project
├──.env
├──.gitignore
├──dags
│└──example_dag_basic.py
├──plugins
├──requirements.txt
└──settings.yamlDescription of the files and directories:.envfile contains the environment variables for the project..gitignorefile contains the default gitignore settings.dagsdirectory contains the sample DAGs.pluginsdirectory contains the sample plugins.requirements.txtfile contains the project dependencies.settings.yamlfile contains the project settings, including the project name,
Airflow version, Python version, and virtual environment path.In our examplesettings.yamlfile would look like this:# Airflow version to be installedairflow_version:"2.6.3"# Python version for the projectpython_version:"3.8"# Path to a virtual environment to be used for the projectmode:virtualenv:venv_path:"PROJECT_DIR/.venv"# Airflow connectionsconnections:# Example connection# - conn_id: example#   conn_type: http#   host: http://example.com#   port: 80#   login: user#   password: pass#   schema: http#   extra:#      example_extra_field: example-value# Airflow variablesvariables:# Example variable# - key: example#   value: example-value#   description: example-descriptionEdit thesettings.yamlfile to customize the project settings.Step 2: Build the ProjectThe build command creates the virtual environment, installs the specified Apache Airflow
version, and sets up the project dependencies.Run the build command from the project directory:cdmy_airflow_project
airflowctlbuildThe CLI relies onpyenvto download and install a Python version if the version is not already installed.Example, if you have Python 3.8 installed but you specify Python 3.7 in thesettings.yamlfile,
the CLI will install Python 3.7 usingpyenvand create a virtual environment with Python 3.7 first.Optionally, you can choose custom virtual environment path in case you have already installed apache-airflow package
and other dependencies.
Pass the existing virtualenv path using--venv_pathoption to theinitcommand or insettings.yamlfile.
Make sure the existing virtualenv has same airflow and python version as yoursettings.yamlfile states.Step 3: Start AirflowTo start Airflow services, use the start command.
This command activates the virtual environment and launches the Airflow web server and scheduler.Example:airflowctlstartmy_airflow_projectYou can also start Airflow in the background with the--backgroundflag:airflowctlstartmy_airflow_project--backgroundStep 4: Monitor LogsTo monitor logs from the background Airflow processes, use the logs command.
This command displays live logs and provides options to filter logs for specific components.Exampleairflowctllogsmy_airflow_projectTo filter logs for specific components:# Filter logs for schedulerairflowctllogsmy_airflow_project-s# Filter logs for webserverairflowctllogsmy_airflow_project-w# Filter logs for triggererairflowctllogsmy_airflow_project-t# Filter logs for scheduler and webserverairflowctllogsmy_airflow_project-s-wStep 5: Stop AirflowTo stop Airflow services if they are still running, use the stop command.Example:airflowctlstopmy_airflow_projectStep 6: List Airflow ProjectsTo list all Airflow projects, use the list command.Example:airflowctllistStep 7: Show Project InfoTo show project info, use the info command.Example:# From the project directoryairflowctlinfo# From outside the project directoryairflowctlinfomy_airflow_projectStep 8: Running Airflow commandsTo run Airflow commands, use theairflowctl airflowcommand. All the commands afterairflowctl airfloware passed to the Airflow CLI.:# From the project directoryairflowctlairflow<airflow_command>Example:$airflowctlairflowversion2.6.3You can also runairflowctl airflow --helpto see the list of available commands.$airflowctlairflow--help
Usage:airflowctlairflow[OPTIONS]COMMAND[ARGS]...RunAirflowcommands.

PositionalArguments:GROUP_OR_COMMANDGroups:celeryCelerycomponentsconfigViewconfigurationconnectionsManageconnectionsdagsManageDAGsdbDatabaseoperationsjobsManagejobskubernetesToolstohelpruntheKubernetesExecutorpoolsManagepoolsprovidersDisplayprovidersrolesManagerolestasksManagetasksusersManageusersvariablesManagevariablesCommands:cheat-sheetDisplaycheatsheetdag-processorStartastandaloneDagProcessorinstanceinfoShowinformationaboutcurrentAirflowandenvironmentkerberosStartakerberosticketrenewerpluginsDumpinformationaboutloadedpluginsrotate-fernet-keyRotateencryptedconnectioncredentialsandvariablesschedulerStartaschedulerinstancestandaloneRunanall-in-onecopyofAirflowsync-permUpdatepermissionsforexistingrolesandoptionallyDAGstriggererStartatriggererinstanceversionShowtheversionwebserverStartaAirflowwebserverinstance

Options:-h,--helpshowthishelpmessageandexitExample:# Listing dags$airflowctlairflowdagslist
dag_id|filepath|owner|paused==================+======================+=========+=======example_dag_basic|example_dag_basic.py|airflow|True# Running standalone$airflowctlairflowstandaloneOr you can activate the virtual environment first and then run the commands as shown below.Example:# From the project directorysource.venv/bin/activate# Source all the environment variablessource.env
airflowversionTo add a new DAG, add the DAG file to thedagsdirectory.To edit an existing DAG, edit the DAG file in thedagsdirectory.
The changes will be reflected in the Airflow web server.Step 9: Changing Airflow Configurationsairflowctlby default uses SQLite as the backend database andSequentialExecutoras the executor.
However, if you want to use other databases or executors, you can stop the project and
either a) edit theairflow.cfgfile or b) add environment variables to the.envfile.Example:# Stop the projectairflowctlstopmy_airflow_project# Changing the executor to LocalExecutor# Change the database to PostgreSQL if you already have it installedecho"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost:5432/airflow">>.envecho"AIRFLOW__CORE__EXECUTOR=LocalExecutor">>.env# Start the projectairflowctlstartmy_airflow_projectCheck theAirflow documentationfor all the available Airflow configurations.Using Local Executor with SQLiteFor Airflow >= 2.6, you can runLocalExecutorwithsqliteas the backend database by
adding the following environment variable to the.envfile:_AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1AIRFLOW__CORE__EXECUTOR=LocalExecutorThis should automatically happen for you when you runairflowctl airflowcommand if
Airflow version==2.6.*.[!WARNING]
Sqlite is not recommended for production use. Use it only for development and testing only.Other CommandsFor more information and options, you can use the--helpflag with each command.Using with other Airflow toolsairflowctlcan be used with other Airflow projects as long as the project structure is the same.Astro CLIairflowctlcan be used withAstro CLIprojects too.Whileairflowctlis a tool that allows you to run Airflow locally using virtual environments, Astro CLI
allows you to run Airflow locally using docker.airflowctlcan read theairflow_settings.yamlfile generated by Astro CLI for reading connections & variables. It
will then re-use it assettingsfile forairflowctl.For example, if you have an Astro CLI project:Run theairflowctl init . --build-startcommand to initializeairflowctlfrom the project directory.
Pressyto continue when prompted.It will then ask you for the Airflow version, enter the version you are using, by
default uses the latest Airflow version, press enter to continueIt will use the installed Python version as the project's python version. If you
want to use a different Python version, you can specify it in theairflow_settings.yamlfile in thepython_versionfield.# From the project directory$cdastro_project
$airflowctlinit.--build-start
Directory/Users/xyz/astro_projectisnotempty.Continue?[y/N]:y
Project/Users/xyz/astro_projectaddedtotracking.
Airflowprojectinitializedin/Users/xyz/astro_project
DetectedAstroproject.UsingAstrosettingsfile(/Users/kaxilnaik/Desktop/proj1/astro_project/airflow_settings.yaml).'airflow_version'notfoundinairflow_settings.yamlfile.WhatistheAirflowversion?[2.6.3]:
Virtualenvironmentcreatedat/Users/xyz/astro_project/.venv
...
...If you see an error like the following, removeairflow.cfgfile from the project directory and removeAIRFLOW_HOMEfrom.envfile if it exists and try again.Error:theremightbeaproblemwithyourprojectstartingup.
Thewebserverhealthchecktimedoutafter1m0sbutyourprojectwillcontinuetryingtostart.
Run'astro dev logs --webserver | --scheduler'fordetails.LicenseThis project is licensed under the terms of theApache 2.0 License