Installing Apache Airflow on Windows 10
Tags: Windows 10, Apache Airflow, Dagscheduler, Python, Ubuntu

This guide is designed to walk you through installing Apache Airflow on a Windows 10 machine using Ubuntu. Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.

System Information:

Windows 10 Pro: 10.0.18363 Build 18363

Ubuntu: 18.04.4 LTS

Python: 2.7.17

Apache Airflow: 1.10.10

Step 1: Installing Linux Subsystem (Ubuntu)

You can find a free copy of Ubuntu in the Microsoft Store here.

Download and install Ubuntu. Before opening the program there are a couple housekeeping items that we need to take care of first.

a. Enable Developer Mode

If you haven't already, you will need to turn on Developer Mode within Windows 10. Type “Developer” into the Windows search bar and select the option that says “Developer Settings”.

In the page that appears, select the bubble next to the “Developer Mode” option. That’s it!

b. Enable Linux Subsystem

In the Windows search bar type “Windows Feature” and select the option that says “Turn a Windows Feature on or off.”

Check the box next to Windows Subsystem for Linux and click OK.

Note: This will require a restart in order to take effect.

c. Download C++ Build Tools (14.0)

In order to use Ubuntu you will need to install the Visual C++ Build Tools that can be found here.

After these steps, our house cleaning is complete! Once you fire up Ubuntu, it will begin its initial installation process which usually only takes a minute or two.

Ubuntu will then prompt you for a username and password. Set these to your desired credentials and press enter. When you arrive at a prompt like the one below, you are ready to move on to step two.

Step 2: Installing PIP

To install Airflow we will need to make sure that pip is installed. Run the following command to install the latest version of pip.

sudo apt-get install software-properties-common sudo apt-add-repository universe sudo apt-get update sudo apt-get install python-setuptools sudo apt install python-pip sudo -H pip install --upgrade pip

Verify the installation by running the command below:

pip -V

Congratulations, pip is now installed!

Step 3: Installing Dependencies

The first step for installing Airflow’s dependencies requires us to run the following commands. These ensure that all the required packages are installed and up to date.

sudo apt-get install libmysqlclient-dev sudo apt-get install libssl-dev sudo apt-get install libkrb5-dev sudo apt-get install libsasl2-dev

Once completed, we need to install PostgreSQL for Airflow. PostgreSQL is a more robust back end database that can be used to create a powerful data pipeline through the UI.

sudo apt-get install postgresql postgresql-contrib

Once this step has completed, run the following command to start the Postgresql service.

sudo service postgresql start

Next, check the status of the cluster and make sure that it is running. Use the following command. You should get an output that looks something like the screenshot below.

pg_lsclusters

We will need to extract a few things from this response for the next piece of code. You want to get the version and the cluster then insert those values in the following piece of code:

sudo pg_ctlcluster <version> <cluster> start

It should confirm that the cluster is running and will look something like this:

Once completed, we can now create a database for Airflow to use. Execute the following command to access psql.

sudo -u postgres psql

Next, we need to create a profile and assign the correct privileges.

CREATE ROLE ubuntu;

CREATE DATABASE airflow;

GRANT ALL PRIVILEGES on database airflow to ubuntu;

ALTER ROLE ubuntu SUPERUSER;

ALTER ROLE ubuntu CREATEDB;

ALTER ROLE ubuntu LOGIN;

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public to ubuntu;

Now we need to setup a password for the ubuntu user:

\password ubuntu

Enter the password, and then again to confirm. Now you can type \q to quit.

Next, we can connect to the Airflow database and verify the connection information. After running the following script you should receive a response that says “You are now connected..”

postgres-# \c airflow

Then run:

\conninfo

Next, hit Ctrl + Z to stop the session and enter the following command to navigate to the config file.

cd /etc/postgresql/*/main/ ls

You should then see a list of files in this directory like the screenshot below:

From here we need to modify some values in this config file to finish setting up Airflow. Use the following command to enter the editor:

sudo nano pg_hba.conf

Modify the line underneath #IPv4 local connections to 0.0.0.0/0

Type Ctrl + S to save and Ctrl + X to exit.

Then, we need to use the same command to open another config file.

sudo nano postgresql.conf

Under the “CONNECTIONS AND AUTHENTICATION” section, modify the following

listen_addresses = ‘*’

Type Ctrl + S to save and Ctrl + X to exit.

Finally, restart PostgreSQL to save and load the changes.

sudo service postgresql restart

*Be sure to allow access in Windows Firewall settings when prompted (if applicable)

Now enter the following command to go back to the root directory

cd ~

Step 4: Installing Apache Airflow

A quick start guide can be found on the Apache Airflow website here.

To install Airflow, run the following command in the terminal:

sudo SLUGIFY_USES_TEXT_UNIDECODE=yes pip install apache-airflow

After that is done installing we can take care of a few things to make sure everything goes smoothly.

We need to add the path to PATH within the terminal window. This allows you to call airflow directly using the airflow command.

Change the <username> parts to your username and then copy the entire line and paste it on a new line within the variables list. Click OK to save the changes.

export PATH=$PATH:/ home/<username>/.local/bin

That’s it! You did it! Apache Airflow is now installed in your environment! You must close and then open a new instance of Ubuntu in order to call Airflow commands.

Note: If you receive an error during install regarding the version of marshmallow you have installed. Run the following commands to install a compatible version.

sudo pip uninstall marshmallow sudo pip install marshmallow==2.19.5

Another Note: The most recent release of WTForms (2.3.0 and higher) will throw an ImportError when attempting to call airflow in the terminal. If you run into this issue you need to uninstall WTForms and reinstall version 2.2.1. The commands below will accomplish this for you.

sudo pip uninstall WTForms sudo pip install WTForms==2.2.1

These last two issues will likely be corrected in future releases, but as of today (4/22/2020) they need to be completed in order to move forward.

Step 5: Apache Airflow Setup

I included the following steps to help first time users of Airflow setup a test environment in which they can explore world of Apache Airflow.

1. The following command will initialize the database.

airflow initdb

Once completed, the necessary config files will be created in the airflow directory. We need to make a couple changes to make sure Airflow is setup correctly. Use the following commands to navigate to and open the config file.

cd airflow ls sudo nano airflow.cfg

Make the following changes to the config file:

dags_folder = /mnt/c/dags base_log_folder = /mnt/c/dags/logs executor = CeleryExecutor load_examples = False expose_config = True sql_alchemy_conn = postgresql+psycopg2://ubuntu:<password>@localhost:5432/airflow broker_url = amqp://guest:guest@localhost:5672// result_backend = amqp://guest:guest@localhost:5672//

**Insert the password you created in the previous step in the <password> section of the sql_alchemy_conn value

Use Ctrl + S to save and Ctrl + X to exit.

Note: The dags and log folder paths above map Airflow to your Windows C: drive. You will need to create two folders. One on your C: drive at C:\dags and a folder inside that folder at C:\dags\logs. You can change the location of these and specify the folders of your choosing. I use the above directory as it is easy to locate and access. You will also avoid any potential permissions issues in this directory.

Once this has been completed run the following:

airflow initdb

If you receive an error relating to the psycopg2 package, run the following commands to correct this issue:

sudo apt-get update -y sudo apt-get install -y libpq-dev pip install psycopg2

Install Rabbitmq

sudo apt install rabbitmq-server

We need to update the config file. Navigate to the rabbitmq config file and change the following:

sudo nano /etc/rabbitmq/rabbitmq-env.conf

Change NODE_IP_ADDRESS=0.0.0.0

Now start the RabbitMQ service.

sudo service rabbitmq-server start

Install Celery

There are newer versions of Celery that are not compatible with airflow. The following command will install a lower version that ensures compatability.

sudo pip install 'celery>=3.1.17,<4.0'

Run Airflow initdb one last time:

airflow initdb

Woohoo! You did it! We can now start up the airflow webserver and scheduler!

Run the first command and let it run. Open a new terminal window and run the second prompt.

airflow webserver -p 8080 airflow scheduler

Once both of these are running successfully, open a tab in your desired browser (Chrome, Firefox, etc.) and enter the following in the url bar:

localhost:8080

When you hit enter, the following page should load:

From here you can now import your dags to the /dags folder and they will appear in Airflow once the db refreshes. For additional help and information on working with Apache Airflow, I recommend looking through the documentation on the Apache Airflow website here.

You can also check out any of the sources I used to compile this tutorial at the links below. Happy coding!

Sources: