Master Apache Airflow: How to Install and Setup the Environment in 10 Minutes | by Dario Radečić | Towards Data Science

Apache Airflow is an open-source workflow management platform (source), and is a must-know technology for modern-day data scientists and engineers.
Today you’ll learn how to install the platform on a virtual machine, configure it, and establish an SSH connection through Terminal and Visual Studio Code.
The article is structured as follows:
Step 1: Download and Install Ubuntu Server
Step 2: Establish an SSH Connection Through Terminal
Step 3: Install Apache Airflow
Step 4: Establish an SSH Connection Through Visual Studio Code
Conclusion
Let’s start with the easiest step. I assume you already have VirtualBox installed. If that’s not the case, please install it before proceeding.
We’ll start by downloading the Ubuntu Server ISO file. It’s around 1GB download, so it shouldn’t take too long:
While it’s downloading, you can configure the virtual machine. Go over to VirtualBox and create a new machine according to the following image:
From there, just click on Next a couple of times and choose an appropriate RAM and storage configuration. I’ve gone with 4GB for the RAM and 25GB for the storage — but feel free to allocate differently.
Once done, you’ll have a new virtual machine on your list:
Let’s tweak a couple of more things while the ISO is downloading. Open the Settings for the new machine and go to Network. Under Advanced, click on the Port Forwarding button:
We’ll need to forward two ports:
Guest 8080 to Host 8250 (or any other, the documentation recommends 8080, but that one is occupied on my machine) for Airflow UIGuest 22 to Host 2222 for SSH connection
Guest 8080 to Host 8250 (or any other, the documentation recommends 8080, but that one is occupied on my machine) for Airflow UI
Guest 22 to Host 2222 for SSH connection
Once done, your window should look like this:
And that’s all — configuration-wise. The ISO file has hopefully been downloaded by now, so run the virtual machine and load in the ISO file for the installation. Install the OS as you normally would. You’ll need to create a user — I’ve called mine airflow.
Here’s a screenshot from the installation:
If you see something similar, then you’re on the right track. Once the installation finishes, you can restart the VM and log in with your user. And that’s it — Ubuntu Server installed!
We can now establish an SSH connection from Terminal (or PuTTY if you’re on Windows).
For macOS and Linux, you’ll have to enter the following line into the Terminal:
ssh -p 2222 airflow@localhost
As shown on the image below:
Keep in mind that airflow indicates the username you have to your user upon installation.
The installation of Apache Airflow is a multi-step process. The whole thing is Python-based, and Ubuntu Server doesn’t ship with Python 3. The following command will change that:
sudo apt install python3-pip
Now you have Python 3.8.x installed (or some newer version), so you’re ready to install Airflow. The following Terminal command will do just that:
sudo pip3 install “apache-airflow==2.0.2” — constraint “https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.8.txt"
Please keep in mind:
Replace 2.0.2 with the version you want
Relace constraints-3.8.txt with your Python version. For example, if you’re using Python 3.9.x, it should be: constraints-3.9.txt
Airflow installed! But that doesn’t mean it’s configured. You’ll also need to initialize the database and create the user. The following command initializes the database for Airflow:
airflow db init
And the following one creates the user:
airflow users create — username admin — firstname <your_first_name> — lastname <your_last_name> — role Admin — email <your_email>
Once done, you can run both Airflow Webserver as a daemon with the following commands:
airflow webserver -D
And that’s it — Apache Airflow is now running!
To verify, open a web browser and go to localhost:8250 (replace 8250 with a host port for Airflow). You should see the following screen:
To continue, enter the database user credentials created a minute ago and click on Sign In. You’ll immediately be presented with the following screen:
And that’s it — you’re in! Don’t worry about a ton of DAGs that are loaded initially. You can quickly delete them from the web application or specify not to load them in the Airflow configuration.
Up next, let’s see how to establish a remote SSH connection to your virtual machine from Visual Studio Code.
If you’re anything like me, a manual file transfer from the host to the client is just too much work. Luckily, a free code editor — Visual Studio Code — can manage the connection for you.
To start, install a free plugin called Remote — SSH:
From there, press on the little blue button on the bottom left corner, and choose the Connect to Host option:
Up next, enter the connection command identical to the one you’ve entered in Terminal:
Visual Studio Code will ask you to add the connection configuration to your list of SSH configurations, so select the first option:
Once again, click on the little blue button to connect to the localhost:
Once you do so, a new Visual Studio Code window will pop up, and you’ll have to enter the password. It’s the same password you’ve chosen for the Ubuntu Server user:
And finally, you can click on the Open Folder button to open any directory on the remote host. Let’s open up the /home/airflow folder:
And that’s it — you’re ready to create your first Airflow DAG. Make sure to put them inside the dags folder (you’ll have to create it first), as that’s where Airflow will try to find them. You can always alter the configuration file, but that’s out of the scope for today.
Let’s wrap things up in the next section.
And there you have it — Apache Airflow installed inside a virtual machine. You could install Airflow locally or with Docker, but I’ve found this approach more common in my professional life.
Up next, we’ll explore how to create and schedule your first DAG, so stay tuned to the blog if you want to learn more.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.
medium.com
Top 3 Reasons Why I Sold My M1 Macbook Pro as a Data Scientist
How to Schedule Python Scripts With Cron — The Only Guide You’ll Ever Need
Dask Delayed — How to Parallelize Your Python Code With Ease
How to Create PDF Reports With Python — The Essential Guide
Become a Data Scientist in 2021 Even Without a College Degree
Follow me on Medium for more stories like this
Sign up for my newsletter
Connect on LinkedIn