Creating an environment with Airflow and DBT on AWS (part 1)
By: Arthur Chaves
Tags: Dbt, AWS, Data Engineering, Big Data, Airflow

2. Installing Airflow and its requirements

First, let’s install the OS prerequisites. After some experiments, the packages on the command below seems to be necessary and enough to do everything we want to:

$ sudo yum -y install wget tar intltool icu4c gettext icu libicu-devel gcc-c++ python3-devel redhat-rpm-config gcc libffi-devel python-devel openssl-devel gzip gcc make expect openssl-devel bzip2-devel libffi-devel

Before we install Airflow, we need to upgrade the sqlite version. Here are the commands. If you have some error with the link of sqlite, check in its page, cause it could be updated.



$ tar xzf sqlite.tar.gz

$ cd sqlite/

$ export CFLAGS="-DSQLITE_ENABLE_FTS3 \

-DSQLITE_ENABLE_FTS3_PARENTHESIS \

-DSQLITE_ENABLE_FTS4 \

-DSQLITE_ENABLE_FTS5 \

-DSQLITE_ENABLE_JSON1 \

-DSQLITE_ENABLE_LOAD_EXTENSION \

-DSQLITE_ENABLE_RTREE \

-DSQLITE_ENABLE_STAT4 \

-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \

-DSQLITE_SOUNDEX \

-DSQLITE_TEMP_STORE=3 \

-DSQLITE_USE_URI \

-O2 \

-fPIC"

$ export PREFIX="/usr/local"

$ LIBS="-lm" ./configure -disable-tcl -enable-shared -enable-tempstore=always -prefix="$PREFIX"

$ make

$ sudo make install $ wget https://www.sqlite.org/src/tarball/sqlite.tar.gz $ tar xzf sqlite.tar.gz$ cd sqlite/$ export CFLAGS="-DSQLITE_ENABLE_FTS3 \-DSQLITE_ENABLE_FTS3_PARENTHESIS \-DSQLITE_ENABLE_FTS4 \-DSQLITE_ENABLE_FTS5 \-DSQLITE_ENABLE_JSON1 \-DSQLITE_ENABLE_LOAD_EXTENSION \-DSQLITE_ENABLE_RTREE \-DSQLITE_ENABLE_STAT4 \-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \-DSQLITE_SOUNDEX \-DSQLITE_TEMP_STORE=3 \-DSQLITE_USE_URI \-O2 \-fPIC"$ export PREFIX="/usr/local"$ LIBS="-lm" ./configure -disable-tcl -enable-shared -enable-tempstore=always -prefix="$PREFIX"$ make$ sudo make install

Now, open the /etc/environment file:

$ sudo vi /etc/environment

And add these lines, so the correct version of SQLite is loaded. When you execute the command above, a blank file is open in the terminal. If you are using vi (as in the above command) to enter the edition mode, press ‘I’. Then, digit or paste the lines below. Press ESC to exit the edition mode. And then digit ‘:w’ and Enter to save the file. Now, digit ‘:q’ and Enter to exit the file. It may be a little tricky at first, but you get used to it.

export LD_LIBRARY_PATH="/usr/local/lib"

export LD_RUN_PATH="/usr/local/lib"

And run the following command to load the modified file:

$ source /etc/environment

Check the sqlite version on the OS and the one recognized by Python with these commands:

$ sqlite3 --version

$ python3 -c "import sqlite3; print(sqlite3.sqlite_version)"

If everything worked fine, both will return a version 3.36 (or above, depending on when you do the procedure).

Checking the SQLite verion on the OS and on Python

Now, let create a user to own the Airflow installation:

$ sudo adduser airflow

And define a password for the user with the command:

$ sudo passwd airflow

I’m proceeding with the password “airflow”, but feel free to choose something better than that. Just pay attention if I use the password somewhere and you have a different password, switch with yours.

Add the new user to sudoers file, using the command below to edit the file:

$ sudo vi /etc/sudoers

Search for the line “root ALL=(ALL) ALL”. Copy this line and paste below, switching the “root” part to “airflow” (the line will be: airflow ALL=(ALL) ALL).

Adding the airflow user to the sudoers file

Switch to the new user with the command and then the password:

$ su airflow -

Let’s create one directory for airflow installation, one for virtual environments and then one venv to install airflow on it:

$ mkdir ~/airflow

$ mkdir ~/.venv

$ python3 -m venv ~/.venv/airflow

Activate the new environment with:

$ source ~/.venv/airflow/bin/activate

Now, run these two commands to install more dependencies needed by Airflow:

$ sudo python3 -m pip install -U pip

$ sudo python3 -m pip install -U setuptools

Now, we are going to install Airflow. From the command in the Apache Airflow page, we have to adapt the python to python3 and pip to pip3. So, we have the following:

$ AIRFLOW_VERSION=2.0.1

$ PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

$ export AIRFLOW_HOME=~/airflow

$ CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

$ pip3 install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Now, go to the Airflow Home directory and initialize the airflow metadata database:

$ cd ~/airflow

$ airflow db init

If everything worked correctly, you should see files like airflow.cfg and airflow.db.

The files in the airflow directory after we initialize it

Now, to run Airflow on the simplest way, we need to run two services. One of them is the scheduler. To do this, just run this on terminal:

$ airflow scheduler

When you run this command, your terminal will be “locked” with all the output from the service. Since we still need to run the second service, we need to open a second terminal. In VSCode, we just need to go to the Terminal menu and select New Terminal. The first terminal will keep running and a fresh new one will be opened. If you are following the whole procedure since the beginning, the remote-ssh connection will be made again with the ec2-user. We need to switch to the airflow user and activate the virtual environment we created:

$ sudo su airflow -

$ cd ~

$ source ~/.venv/airflow/bin/activate

And finally, run the second service we need:

$ airflow webserver

We have enough to check if Airflow is running. Go to your browser, and use as the URL the IP or the hostname of your instance with the port 8080 (http://#.##.###.###:8080). It can take some seconds until the application is accessible. If you have some error using this address, check if your browser is trying to use https instead of http. It has to be http.

The airflow initial page

Now, even if the Airflow page was opened correctly, we still cannot log in. We need to create a user in order to do that. Since the two terminals we used are running services, we need to open one more, to run commands on the instance again. Open the new terminal, go back to our user and python environment:

$ sudo su airflow -

$ cd ~

$ source ~/.venv/airflow/bin/activate

Now run the airflow command below to create the user:

$ airflow users create -u admin -p admin -f FirstName -l LastName -r Admin -e name@mail.com

Here you can switch the parameters to user you preferred username, password etc.

Now, go back to your browser and use the login and password you chose. You will see a list of example DAGs.