Running R scripts in Airflow using Airflow BashOperators
By: Laura Calcagni
Tags: Data Science, R, Airflow

Getting started

First of all, create a virtual environment (with Python and R) and activate it. In this case, I created it using anaconda:

cd path/to/ Airflow-R-tutorial

conda create -n my_airflow_env r-essentials r-base

conda activate my_airflow_env

Now, install apache-airflow:

sudo pip3 install apache-airflow

(Note: It is possible that you'll need to execute sudo pip3 install SQLAlchemy==1.3.18 to override the newer SQLAlchemy version.)

It is necessary to create the AIRFLOW_HOME directory and set the environment variable AIRFLOW_HOME . To be more explicit, I called this directory airflow_home :

cd path/to/ Airflow-R-tutorial

mkdir airflow_home

export AIRFLOW_HOME=`pwd`/airflow_home

Now check if everything is ok:

airflow version

This will create some files in the directory that you designed as AIRFLOW_HOME .

Run the following to start the database:

airflow db init

You will see a new file airflow.db in your home directory.

Finally, create a user:



--username admin \

--firstname FIRST_NAME \

--lastname LAST_NAME \

--role Admin \

--email airflow users create \--username admin \--firstname FIRST_NAME \--lastname LAST_NAME \--role Admin \--email admin@example.org

Creating your first DAG

Now we will create our first DAG containing four tasks (A, B, C and D). In this case, each task will be an R script.

Folders and files

Create the dags and scripts directories inside the airflow_home directory:

cd path/to/ Airflow-R-tutorial /airflow_home

mkdir dags

mkdir scripts

Now create the following files inside them:

cd path/to/ Airflow-R-tutorial /airflow_home/dags

touch my_first_dag.py cd path/to/my_airflow_project/airflow_home/scripts

touch A_task.R

touch B_task.R

touch C_task.R

touch D_task.R

touch run_r.sh

The tasks

For this tutorial, I have chosen four simple tasks that may not be useful but will serve as an example. In this subsection, I will create each task as an R script.

Task A ( A_task.R file): executes a GET request to this Random User Generator API (a free and open-source API for generating random user data) and obtains a list of 200 random users from Canada. Then exports the results as a users.csv file.

library(httr)

library(jsonlite) res = GET(

url = “https://randomuser.me/api/",

query = list(

results=200,

nat="ca",

inc="gender,name,dob"

)

) data = fromJSON(content(res, "text")) write.csv(data$results,”users.csv”, row.names = FALSE)

Task B ( B_task.R file): opens the users.csv file and builds a bar plot(counts by gender). Exports the result as a counts_by_gender.png file.

library(ggplot2)



# Load data

data <- read.csv("./airflow_home/scripts/users.csv", header=TRUE)



# Barplot

p <- ggplot(data, aes(x = as.factor(gender), fill = gender)) +

geom_bar(stat = "count", position = "stack") +

labs(x = "Gender",

y = "Count")



png("./airflow_home/scripts/counts_by_gender.png")

print(p)

dev.off()

Task C ( C_task.R file): opens the users.csv file and builds a bar plot(counts by age). Exports the result as a counts_by_age.png file.

library(ggplot2)



# Load data

data <- read.csv("./airflow_home/scripts/users.csv", header=TRUE)



# Barplot

p <- ggplot(data, aes(x = as.factor(dob.age))) +

geom_bar(stat = "count", position = "stack", fill = "#FF6666") +

labs(x = "Age",

y = "Count")



png("./airflow_home/scripts/counts_by_age.png", width = 800, height = 400)

print(p)

dev.off()

Task D ( D_task.Rmd file): opens the files generated by tasks B and C and builds a simple report html report.

---

title: “Simple Report”

output: html_document

--- ```{r setup, include=FALSE}

knitr::opts_chunk$set(echo = TRUE)

``` ### Counts by gender

![](counts_by_gender.png) ### Counts by Age

![](counts_by_age.png)

The tasks dependence is shown in the following picture: task B and C will only be triggered after task A completes successfully. Task D will then be triggered when task B and C both complete successfully.

Bash scripts and commands

The contents of the run_r.sh file will be useful to run the A, B and C tasks:

#!/usr/bin/env Rscript args = commandArgs(trailingOnly=TRUE)

setwd(dirname(args[1])) source(args[1])

(Take into account that you may need to change the interpreter on line #!/usr/bin/env Rscript )

Now you can execute the scripts containing tasks A, B and C with the following commands:

cd path/to/ Airflow-R-tutorial /airflow_home/scripts

chmod u+x run_r.sh ./run_r.sh path/to/scripts/folder/A_task.R # To run task A

./run_r.sh path/to/scripts/folder/B_task.R # To run task B

./run_r.sh path/to/scripts/folder/C_task.R # To run task C

To run task D use:

Rscript -e "rmarkdown::render('D_task.Rmd')"

The workflow

Now that we have all the necessary pieces, let's start coding our DAG.

The contents of the file my_first_dag.py are the following:

import airflow

from airflow.models import DAG

from airflow.operators.bash import BashOperator

import os # Get current directory

cwd = os.getcwd()

cwd = cwd + '/airflow_home/scripts/' # Define the default arguments

args = {

'owner': 'your_name',

'start_date': airflow.utils.dates.days_ago(2),

} # Instantiate the DAG passing the args as default_args

dag = DAG(

dag_id='my_dag_id',

default_args=args,

schedule_interval=None

) # Define the 4 tasks:

A = BashOperator(

task_id='A_get_users',

bash_command=f'{cwd}run_r.sh {cwd}A_task.R ',

dag=dag,

) B = BashOperator(

task_id='B_counts_by_gender',

bash_command=f'{cwd}run_r.sh {cwd}B_task.R ',

dag=dag,

) C = BashOperator(

task_id='C_counts_by_age',

bash_command=f'{cwd}run_r.sh {cwd}C_task.R ',

dag=dag,

) command_line = 'Rscript -e "rmarkdown::render('+ "'" + f'{cwd}D_task.Rmd' + "')" + '"'

D = BashOperator(

task_id='D_html_report',

bash_command=f'{command_line} ',

dag=dag,

) # Define the task dependencies

A >> B

A >> C

[B, C] >> D

Basically, I defined the default_args and then instantiate the DAG. After that, I generated the tasks by instantiating Bash Operators objects and finally, I set up the task dependencies rules.

For more information about how to create DAGs I suggest that you visit this link.

Run your DAG

It's time to access the Airflow UI:

airflow webserver

Note: You can start this as a daemon by adding -D to your command.

Go ahead and open localhost:8080 to access the Airflow UI.