Airflow: How To Refresh Stock Market Data While You Sleep — Part 1 | by AnBento | Towards Data Science

Apache Airflow is a workflow management platform that allows users to programmatically schedule jobs that run on a standalone basis and to monitor them through its user interface. Airflow’s main components are a Webserver (Airflow’s UI build as a Flask App), a Scheduler, an Executor and a Metadata Database.
As a BI Developer I worked close enough to data engineers to learn pretty early what Airflow was very good at, but I never really put the theory into practice until I started with side projects.
In effect, while I was building an app to visualize stock prices and stock stats with Dash, I realized that every time I wanted data to be updated, I had to re-run the entire code manually. This process takes a very long time when you need to extract, manipulate and make available data for hundreds of stock tickers. I knew that if I wanted to scale up my project, I needed a tool to orchestrate the repetitive tasks behind the scenes while I was focused on building new interesting features.
In the first part of this tutorial, I will show you how I eventually started to combine Python and Airflow to create an automated ETL pipeline that:
Downloads stock prices from Yahoo Finance and saves them into a list;Performs advanced manipulation on the stock prices to generate a Stock Prices History DF and a Stock Prices Statistics DF to be used in different parts of my project;Save data as CSV files in a specific folder.
Downloads stock prices from Yahoo Finance and saves them into a list;
Performs advanced manipulation on the stock prices to generate a Stock Prices History DF and a Stock Prices Statistics DF to be used in different parts of my project;
Save data as CSV files in a specific folder.
But first let’s activate a virtual environment and install Airflow.
For this tutorial I’ll assume you are familiar with virtual environments and that you are using Python 3.6 or above as I found the installation being buggy otherwise (simply use python -V if you wish to double check):
Firstly, activate your virtual environment, then run pip install apache-airflow. To verify that the installation was successful type the command airflow version .Now navigate to Airflow main directory with cd ~/airflow/ and create a new folder named “dags” with mkdir dags. In Airflow a DAG (or Directed Acyclic Graph) consists of a number of operators that describe the individual tasks that need to be performed to achieve a specific goal. For example if you wished to run a Python function you would use the Python Operator , whereas if you wished to execute a Bash command you would use the BashOperator and so on. For a complete list of available operators have a look at the official documentation.Call the airflow initdb command to initiate the SQLite database where Airflow will store the metadata that are necessary to manage your workflow. Note that a SQLite database is not an optimal choice in a production environment, but it comes handy when you are getting started with Airflow as it is really easy to configure in local.A number of new files will appear in the airflow main folder among which airflow.cfg . This file includes all the configuration options that will be applied to the web interface and to your DAGs. I highly recommend to open the file with an editor and set load_examples=False . The option is set to True by default and unless it is changed, you will see a long list of default DAG examples on the UI every time you run the Webserver and this in my opinion is a bit confusing and untidy.Create an empty file with .py extension that will include your original Python Script + DAG Schedule + DAG Operators + Tasks Hierarchy. Basically, because Airflow is Python-based you can simply embed your DAG in a familiar Python script and leave its execution to Airflow’s Scheduler (more on this later). For this tutorial, I initially created an empty file named stocks_analysis.py .
Firstly, activate your virtual environment, then run pip install apache-airflow. To verify that the installation was successful type the command airflow version .
Now navigate to Airflow main directory with cd ~/airflow/ and create a new folder named “dags” with mkdir dags. In Airflow a DAG (or Directed Acyclic Graph) consists of a number of operators that describe the individual tasks that need to be performed to achieve a specific goal. For example if you wished to run a Python function you would use the Python Operator , whereas if you wished to execute a Bash command you would use the BashOperator and so on. For a complete list of available operators have a look at the official documentation.
Call the airflow initdb command to initiate the SQLite database where Airflow will store the metadata that are necessary to manage your workflow. Note that a SQLite database is not an optimal choice in a production environment, but it comes handy when you are getting started with Airflow as it is really easy to configure in local.
A number of new files will appear in the airflow main folder among which airflow.cfg . This file includes all the configuration options that will be applied to the web interface and to your DAGs. I highly recommend to open the file with an editor and set load_examples=False . The option is set to True by default and unless it is changed, you will see a long list of default DAG examples on the UI every time you run the Webserver and this in my opinion is a bit confusing and untidy.
Create an empty file with .py extension that will include your original Python Script + DAG Schedule + DAG Operators + Tasks Hierarchy. Basically, because Airflow is Python-based you can simply embed your DAG in a familiar Python script and leave its execution to Airflow’s Scheduler (more on this later). For this tutorial, I initially created an empty file named stocks_analysis.py .
Find below the bash commands that summarize what has been done so far:
(airflow_ve) AnBento-2:airflow $ mkdir dags(airflow_ve) AnBento-2:airflow $ airflow initdb(airflow_ve) AnBento-2:airflow $ lsairflow-scheduler.err airflow-scheduler.pid airflow-webserver.out airflow.db unittests.cfg airflow-scheduler.log airflow-webserver.err airflow-webserver.pid dags airflow-scheduler.out airflow-webserver.log airflow.cfg logs(airflow_ve) AnBento-2:airflow $ open airflow.cfg(airflow_ve) AnBento-2:airflow $ cd dags(airflow_ve) AnBento-2:dags $ touch stocks_analysis.py(airflow_ve) AnBento-2:dags $ ls__pycache__ stocks_analysis.py
If you followed all the steps, you should now be ready to launch Airflow Webserver and Scheduler, but before that, you need to fill the stock_analysis.py file with a DAG.
The full code I am going to use in this tutorial to refresh stock prices, set a DAG and its operators, is available on my GitHub account.
If you are following along it’s now time to edit the empty file with open stock_analysis.py and paste the entire Python script you find in the code_tutorials/Airflow Stock Prices ETL folder. After saving the changes and before doing anything else, make sure to install all the following packages in the environment:
Airflow should now be able to recognize a valid DAG and you can easily check it with airflow list_dags :
(airflow_ve) AnBento-2:dags $ airflow list_dags[2020–06–27 09:40:35,002] {__init__.py:51} INFO — Using executor SequentialExecutor[2020–06–27 09:40:35,002] {dagbag.py:396} INFO — Filling up the DagBag from /Users/anbento/airflow/dags — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -DAGS — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -stocks_analysis_ETL_7AM
The screen above did not throw an error and you can tell that a DAG was in fact identified by the system because stock_analysis_ETL_7AM is exactly the name I assigned to the DAG:
In effect, I would suggest to assign a meaningful name to your DAGs (following the convention project_name + process + schedule interval ) as it helps to visualize immediately which job you are looking at and when it is supposed to run and this adds clarity when dealing with multiple workflows.
The code above tells Airflow when to schedule the job, what the job needs to accomplish and in which order. It can be split into three components, starting from the top:
Airflow DAG + Default Args: here you will specify the job name, the start_date and the schedule_interval together with the additional default_args that are defined in the Python dictionary. Here are a few suggestions that will spare you a lot of headaches and hours spent reading countless tutorials and Stack Overflow posts:
Airflow DAG + Default Args: here you will specify the job name, the start_date and the schedule_interval together with the additional default_args that are defined in the Python dictionary. Here are a few suggestions that will spare you a lot of headaches and hours spent reading countless tutorials and Stack Overflow posts:
Set a static start_date in the past like for example start_date = datetime(2020, 6, 23) → In effect, in order to work properly, Airflow requires for any jobs to have a start date that is already in the past and in order to avoid unpredictable behavior it is better to use a static syntax instead of something like start_date = datetime.now()- timedelta(days=3) .
Set catchup = False → In this way you are making clear that despite your start_date is in the past, you do not wish for Airflow to backfill your data, meaning that when you run it for the first time, the job will not be executed multiple times to cover all the missing runs in between the start_date and the date when the DAG was switched on. Instead, Airflow will run it only once (immediately) and after that moment it will stick to the schedule.
Change the DAG Name when you change the schedule_interval → If you wish to change the run time from 7 AM to let’s say 8 AM, it is good practice to also change the DAG name to stock_analysis_ETL_8AM . Airflow will treat this DAG as a brand new one, in this way avoiding to have two DAGs with the same name stored in the metadata database...and yes, you will be able to delete the old DAG from the UI.
2. DAG Operators: for this tutorial, I am going to use three PythonOperators to run three Python functions. Each Python operator is a separate task that needs to be passed to Airflow Sequential Executor in order to achieve the expected result. Each task is identified by a task_id and in order to avoid confusion I would suggest for you to make sure that task name and task id always match. In my case the tasks are:
fetch_prices_task → this task will call a Python function named fetch_prices_function() that downloads the full price history for the 5 stocks in the tickers list from Yahoo Finance, removes duplicates, applies simple manipulation and saves them in the stock_prices list:
If you run the function it should return something similar to this:
Because I am going to use the stock_prices list as an input for the other two Python functions, this is is the first task that needs to be executed in the order. The task execution hierarchy is important as it will allow me not to download stock prices at each step, but to do it only once (at beginning), so that each task will run faster and the overall process will be more efficient.
stocks_plot_task → this task will call the second Python function named stocks_plot_function() that takes the stock_prices list as an input and concatenates each stock price history in a single DataFrame that will then be used to plot stock prices trends over time. Data is finally saved in a CSV file named stocks_plots_data.csv . You can of course decide to take a more advanced step and load data incrementally into a DB of your choice - this i s actually the right approach - but to keep things simple and focus on Airflow capabilities, let’s stick to CSV files for now:
When the function is run, it displays the first 5 rows of the stock_plots_data DF:
stocks_table_task → this final task will call the third Python function named stocks_table_function() that again takes the stock_prices list as an input by keeping just the last available Adj Close price then fetches a number of statistics from Yahoo Finance. Eventually, the output is saved to a CSV file named stocks_table_data.csv :
The output of the function above is a DF with 5 rows and 13 columns of which 11 are key statistics:
3. Task Hierarchy → This is the last component of the DAG and it is used to specify the order according to which you wish for Airflow to execute the three tasks. In my case I opted for:
fetch_prices_task >> stocks_plot_task >> stocks_table_task
Meaning that the fetch_prices_task will be executed first (it makes sense as it generates the stock_prices list that is then used in the other two tasks), followed by the stocks_plot_task and the stocks_table_task .
In this case, you can also decide to combine the second and third tasks, since they are not dependent on each other:
fetch_prices_task >> [stocks_plot_task, stocks_table_task]
Note how in Airflow dependencies can be set syntactically (set_upstream() , set_downstream() ) or using bit-shift operators like I did.
If you tried to test the functions as they are in a Python IDE, you probably noticed that they simply...fail!. This is because in order for one task output to be called as an input in other tasks down the line, Airflow uses an inter-task communication Python class named XCom. Therefore I had to modify the original functions by adding:
A keyword argument (**kwargs) for all three of them;
The following two lines for the stocks_plot_function() and stocks_table_function():
ti = kwargs[‘ti’]stocks_prices = ti.xcom_pull(task_ids=’fetch_prices_task’) 
provide_context = True for each Python Operator.
In effect, as the documentation (pretty concisely...) explains, a task can make an output available by calling the xcom_push() method or simply by returning the output at the end of the function it calls (like I am doing with return stock_prices ).
In a similar fashion, a task down the line can fetch that output by calling the xcom_pull() method specifying exactly the task from which the input should be retrieved ( task_ids ='fetch_prices_task' ).
Yeah I know...if you are using Airflow for the first time you might be wondering why you cannot just define stock_prices=[] outside the fetch_prices_function() and use it as a global list in the other two functions. The thing is that Airflow parses Python code all in one go, therefore it needs a more complex system for the tasks to “communicate” with each other. If you still have doubts on the topic this is another excellent post that helped me in the recent past.
The Final Step: How to activate the Stocks Analysis ETL job
Now that you have a grasp of what the code is doing and how a DAG is built, it’s time to open two separate Bash windows. On the first window you should run the command to launch the Airflow Webserver:
(airflow_ve) AnBento-2:airflow $ airflow webserver
Whereas on the second window you should launch the Airflow Scheduler:
(airflow_ve) AnBento-2:airflow $ airflow scheduler
If you navigate to the default http://0.0.0.0:8080/admin/ local address you will hopefully see the screen below:
Airflow is correctly displaying the stocks_analysis_ETL_7AM DAG but in order for the process to start, the job needs to be switched on manually. As soon as you do that, Airflow Sequential Executor will immediately run all the three tasks according to their hierarchy:
The dark-green color indicates that all the 3 tasks were successfully executed and you can verify it by clicking on Recent Tasks:
This means that 2 updated CSV files containing stocks data have been saved to the specified location. If something went wrong and you wished to double-check the code, you can do that within the user interface by clicking on DAG_name > Code:
Congratulations! If you were so motivated to get to the end of PART 1 of this Airflow tutorial, you should now be able to build your first ETL local pipeline and employ it to refresh data behind your personal projects. You have also learnt how to use the XCom class to dynamically connect tasks and I hope this knowledge will come handy as you start building more complex DAGs.
In PART 2, I will show you how to deploy Airflow DAGs in production so that they can really run while you sleep, but before that, have a look at these last two suggestions:
Building an ETL Pipeline in local means that unless your machine is switched on, DAGs will not run at the scheduled time. In effect in order for Airflow to work properly, both webserver and scheduler need to be running in the background. If you keep them running and your system goes idle, Airflow will cover the jobs it missed during the period of inactivity as soon as you re-activate the system. However, deploying DAGs to production will solve this issue.
Understanding the difference between Execution Date and Start Date and the way Airflow executes workflows is often not straightforward. To help you get the gist, have a look at this article that addresses the topic in depth.
Please let me know if you liked the tutorial in the comments and I hope to see you in Part 2!