Airflow: In a Time when Timing is of the Essence — Part I
By: Bruno Filipe Faustino
Tags: Airflow, Migration, Business Intelligence, Data, AWS

Photograph by Bruno Filipe Faustino

Talkdesk is an event data-driven company. Due to its constant growth and given the predicted general data growth[1], it has a constant need to withstand scalability concerns, sometimes evolving its technological stack. This evolution is aligned with Talkdesk’s goal of handling its own data and extracting knowledge and actionable insights.

After [2], where José Cabeda wrote about Talkdesk’s Business Intelligence (BI) team moving its infrastructure to use PrestoSQL[3] in distinct development environments, one of the open topics was the next steps in improving our BI infrastructure, in order to cope with the evolving architectural needs.

Our Limitations

Until a while ago, our processes orchestration was being handled by Amazon Web Services (AWS) Data Pipelines[4], as described in [2]. However, we were facing some limitations:

Extract, Transform and Load (ETL) : using AWS Data Pipeline we could only preload transformations using SQL commands, while also taking advantage of graphical pipelines to guide our ETL flow.

: using AWS Data Pipeline we could only preload transformations using SQL commands, while also taking advantage of graphical pipelines to guide our ETL flow. Database connectors : plugging new connectors was not a straightforward process when we were plugging in new connectors, e.g., when we made the move to PrestoSQL[2]. AWS Data Pipelines supports a limited list of data node types, e.g.: Amazon DynamoDB[5], SQL and Amazon Redshift[6] tables and Amazon S3[7] locations, but not Software as a Service (SaaS) data sources.

: plugging new connectors was not a straightforward process when we were plugging in new connectors, e.g., when we made the move to PrestoSQL[2]. AWS Data Pipelines supports a limited list of data node types, e.g.: Amazon DynamoDB[5], SQL and Amazon Redshift[6] tables and Amazon S3[7] locations, but not Software as a Service (SaaS) data sources. Pricing : it is based on how often our processes run or are evaluated to run and whether they run on AWS or on-premises.

: it is based on how often our processes run or are evaluated to run and whether they run on AWS or on-premises. Notification and Logging : even though we had Amazon Simple Notification Service (SNS)[8] to support our AWS Data Pipeline, we could not get a clear picture in the notifications sent of where a pipeline had failed and quickly search over its logging.

: even though we had Amazon Simple Notification Service (SNS)[8] to support our AWS Data Pipeline, we could not get a clear picture in the notifications sent of where a pipeline had failed and quickly search over its logging. Reproducibility and Standardization : since AWS Data Pipeline could not be configured via Terraform[9].

: since AWS Data Pipeline could not be configured via Terraform[9]. Regional Availability: not all AWS services were available in all AWS Regions, e.g., AWS Data Pipeline was not available in Canada[10].

Airflow, But Why?

By the end of 2019, we evaluated some options to migrate the engine orchestrating our data pipelines. We looked at Airflow[11], Autosys[12], Dagster[13], Luigi[14] and Prefect[15].

In the first round we discarded three of these options:

Autosys: because it was paid and users were reporting some difficulties[16, 17].

Dagster: because it required Airflow to start.

Luigi: because it did not have a native scheduler.

Bearing in mind the limitations we were trying to tackle, each of these options would require further development to fit our use case.

Our choices came down to Airflow and Prefect, and we decided to go with Airflow at that time. This happened mainly because, unlike Airflow, Prefect seemed to be new and not yet close to production-ready, as we weren’t able to find enough business success cases similar to our use case.

For our use case, one of the major drawbacks in Prefect was the lack of a web server/UI to control the created flows and tasks in the Open Source version — Prefect core. They had a paid cloud service scheduler that uses the Prefect core, but no screens were found at the time and neither was the pricing. Obviously, since we are working with Python, we could always implement a Flask microservice or similar to work on top of the Prefect core, but the added development required made it a non-viable option for us at that time.

Airflow

Airflow is an open-source workflow management platform where we are able to create, orchestrate, schedule, and monitor workflows.

Before jumping in to a brief description of Airflow’s components and how we tackled the migration, there are a few key concepts that help understand how it all pans out:

Direct Acyclic Graph (DAG) : a directed graph with no directed cycles — an object that represents the Workflow.

: a directed graph with no directed cycles — an object that represents the Workflow. DAG Run : represents an execution of a DAG, with its own Execution Date;

: represents an execution of a DAG, with its own Execution Date; Execution Date : represents the timing for a DAG execution taking into account its schedule time. On new executions, the Execution Date will always be the last complete schedule, e.g., if today is August 4th and the DAG has a daily schedule interval, then the DAG will only run for August 3rd.

: represents the timing for a DAG execution taking into account its schedule time. On new executions, the Execution Date will always be the last complete schedule, e.g., if today is August 4th and the DAG has a daily schedule interval, then the DAG will only run for August 3rd. Operator : describes a single Task in our Workflow. Ideally, these should be idempotent, i.e., considering the same data input, the data output should be the same regardless of the time of execution;

: describes a single Task in our Workflow. Ideally, these should be idempotent, i.e., considering the same data input, the data output should be the same regardless of the time of execution; Task : an instance of the Operator.

: an instance of the Operator. Task Instance : a specific run of a Task, i.e., a DAG’s Task in an execution’s point in time.

: a specific run of a Task, i.e., a DAG’s Task in an execution’s point in time. Workflow: represents a combination of all the concepts above.

All of these concepts combined lead to what we know as Airflow and can be looked into in the following diagram (Figure 1):

Figure 1— Airflow’s concepts. Image by Talkdesk.

With that being said, we can jump in to the Airflow components:

Web Server : a UI, where we can start, stop and monitor our Workflows, among other possible operations and visualizations[18].

: a UI, where we can start, stop and monitor our Workflows, among other possible operations and visualizations[18]. Scheduler : responsible for scheduling the Workflows’ Tasks, evaluating their dependencies and their time schedules, i.e.: start date and schedule interval.

: responsible for scheduling the Workflows’ Tasks, evaluating their dependencies and their time schedules, i.e.: start date and schedule interval. Metastore Database : where all data, for instance, on users and Workflows, is stored.

: where all data, for instance, on users and Workflows, is stored. Executor : we have different types of executors[19] to use defining how our Workflows’ Tasks are executed.

: we have different types of executors[19] to use defining how our Workflows’ Tasks are executed. Worker: receives the Workflows’ Tasks’ processes that are scheduled by the Scheduler and sent by the Executor.

Below (Figure 2), Airflow’s architectural diagram with multi nodes is shown — based on [20] — where all concepts above come into play, adapted, and using our chosen technological stack.

Figure 2— Airflow’s architecture with multi nodes. Image by Talkdesk.

As in our use case above, in order to have a distributed Airflow instance, we had to pick one Executor that would allow such functionality. From the ones available, we picked the Celery Executor.

Celery is a distributed system that processes multiple queued messages with real-time focus, supporting also task scheduling. Using this, Airflow’s Workers can be scaled out and be distributed, requiring also a backend for the messages exchange — Redis[21] was chosen.

The decoupled nature of Celery workers was a great fit for our environments. For instance, our production instances run in Kubernetes[22] clusters, and using dumb workers seemed like a better way to have scalable and decoupled workloads being tackled. Furthermore, we gained the possibility of using Kubernetes’s Horizontal Pod Autoscaler (HPA)[23] in order to spawn more workers when needed.

We are aware that Airflow has a Kubernetes executor, but at Talkdesk the Kubernetes clusters are behind a specific developer interface that restricts such interactions. Our option with Celery and Kubernetes’ HPA was a viable and solid choice.

Migration

Architecture

By adding Airflow to our Architecture family portrait, we had to revamp some parts (Figure 3). At the time that this post was written we were using the 1.10.12 version of Airflow:

Figure 3— Talkdesk’s BI team current infrastructure architecture. Image by Talkdesk.

As in [2], we have the User using Looker, which is querying PrestoSQL, instead of querying directly the Data Warehouse, Staging Area, or Data Platform tables. The same happens with Airflow, where we have it querying both PrestoSQL and the PostgreSQL[24] tables. We were forced not to use PrestoSQL for both read and write in our multiple workflows’ SQL queries because, there is a limitation between PrestoSQL and PostgreSQL regarding DELETE SQL operations[25], and this was important to keep our processes idempotent.

We then decided to have our ETL workflows in Airflow reading from PrestoSQL and writing the data to a temporary table in our Data Warehouse in PostgreSQL. This way, all ETL queries that we do after creating the temporary table, could run in-house in the same PostgreSQL database instance, avoiding possible performance overhead issues that arise from having multiple data being pulled and pushed in and out of PrestoSQL to the multiple databases instances that we used at the same time.

The remaining still works as in [2], where we have Fivetran[26] syncing the data into our Staging Area — intermediate tables used by our ETL processes — and have Data Platform service — an in-house service to ingest, process, and make the events generated by Talkdesk’s systems available and accessible to other users or application systems — in this case, to be queried by our ETL processes.

Conclusion

Wrapping it all up at a higher level, we ended up with the following end-to-end flow (Figure 4):

Figure 4— High-level end-to-end flow. Image by Talkdesk.

During the first steps of this migration process, we came across multiple challenges such as making our Continuous Integration/Continuous Delivery (CI/CD) ecosystem more performant during development.

Our CI/CD ecosystem relied on pulling from Git[27] the required code to generate the application image — in this case, Airflow’s Docker image. Since we had all DAG and SQL files in the same repository as the Airflow image content, every time we had to change a DAG, SQL code, or part of an ETL, we had to rebuild the Airflow Docker image and noticed the slowness and risk of always changing it when it wasn’t needed.

With this in mind, and as an alternative to Kubernetes’s Git-sync[28], we decided to move all DAG and SQL files to a separate Git repository and take advantage of our CI/CD to pull them from their repository and store them in S3 as runnable artifacts. Airflow could then download these artifacts, today using AWS CLI[29], and load the DAG and SQL files.

In this blog post we detailed how our infrastructure had to adapt to the inclusion of Airflow. In Part II we will narrow down a few more challenges like the one above and how they helped with our ETL limitations, e.g.: generating team access control and understanding and putting Airflow’s main concepts into practice.

Stay tuned.

Acknowledgments

Thanks to Tiago Caxias and Anaís Dias for the multiple brainstorms over this blog post and all the time spent during its content reviews.

Last but not least, thanks to all Talkdesk’s BI, Site Reliability Engineering (SRE), and Security teams’ members for the effort put forth during this migration development and support.

References

[1] — IDC: The premier global market intelligence company. 2020. Worldwide Global Datasphere Iot Device And Data Forecast, 2019–2023. [online] Available at: <https://www.idc.com/getdoc.jsp?containerId=US45066919> [Accessed 1 September 2020].

[2] — Medium. 2020. Business Intelligence — A Road From Staging To Production. [online] Available at: <https://engineering.talkdesk.com/business-intelligence-a-road-from-staging-to-production-b708a662c668> [Accessed 1 September 2020].

[3] —Prestosql.io. 2020. Presto | Distributed SQL Query Engine For Big Data. [online] Available at: <https://prestosql.io/> [Accessed 15 December 2020]. .

[4] — Construct your ETL with AW Data Pipeline, a., 2020. Amazon Data Pipeline — Managed ETL Service — Amazon Web Services. [online] Amazon Web Services, Inc. Available at: <https://aws.amazon.com/datapipeline/> [Accessed 15 December 2020]..

[5] —Amazon Web Services, Inc. 2020. Amazon Dynamodb | Nosql Key-Value Database | Amazon Web Services. [online] Available at: <https://aws.amazon.com/dynamodb/> [Accessed 15 December 2020]..

[6] — Amazon Web Services, Inc. 2020. Amazon Redshift — Cloud Data Warehouse — Amazon Web Services. [online] Available at: <https://aws.amazon.com/redshift/> [Accessed 15 December 2020]..

[7] —Amazon Web Services, Inc. 2020. Cloud Object Storage | Store & Retrieve Data Anywhere | Amazon Simple Storage Service (S3). [online] Available at: <https://aws.amazon.com/s3/> [Accessed 15 December 2020]..

[8] —Amazon Web Services, Inc. 2020. Amazon Simple Notification Service (SNS) | Messaging Service | AWS. [online] Available at: <https://aws.amazon.com/sns/> [Accessed 15 December 2020]..

[9] —Terraform by HashiCorp. 2020. Terraform By Hashicorp. [online] Available at: <https://www.terraform.io/> [Accessed 15 December 2020]..

[10] —Amazon Web Services, Inc. 2020. AWS Regional Services. [online] Available at: <https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/> [Accessed 1 October 2020] .

[11] — Apache Airflow. 2020. Home. [online] Available at: <https://airflow.apache.org/> [Accessed 15 December 2020]..

[12] —Broadcom.com. 2020. Autosys Workload Automation. [online] Available at: <https://www.broadcom.com/products/software/automation/autosys> [Accessed 15 December 2020]. .

[13] — Docs.dagster.io. 2020. Dagster — A Data Orchestrator For Machine Learning, Analytics, And ETL. [online] Available at: <https://docs.dagster.io/> [Accessed 15 December 2020]..

[14] — GitHub. 2020. Spotify/Luigi. [online] Available at: <https://github.com/spotify/luigi> [Accessed 15 December 2020]..

[15] — Prefect.io. 2020. Prefect — The New Standard In Dataflow Automation — Prefect. [online] Available at: <https://www.prefect.io/> [Accessed 15 December 2020]..

[16] — User, V., 2020. CA Workload Automation Is Reliable And Easy To Use | Trustradius. [online] TrustRadius. Available at: <https://www.trustradius.com/reviews/ca-workload-automation-2018-01-20-14-40-04> [Accessed 20 August 2020].

[17] — Mukherjee, A., 2020. Job Scheduling Software Orchestrates And Automates Business Processes And Tasks From A Single Point Of View. | Trustradius. [online] TrustRadius. Available at: <https://www.trustradius.com/reviews/ca-workload-automation-2018-11-19-12-20-27> [Accessed 20 August 2020].

[18] —Airflow.apache.org. 2020. UI / Screenshots — Airflow Documentation. [online] Available at: <https://airflow.apache.org/docs/apache-airflow/stable/ui.html> [Accessed 7 December 2020] .

[19] — Airflow.apache.org. 2020. Executor — Airflow Documentation. [online] Available at: <https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html> [Accessed 7 December 2020].

[20] — Chattaraj, J., Shrivastava, P. and Villamariona, J., 2020. Understand Apache Airflow’S Modular Architecture | Qubole. [online] Qubole. Available at: <https://www.qubole.com/tech-blog/understand-apache-airflows-modular-architecture/> [Accessed 20 August 2020] .

[21] —Redis.io. 2020. Redis. [online] Available at: <https://redis.io/> [Accessed 15 December 2020]. .

[22] —Kubernetes. 2020. Production-Grade Container Orchestration. [online] Available at: <https://kubernetes.io/> [Accessed 15 December 2020]. .

[23] — Kubernetes. 2020. Horizontal Pod Autoscaler. [online] Available at: <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/> [Accessed 8 October 2020].

[24] — PostgreSQL. 2020. Postgresql. [online] Available at: <https://www.postgresql.org/> [Accessed 15 December 2020]..

[25] —Prestosql.io. 2020. 6.20. Postgresql Connector — Presto 329 Documentation. [online] Available at: <https://prestosql.io/docs/329/connector/postgresql.html> [Accessed 26 August 2020] .

[26] — Fivetran | Data to Snowflake, Redshift, BigQuery, Azure. 2020. Fivetran | Data To Snowflake, Redshift, Bigquery, Azure. [online] Available at: <https://fivetran.com/> [Accessed 15 December 2020]..

[27] —GitHub. 2020. Github: Where The World Builds Software. [online] Available at: <https://github.com/> [Accessed 15 December 2020]..

[28] —GitHub. 2020. Kubernetes/Git-Sync. [online] Available at: <https://github.com/kubernetes/git-sync> [Accessed 7 December 2020] .

[29] — Amazon Web Services, Inc. 2020. AWS Command Line Interface. [online] Available at: <https://aws.amazon.com/cli/> [Accessed 3 December 2020].