Data Traffic Control with Apache Airflow
Tags: Workflow, Apache Airflow, Data Science, Docker

by Nicolas GOLL PERRIER (Data Engineer)

Leboncoin is the 7th most accessed website in France; this web traffic activity generates a lot of different events that need to be collected, transformed, reconciled, analysed and understood in order to fully grasp our customers and market, and improve upon our classified ads services.

In order to handle the diversity, volume, and complexity of events, we build workflows to ensure that our data is processed in an orderly and coherent way.

A good analogy for this job is the flight traffic control at the core of any airport. Each transiting dataset is like a plane carrying valuable information requiring to go from point A to point B. Our responsibility is to ensure on time and accurate transit of those “planes”, but we can neither control the “weather”, nor the behavior of those precious “passengers”.

So we need to be vigilant and establish the best flight paths for the streams and batches of data we ingest and transform, re-shuffle the priorities, stall some in-flight datasets at a certain “altitude” (postponing non critical information availability), while re-allocating “fuel reserves” (CPU, Memory, Bandwidth…) depending on the actual and current business priorities of the company.

From Airstrip to Airport

In the beginning, we were able to somewhat cope with the growth of the site with traditional Business Intelligence tools and practices (good old Extract Transform Load tools, cron-based orchestration, limited dashboards…). But as Leboncoin kept growing, we had to adapt our data-operations accordingly.

Therefore, we had to completely rethink our data architecture several years ago to:

Scale our data pipelines to process workloads of up to several terabytes a day efficiently

Adapt to the platform’s technological and organisational shift to a micro-service architecture

Quickly detect failures and inconsistencies in the many data processes run each day/hour

Respond to internal or external faults without impacting the quality, conformity, and availability of actionable information to our business users

Flying through the data-hurricane

Nowadays, we have different kinds of data processing batches running every hour of the day: Spark transformations, Kafka topics S3 archiving processes, Data-Warehouse loads, dataset indexing, business extracts, regulatory extracts, Machine Learning training and predictions… Too many to manually ensure that all our data stores are properly up to date and missing no data-points.

Moreover, data engineering and analytics are downstream the information system. Every incident, failure, bug, inconsistency will at some point end-up in our pipelines: so some input dataset may have changed its format without notice, a third-party data-source might become unavailable during the night, a real-life event triggered an unexpected increase in a specific customer activity, which produces more events and takes longer to process…

An information system is a living organism. We might prevent some of those things from happening, and we have contingency plans, but we can’t predict everything, especially in a quickly growing and evolving organisation.

So when dealing with our large set of data transformations, we need to guarantee to our customers, both internal and external, that those transformation will run consistently and quickly enough to provide relevant, accurate, and up to date insights. And if something were to fail, quickly assess the impact of the failure on all depending systems, reports, and dashboards.

Several tools such as Amazon S3, Apache Spark or Redshift have helped us tackle most the scaling issues, but as data processing workflows kept accumulating and were getting more and more complex, we needed a proper way to ensure that those ran like clockwork. And we were fortunate enough to try on Apache Airflow early on, just after it was open-sourced.

Airflow: Pre-flight checklist

Airflow has been developed with data-engineering challenges in mind. It is above all a DAG (Directed Acyclic Graph, a fancy way of saying “a workflow that does not loop”) management platform, or a worfklow orchestrator if you prefer.

Airflow

Airflow enables you to define, schedule, run, monitor and manage your workflows in pure Python code, while also providing the tools and UI to handle those workflow operations

It provides:

Retry mechanisms to ensure that each and every anomaly can be detected, and automatically or manually healed over time (with as little human intervention as possible)

to ensure that each and every anomaly can be detected, and automatically or manually healed over time (with as little human intervention as possible) Priority aware work queue management , ensuring that the most important tasks are run first and complete as soon as possible

, ensuring that the most important tasks are run first and complete as soon as possible Resource pooling system to ensure that, in a high concurrency environment, thresholds can be set to avoid overloading input or output systems

system to ensure that, in a high concurrency environment, thresholds can be set to avoid overloading input or output systems Backfill capabilities to identify “missing” past runs, and automatically re-create and run them

capabilities to identify “missing” past runs, and automatically re-create and run them Full history of metrics and statistics to view the evolution of each task performance over time, and even assess data-delivery SLAs over time

to view the evolution of each task performance over time, and even assess data-delivery SLAs over time An horizontally scalable set of alternatives to the way tasks are dispatched and run on a distributed infrastructure

set of alternatives to the way tasks are dispatched and run on a distributed infrastructure A centralized, secure place to store and view logs and configuration parameters for all task runs

Fig 1 — Example representation of a DAG in Airflow

All these features allow us to run more than 10,000 automated tasks each day without breaking a sweat. Even in the case of a major failure, recovery requires very little human labor, as the system eventually heals itself automatically.

Airflow’s architecture relies on several components, and a few auxiliary tools: