Creating a Spark Streaming ETL pipeline with Delta Lake at Gousto
By: André Sionek
Tags: Data Engineering, Data, Databricks, Gousto, Spark

The EMR job alone was taking about two hours to complete. But since it was processed overnight, fresh data was available every morning for reporting.

Then everything changed.

Files started arriving in 15 minutes chunks, instead of once a day. This was an old request from operations that was finally being fulfilled by the factory software. Of course, we immediately got a request to update the reports built on top of this data at the same frequency.

As you can already notice, with a job that takes two hours to execute, the best we could do was a two hours refresh latency on reports. With the company shifting towards a Supply Chain Data Strategy, we had to rethink our ETL to deliver production line data in almost real-time.

I will try to explain why we made some choices, how our stack evolved and also provide some code to help out others that are facing similar challenges.

Why was it taking two hours?

The first approach was to understand why that job was taking so long to execute and try to tune it for better performance. We noticed the processing was inefficient and that the bigger our tables, the worse it would get. Either by requiring more time to complete, or more processing power.

Here is a brief description of what our Spark job was doing:

Adding new partitions to raw table

Our Spark job was first running MSCK REPAIR TABLE on Data Lake Raw tables to detect missing partitions. This can be a costly operation because it needs to scan the table’s sub-tree in the file system (the S3 bucket). This was already taking a lot of time to complete, and it would only take longer over time since we were adding new partitions with multiple small files every day.

Defining dataframes for Raw and Cooked Tables

Spark is lazy, so just creating the dataframes for raw and cooked tables is not a problem. Just remember that raw is composed of the original CSV files and Cooked is composed of transformed Parquet files. Both dataframes were not being cached, this will be important to understand the subsequent steps.

Counting rows in both dataframes

The idea here was to stop processing early on if there was no data to be moved from raw to cooked — only if the count in raw was greater than the count in cooked, the job would continue for that table.

To count rows Spark will scan all CSV files in all partitions in raw and aggregate the data. For cooked this is better since the data is already stored as Parquet, so its metadata will already contain rowcounts .

Selecting rows to be inserted

Here the job was calculating the maximum value of a timestamp column in cooked and then using it to select rows in raw that were greater than max.

Because the dataframes were not cached, Spark was again reading Parquet metadata to get the max value. Applying filter on raw is lazy, so no problems here.

Union cooked dataframe to filtered rows

The selected rows from raw were then unioned back to cooked dataframe. Lazy operation again.

Write the new table to Data Lake Cooked

Finally, it would call write to Data Lake Cooked. Here, Spark was reading again all CSVs in raw and all Parquet in cooked to execute the operation. It was also overwriting the cooked table, rewriting all files in all partitions.

Rinse and repeat for multiple tables and data sources

To deliver this data required by operations we had to process eight tables. However other sources were also using the same job, and it was not possible to decouple them without creating separate clusters for each data source.

Here is a simple diagram of this process to make it easier to understand: