Data Lakehousing in AWS
By: George Pongracz
Tags: Delta Lake, Engineering, Apache Spark, Amazon Redshift, Data Lakehouse

Extending OSS Delta Lake with Redshift

The open source version of Delta Lake lacks some of the advanced features that are available in its commercial variant.

Caching & Data Layout Optimisation

The use of Amazon Redshift offers some additional capabilities beyond that of Amazon Athena through the use of Materialized Views.

Materialized Views can be leveraged to cache the Redshift Spectrum Delta tables and accelerate queries, performing at the same level as internal Redshift tables. Materialised views refresh faster than CTAS or loads.

Redshift Docs: Create Materialized View

Redshift sort keys can be used to similar effect as the Databricks Z-Order function.

Redshift Docs: Choosing Sort Keys

Redshift Distribution Styles can be used to optimise data layout. This technique allows you to manage a single Delta Lake dimension file but have multiple copies of it in Redshift using multiple materialized views, with distribution strategies tuned to the needs of the the star schema that it is associated with.

Redshift Docs: Choosing a Distribution Style

Delta File Data Layout Optimisation

Delta Lake files will undergo fragmentation from Insert, Delete, Update and Merge (DML) actions. Just like parquet, it is important that they be defragmented on a regular basis, to optimise their performance, which should be done regularly.

The open source version of Delta Lake currently lacks the OPTIMIZE function but does provide the dataChange method which repartitions Delta Lake files.

The one input it requires is the number of partitions, for which we use the following aws cli command to return the the size of the delta Lake file.

eg something like:

aws s3 ls --summarize --recursive "s3://<<s3_delta_path>>" | grep "Total Size" | cut -b 16-

Spark likes file subpart sizes to be a minimum of 128MB for splitting up to 1GB in size, so the target number of partitions for repartition should be calculated based on the total size of the files that are found in the Delta Lake manifest file (which will exclude the tombstoned ones no longer in use).

Databricks Blog: Delta Lake Transaction Log

We found the compression rate of the default snappy codec used in Delta lake, to be about 80% with our data, so we multiply the files sizes by 5 and then divide by 128MB to get the number of partitions to specify for the compaction.

Delta Lake Documentation: Compaction

Once the compaction is completed it is a good time to VACUUM the Delta Lake files, which by default will hard delete any tomb-stoned files that are over one week old.

Delta Lake Documentation: Vacuum

Some Advice

Spectrum DDL

It is important to specify each field in the DDL for spectrum tables and not use “SELECT *”, which would introduce instabilities on schema evolution as Delta Lake is a columnar data store. When the schemas evolved, we found it better to drop and recreate the spectrum tables, rather than altering them.

This is important for any materialized views that might sit over the spectrum tables. If the spectrum tables were not updated to the new schema, they would still remain stable with this method.

Materialized View DDL

As tempting as it is to use “SELECT *” in the DDL for materialized views over spectrum tables, it is better to specify the fields in the DDL. We found it much better to drop and recreate the materialized views if the schema evolved.

If the fields are specified in the DDL of the materialized view, it can continue to be refreshed, albeit without any schema evolution.

This is preferable however to the situation whereby the materialized view might fail on refresh when schemas evolve.

Thank you

Databricks

I would like to thank Databricks for open-sourcing Delta Lake and the rich documentation and support for the open-source community.

AWS

I would like to thank the AWS Redshift Team for their help in delivering materialized view capability for Redshift Spectrum and native integration for Delta Lake.

I would also like to call out Mary Law, Proactive Specialist, Analytics, AWS for her help and support and her deep insights and suggestions with Redshift.

SEEK Enterprise DataOps Team

I would like to thank my fellow Senior Data Engineer Doug Ivey for his partnership in the development of our AWS Batch Serverless Data Processing Platform.

I would also like to call out our team lead, Shane Williams for creating a team and an environment, where achieving flow has been possible even during these testing times and my colleagues Santo Vasile and Jane Crofts for their support.

SEEK is a great place to work ❤️