Big Data File Format Cost Efficiency - Millions of Dollars Deal
Channel: The ASF
Date: 2020-10-21

Big Data File Format Cost Efficiency - Millions of Dollars Deal
Xinli Shang, Juncheng Ma

A presentation from ApacheCon @Home 2020
https://apachecon.com/acah2020/

Reducing the size of data at rest and in transit is critical to many organizations, not only because it can save the cost of storage but also can improve the IO usage and traffic volume in the network. We will present how to translate hundreds of petabytes data compressed in GZIP in the data lake to a more efficient compression method - ZSTD which can reduce the data size by 10% and save millions of dollars. We will show the recent Apache Parquet format improvement that makes ZSTD compression to be easily set up (PARQUET-1866). The tech talk will also demonstrate how we solve the challenges of the compression translation speed by improving the throughput by 5X (PARQUET-1872). As a result, the translation time of large scale data sets can be reduced from months to days and save compute vCores correspondingly. The translation needs to be in a safe way to prevent data corruption and incompatibility. We will also show the technicals that are built into the compression translation tool to prevent them from happening. Another significant storage size reduction (up to 80%) can be done by 1) reordering the columns in Parquet to make it more friendly to encoding and compression, 2) encoding with BYTE_STREAM_SPLIT (PARQUET-1622) that is more efficient for floating type data, 3) reducing geolocation data precision to make RLE more efficient, 4) pruning unused columns (PARQUET-1800). We will show above every technique, the effectiveness of each and the reason behind it. We will also show the tools like Parquet column-size (PARQUET-1821) that can help users to identify the candidate tables to apply the above techniques.

Xinli Shang:
Xinli Shang is a tech lead on the Uber Data Infra team, Apache Parquet Committer. He is passionate about big data file format for efficiency, performance and security, tuning large scale services for performance, throughput, and reliability. He is an active contributor to Apache Parquet. He also has many years of experience developing large scale distributed systems like S3 Index, and operating system Windows.
Juncheng Ma :
Software Developer at Uber