Uber achieves millions $ of savings managing disk IO across HDFS cluster - L Gao, E Sethuramalingam
Channel: The ASF
Date: 2021-10-11

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

HDFS is the main storage platform for the data lake in Uber that holds hundreds of PB of data in two data centers. In the past year, we have spent a lot of effort improving the storage efficiency of our HDFS clusters. One of the most important findings we had is that the datanode IO across the cluster is under-utilized. Although the top node utilization is high, the average IO utilization is low.

By balancing disk IO utilization across the cluster, we were able to adopt much denser disk (from 4TB/8TB to 16TB) to our fast-growing cluster with even better user experience, that reduced storage unit cost by 45%.

First, we built a way to perform storage tiering on the same disk. By placing both HOT/COLD blocks on the same HDFS volume, we can control the disk hotness by configuring HOT/COLD ratio on the disk. Therefore, we can balance the IO utilization across different hardware SKUs and also across the cluster.

Second, we improved the HDFS client to avoid reading from busy datanodes. We implemented fast-switch read, that HDFS stateful read can detect the datanode slowness and try reading from another replica. Also, we created a way to deprioritize certain datanodes from namenode, so clients are less likely to access them.

In the talk, we will share the details of the above new HDFS features, how we manage hardware/software in our clusters, and the status of sharing them in the open-source community.