YARN Resource Management and Dynamic Max - F. Liu, F. Tian, P. Golash, H. Zhang, S. Zhang
Channel: The ASF
Date: 2021-10-11

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

As one of the largest YARN deployments in the industry, Uber YARN has over 10k nodes,  over 230k daily completed jobs and  over 500 queues on the primary production cluster, the resource management has been very challenging. Yarn Resource Management (YRM) is a user-friendly and efficient self-service system that decentralizes YARN queue property and capacity management to organization resource administrators. It manages queues and partitions for all Uber YARN and Athena clusters. YARN Dynamic Max is a service built upon YRM. It dynamically adjusts queue capacities based on cluster utilization to maximize cluster resource utilization while ensuring the cluster does not overheat.Apache Hadoop YARN is an integral part of on-premisw solutions and it will be for the foreseeable future. We also believe, it will have an important role in the Cloud as well allowing users to move their solutions both to private and public cloud as easily as possible. For this reason, the development of YARN took a new momentum in the last year. In this talk, we will talk about the latest updates from the community which will be released in Hadoop 3.3.x/3.4.x releases: For compute acceleration devices (including GPU/FPGA, etc.), we will talk about better GPU support (including GPU hierarchy scheduling support, Nvidia-docker v2 support, etc.); YARN’s device plugin framework to allow developers easier add new compute-acceleration devices; FPGA support, etc. For scheduling related improvements, we will talk about new improvements of global scheduling in CapacityScheduler, new efforts to bring dynamic-queue-creation/absolute-resource into production, and one of the recent work, fs2cs, which allows user migrate from FairScheduler to CapacityScheduler. Apart from these, we will talk about new containerization improvements: runc support, improvements of log aggregation to better support cloud storage, etc. Audiences will get the latest development progress of Apache Hadoop YARN, and help them to make a decision when upgrading to Hadoop 3.x.