RADIX: HPC Data Center Performance Monitoring and Management
Creators: name: University Of California Office Of The President
Publisher: name: University of California Office of the President
publisherIdentifier: https://ror.org/00dmfq477
publisherIdentifierScheme: ROR
Year: 2024

description: High-performance scientific applications at the National Laboratories and many Internet and business applications run in data center computing environments with thousands to hundreds of thousands of computers and thousands of hard drives storing terabytes to petabytes of data. Planning, provisioning, and managing data center performance is a tremendous task, made more of an art than a science by the lack of effective performance management tools and metrics. Performance planning and provisioning is based on educated guesses, while performance management often occurs only in hindsight, when failures are reported by users. Due to the lack of a principled scientific basis for real performance management, these tasks are usually the domain of grizzled veterans who rely on intuition based on long experience. These techniques work reasonably well for smaller configurations, but scale poorly as systems grow to thousands or millions of components interconnected in increasingly complex⎯and increasingly unmanageable⎯configurations. Our solution, RADIX, combines extensive experience with high-end computing, I/O, and storage systems with cutting-edge research in CPU, I/O, and storage performance management to provide a novel solution to the difficult problem of large-scale high-performance data center performance management. RADIX includes metrics, tools, operating system components, and user-interfaces for data center performance planning, provisioning, monitoring, and management. Specifically, RADIX includes: 1) A performance model and metrics for predicting, managing, and monitoring the performance of each of the system components; 2) Tools for performance profiling, prediction, planning, and provisioning; 3) Operating system components for runtime management of the CPU, network, disk, and cache at runtime; and 4) Monitors and visualization tools for runtime performance monitoring and fault detection and isolation. RADIX is based on our previous research on the Resource Allocation and Dispatching (RAD) resource management model and its application to CPU, storage, networking, and cache performance management. RAD is a uniform resource management model consisting of resource shares and reservation granularities that enables hard and soft performance guarantees with good performance for each of the devices in the HPC data center computing environment. Importantly, RADIX provides formal, uniform metrics and mechanisms for HPC performance planning, provisioning, monitoring, and management. Our RAD-based resource management solutions also provide strong isolation between clients—users of the resources—ensuring performance independent of other workloads. For a given workload, this allows high-performance guarantees on resource- and workload-specific metrics such as disk requests per second, something not possible with other resource management schemes. RADIX will improve the development, management, performance, and uptime of large (and small) data centers, with profound impact for the developers and users of such systems and, ultimately, for the scientific research and business applications that execute upon such systems. By providing a better understanding of, and metrics for, data center performance, RADIX will allow system HPC systems designers to target the specific performance required of their target applications. RADIX's runtime performance management components will both guarantee and improve the performance of applications by isolating workloads, reducing the performance penalty of concurrently executing workloads and avoiding the unnecessary over-provisioning that is currently used to ensure adequate performance. RADIX's runtime performance monitoring relative to the guaranteed performance will ensure that problems are detected as they occur, at their source, allowing for rapid response and repair. RADIX's provisioning tools and resource guarantees should also prevent the catastrophic cascading failures frequently caused by individual component failures in such systems. UCSC is a center of excellence in storage systems and Profs. Brandt is a top researchers in storage systems, real-time systems, and performance management. Dr. Loncaric leads the group that designs and deploys new systems into the LANL HPC Data Center, one of the largest in the world supporting a tremendous variety of applications in physical, life, and social sciences.