Omri Mor1, Jiakun Yan1, Hartmut Kaiser2, Marc Snir1 1Department of Computer Science, University of Illinois Urbana-Champaign 2Center of Computation and Technology, Louisiana State University A Lightweight Communication Interface for Asynchronous Many-Task Systems Lightweight Communication Interface (LCI) [2] Low-level Communication Library: - Intended user: high-level library developers - Similar to UCX, libfabric, and GASNet, as opposed to MPI Designed for AMT Runtimes: - Applies to other irregular applications, such as graph analytics or sparse linear algebra - Easily modifiable research library, for studying communication API design and implementation Major Features: - Multithreaded performance as top priority - Versatile communication interface - Explicit control of communication behaviors and resources Motivation Modern Parallel Architecture: - Increased intra-node parallelism. - Increased heterogeneity. - Powerful Interconnect. Task-based Programming Model: - Programmers decompose their program into tasks along with their dependencies. - The runtime will handle the mapping, scheduling and data movement. - E.g. HPX, Legion, PaRSEC. - Communication layer: MPI/GASNet. New Communication Pattern: - Multithreaded. - Irregular destinations. - Small messages. HPX [3] and PaRSEC [1]: New LCI Backends [4, 5] High Performance ParalleX (HPX): - Implementation of ParalleX execution model, standards-focused C++ API - Asynchronous task execution and arbitrary task graph construction. - Developed by the STE||AR Group Parallel Runtime Scheduling and Execution Controller (PaRSEC): - Generic framework for executing distributed task-based applications - Support for multiple Domain-Specific Language backends - Developed by the Innovative Computing Laboratory (ICL) at the University of Tennessee, Knoxville Task-based Multi-Chain Ping-Pong Motivation: - Communication layer benchmarks necessary to measure and improve efficiency - Traditional communication benchmarks do not directly apply to AMT runtimes Design: - “Chains” of tasks spawned on two nodes and alternating between nodes - Task chains associated with data movement - Tasks optionally execute computation on data Variables: - Message size, chain count, chain length, and task granularity Measurement: - Basic communication characteristics: latency, message rate, and bandwidth - Realistic cache effects with task modification of communication buffers - Communication-communication and computation-communication overlap Evaluation Execution Environment: - Expanse cluster at San Diego Supercomputer Center - 2× AMD EPYC 7742: 128 cores @ 2.25 GHz with 256 GiB DDR4 - Mellanox ConnectX-6: 2× HDR InfiniBand links in a Hybrid Fat-Tree topology - Rocky Linux 8.8 with Linux 4.18.0-477.15.1 - Open MPI 4.1.5 with UCX 1.14.0, using AMD Optimizing C/C++ Compiler Results: References 1. George Bosilca et al. “DAGuE: A generic distributed DAG engine for High Performance Computing”. Parallel Computing. 38, 1. 2012. (https://github.com/icldisco/parsec) 2. Hoang-Vu Dang, Marc Snir, and William Gropp. "Towards millions of communicating threads." Proceedings of the 23rd European MPI Users' Group Meeting. 2016. (https://github.com/uiuc-hpc/LC) 3. Hartmut Kaiser et al. "HPX: A task based programming model in a global address space." Proceedings of the 8th International Conference on Partitioned Global Address Space Programming Models. 2014. (https://github.com/STEllAR-GROUP/hpx) 4. Omri Mor, George Bosilca, and Marc Snir. "Improving the Scaling of an Asynchronous Many-Task Runtime with a Lightweight Communication Engine." Proceedings of the 52nd International Conference on Parallel Processing. 2023. 5. Jiakun Yan, Hartmut Kaiser, and Marc Snir. "Design and Analysis of the Network Software Stack of an Asynchronous Many-task System--The LCI parcelport of HPX." Proceedings of the SC'23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. 2023. Motivation Modern Parall