CSSI Frameworks: Performance Engineering Scientific Applications with MVAPICH and TAU using Emerging Communication Primitives PI: Dhabaleswar K. (DK) Panda (OSU), Co-PIs: Sameer Shende (UO), Ahmad Abdelfattah (UTK), Yifeng Cui (SDSC) Institutions: The Ohio State University (OSU), University of Oregon (UO), University of Tennessee (UTK), San Diego Supercomputer Center (SDSC) 1. Load-aware designs for MPI asynchronous communication 2. Cross runtime coordination for MPI+X applications 3. Partitioned point-to-point primitives for efficient communication 4. Application-aware neighborhood collective communication 5. Support for adaptive persistent collective communication 6. Coordinated communication kernels on GPUs 7. On-the-fly compression for accelerating scientific applications Proposed Framework Proposed Innovations Motivation & Research Challenges NSF CSSI PI Meeting, Charlotte, NC, August 12-13, 2024 Award #: OAC-2311830 1. Designing and Optimizing GPU-aware Nonblocking MPI Neighborhood Collective Communication for PETSc, K. Khorassani, C. Chen, H. Subramoni, and DK Panda, 37th IEEE International Parallel & Distributed Processing Symposium (IPDPS '23), May 2023 2. Accelerating MPI AllReduce Communication with Efficient GPU-Based Compression Schemes on Modern GPU Clusters, Q. Zhou, B. Ramesh, A. Shafi, M. Abduljabbar, H. Subramoni, D. K. Panda, ISC High Performance (ISC '24), May 2024 3. MPI performance engineering with the MPI tool interface: The integration of MVAPICH and TAU, Srinivasan Ramesh, Aurèle Mahéo, Sameer Shende, Allen D. Malony, Hari Subramoni, Amit Ruhela, Dhabaleswar K. (DK) Panda, Parallel Computing, 2018 4. Lossy all-to-all exchange for accelerating parallel 3-D FFTs on hybrid architectures with GPUs, S. Cayrols, J. Li, G. Bosilca, S. Tomov, A. Ayala and J. Dongarra, 2022 IEEE International Conference on Cluster Computing (CLUSTER), 2022 5. Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard models, Callaghan S, Maechling PJ, Silva F, Su M-H, Milner KR, Graves RW, Olsen KB, Cui Y, Vahi K, Kottke A, Goulet CA, Deelman E, Jordan TH and Ben-Zion Y,. Front. High Perform. Comput. 2:1360720, 2024 6. Accelerating Large Language Model Training with Hybrid GPU- based Compression, L. Xu, Q. Anthony, Q. Zhou, N. Alnaasan, R. Gulhane, A. Shafi, H. Subramoni, D. Panda, IEEE/ACM International Symposium on Cluster, Cloud, and Internet Computing 2024, May 2024. Research Publications The proposed project seeks to co-design and integrate two powerful tools – MVAPICH and TAU – to demonstrate the new possibilities for performance-guided control and optimization for two large scale applications — AWP-ODC and heFFTe. 1. What are the challenges in achieving cross-runtime coordination and performance engineering for HPC applications? 2. Is it possible to leverage newer features available in modern GPU computing, the MPI 4.0 standard, and high- performance networking hardware to deliver better performance for HPC applications through efficient performance engineering? 3. Can MVAPICH and TAU co-design be leveraged in heFFTe and AWP-ODC to determine the reasons and guide resolutions for current strong scalability limitations in these applications? 4. How to reduce MPI communication costs for memory- bound heFFTe and AWP-ODC through new MPI features and integration with application-specific designs, mixed- precision algorithms, and real-time data compression? Supported by: OAC-2311830, OAC-2311831, OAC-2311832, OAC-2311833 Future Work Software Release, Community Engagement and Metrics MVAPICH (Initial Results) HeFFTe (Initial Results) TAU (Initial Results) AWP-ODC (Initial Results) Proposed integrated MPI profiling and performance engineering framework for MVAPICH and TAU, aiding HPC applications (AWP-ODC and heFFTe) Strong scalability on 3D FFTs of size 10243, using 24 MPI processes (1 MPI per Power9 core) per node (blue), and 24 MPI processes (4 MPI per GPU-V100) per node (red) Use T