Next Generation Science Applications for the Next Generation of Supercomputing

The Trinity supercomputer deployment by Los Alamos and Sandia National Laboratories represents the first Advanced Technology System (ATS) deployment for the United States National Nuclear Security Administration (NNSA). The platform will be one of the largest XC40 deployments in the world when final integration of its nearly ten-thousand nodes of dual-socket Intel Xeon Haswell and ten-thousand nodes of Intel Knights Landing processors is completed during 2017. In the deployment of the Knights Landing partition, we have recognized a number of novel hardware technologies that are in many ways sentinels for the next generation of supercomputers. These include wide vector processing units (VPUs), greater use of symmetric multi-threading (SMT) as well as multiple pools of memory, each with differing and complex performance characteristics. Trinity, therefore, represents the start of significant change in the production computing landscape for the NNSA. Recognizing that these new hardware features may pose unique challenges for legacy codes has created an effort associated analyzing the performance of our application portfolio on the novel partitions of Trinity. We summarized our early findings in this context at CUG 2016 covering early evaluations of Sandia’s SIERRA engineering suite on Haswell and Knights Landing systems. In this paper however, we turn our attention to the performance of a suite of new applications that have been written from the ground up to be portable across computing architectures, parallel in terms of multi-node and on-node threading and to feature more flexible componentbased code design. These applications strongly leverage technologies such as Kokkos, Sandias C++ Performance Portability programming mode, the Trilinos linear solver library and our broader performance analysis capabilities in a close knit codesign program. Driven by the NNSAs Advanced Technology Development and Mitigation (ATDM) program, the new codes represent prototypes of fully-capable production science codes that will execute with high levels of efficiency on the next-generation of supercomputing platforms including Trinity and beyond. In our talk, we will specifically study the new SPARC computational fluid dynamics code developed under the ATDM program, reporting on its performance on both the Haswell and Knights Landing partitions of the Trinity supercomputer. We show a range of performance observations but importantly, that through very careful design, we can provide strong MPI and threading performance on these machines countering some of the concern that the use of OpenMP threading will always lead to poorer performance than pure MPI instantiations. Such conclusions are important within the NNSA because they speak to the ability to maintain scientific delivery on future machines while more novel threading algorithms are added to our production code suites. While considerably more work is required in this area, and this continues to be done under the ATDM program, early studies such as this are helping to characterize what performance levels we might expect to find on systems along the path to Exascale. By engaging with the broader Cray User Group community, we expect to be able to share our observations and invite feedback for the approaches we are pursuing so that the HPC community at large can benefit from the next generation of supercomputing hardware designs.