D3.4 – Report of final public release of   high-performance software components
Creators: name: Jalby, William
Publisher: name: Zenodo
Year: 2024

description: The primary mission of the TREX Centre of Excellence (CoE) was development, promotion, and maintenance of high-performance software solutions in the field of computational quantum chemistry, physics, materials science, and nanotechnologies to take full advantage of the upcoming exascale architectures.

The TREX CoE targeted primarily QMC methods which are among the most accurate, albeit also numerically most expensive electronic structure methods. In combination with their uninhibited scaling with the numbers of CPUs and GPUs, it is our strong belief that they will become widely adopted in the near future. At this time, though, the QMC community is significantly smaller than both the DFT and the wave function theory (WFT) communities and so is the number of QMC codes available to the community.

One major goal of TREX is to develop massively-parallel quantum Monte Carlo (QMC) codes, highly optimized and made user friendly to run efficiently on future-generation supercomputers. To achieve that goal, a combined approach was used: first, “clean” and optimize target applications and second, develop a library of high-performance kernels (Quantum Monte Carlo Kernels: QMCKL) to be integrated into the codes.In the previous deliverable D3.1 (Initial report on the performance characteristics on relevant hardware for upcoming supercomputers) covering the first 18 months, we reported the success in optimizing and fine-tuning the TurboRVB and CHAMP codes and we analyzed the good scaling properties of the QMC = CHEM code. In D3.2 (Initial public release of high-performance software components) the general guidelines for QMCKL library development were laid down and initial kernel versions (including a simple pedagogic version) were produced and tested.

In the second part of the project (described in this report), QMCKL kernels were optimized and ported successfully across different CPU architectures: ARM NEOVERSE N1/V1, Skylake/IceLake/Sapphire Rapids, ZEN3/ZEN4 and on different compilers: ICC/IFort/ICX/IFX, AOCC, GCC/Gfortran and ARM Clang/Flang. Across all of these CPUs and environments, a single code version was used and obtained good performance and scalability. Then, a GPU version of the library was developed in two “Open” flavors (OpenMP and OpenACC) and ported on three different GPUs: NVIDIA A100, AMD MI210/250 and Ponte Vecchio. Due to the weakness of the software stack on GPU, the porting effort was much larger than moving from X86 to ARM. As a final result on QMCKL, the best GPUs offer limited potential gains 1 to 2x) versus an old full CPU 52 cores Skylake. Part of the limited performance gain is the size of the data used in QMC applications which does not favor full use of the large number of units present in a GPU.

Then, porting CHAMP and TurboRVB codes across a wide range of architectures/software (the same ones used for QMCKL) was successfully carried out. Tuning/optimization effort was pursued, focusing particularly on scaling (intra node and multinode). Finally, integration of QMCKL routines in the three reference codes has been performed delivering from 1,2 X performance gain for QMC=CHEM to over 2X gain for CHAMP and TurboRVB. Last but not least, all of the codes TREX CoE developed are now publicly available via the TREX web link which also offers support triage.

The challenge over the next decade is to pursue the development of the TREX flagship codes and QMCKL. These developments should not only cover migration to the next generation of computing devices (CPU and GPU) but also the integration of new functionalities in the codes such as new models. This would require the development of a specific software environment (extension of Continuous Integration framework) to support fast code changes. These issues are not limited to QMC codes and they should be tackled on the much broader scale for a vast majority of scientific codes.

Visit the TREX website to learn more.