MPIrigen: MPI Code Generation through Domain-Specific Language Models
Authors: name: Nadav Schneider, name: Niranjan Hasabnis, name: Vy A. Vo, name: Tal Kadosh, name: Neva Krien, name: Mihai Capotă, name: Abdul Wasay, name: Guy Tamir, name: Ted Willke, name: Nesreen Ahmed, name: Yuval Pinter, name: Timothy Mattson, name: Gal Oren

The imperative need to scale computation across numerous nodes highlights the
significance of efficient parallel computing, particularly in the realm of
Message Passing Interface (MPI) integration. The challenging parallel
programming task of generating MPI-based parallel programs has remained
unexplored. This study first investigates the performance of state-of-the-art
language models in generating MPI-based parallel programs. Findings reveal that
widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
code models) exhibit notable performance degradation, when generating MPI-based
programs compared to general-purpose programs. In contrast, domain-specific
models such as MonoCoder, which are pretrained on MPI-related programming
languages of C and C++, outperform larger models. Subsequently, we introduce a
dedicated downstream task of MPI-based program generation by fine-tuning
MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
an innovative preprocessing for completion only after observing the whole code,
thus enabling better completion with a wider context. Comparative analysis
against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
method, demonstrates that MPIrigen excels in generating accurate MPI functions
up to 0.8 accuracy in location and function predictions, and with more than 0.9
accuracy for argument predictions. The success of this tailored solution
underscores the importance of domain-specific fine-tuning in optimizing
language models for parallel computing code generation, paving the way for a
new generation of automatic parallelization tools. The sources of this work are
available at our GitHub MPIrigen repository:
https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen