# A Survey on Malleability Solutions for High-Performance Distributed Computing
## 4 MPI Malleability
### Dmr  
The Dynamic Management of Resources (DMR) [38] malleability framework consists of two main components: an RMS and a parallel distributed runtime based on MPI DMR provides the communication layer between them and allows malleable application execution in a cluster workload. SLURM is the RMS used by DMR and, as in other solutions, it is responsible for monitoring the resource utilization and jobs requests. SLURM has been extended with the capability of scheduling malleable jobs and managing dynamic resources. In a nutshell, job malleability in DMR works as follows:  
* When an application is running, it has to periodically contact the RMS in order to show its rescaling willingness. * This communication is established in a synchronization point, defined by users, where the reconfiguration operation can be initiated. * Then, SLURM evaluates the global system status to decide whether to perform that reconfiguration and communicate that decision to the runtime. * If this call resolves to resize the job, SLURM reallocates the resources and returns the resulting number of processes, which can mean to expand or shrink the job. * Finally, the runtime, with the guidelines of the application, redistributes the data between parent and children, and the job continues the execution with the new process layout. Furthermore, the current implementation of DMR accommodates three scheduling modes with different degrees of freedom when requesting a resize action:  
* Applications are allowed to **strongly suggest** a specific operation. For instance, to expand a job, users are allowed to define the \"minimum\" number of requested nodes to a value greater than the number of allocated nodes of that job. * Determine a **sweet spot** number of processes to run an application. Users can configure a malleable application with a preferred size from which the execution cannot be scaled down; however, the scheduler can decide to grant an expansion up to its specified maximum. * **Wide expansion**:
* Expand a job as long as there are sufficient available resources if:
* There is no job pending for execution in the queue. * No pending job can be executed due to insufficient available resources. * Shrink a job if there is any queued job that could be initiated. DMR counts with two different approaches: using an API or a library. The API is based on the OmpSs programming model [39] and uses their compiler directives for the user code. The library is a module that can be loaded during the compilation of an MPI program providing the logistics to support malleability. They are described below:  
* The DMR API [40] implements a communication layer between the OmpSs runtime (Nanos++ ([[https://pm.bsc.es/nanox](https://pm.bsc.es/nanox)]([https://pm.bsc.es/nanox](https://pm.bsc.es/nanox))) (accessed on 5 May 2022)) and SLURM that allows the reconfiguration of MPI applications (see Figure 4). This API is based on the OmpSs off-load semantics [41] for automatically managing processes and redistributing data among them. Nanos++ runtime is extended with the logic to handle job reconfigurations. In this regard, a new routine, named dmr_check_status, is provided to the users. This function is responsible for triggering the reconfiguration. Moreover, the authors in [42] also present an asynchronous version of the function, such that when an action is scheduled, DMR sets up the new process layout while the application continues its execution. This version deploys the reconfiguration in the next iteration, to reduce the effective reconfiguration time. Since DMR API relies on OmpSs, the reconfiguration is conducted by #pragma directives, with which, users can define data dependencies and the communication pattern among processes. Listing 5 showcases the use of this directive where data dependencies are explicitly indicated. After the #pragma users indicate the resuming function for continuing with the execution.