# A Survey on Malleability Solutions for High-Performance Distributed Computing
## 4 MPI Malleability
### Elastic Execution of MPI Applications  
In [19], the authors propose an extension to the MPI Standard 2.0 to better support moldable and malleable applications on large-scale distributed memory systems. This consists of a new API that extends the MPICH library and allows the dynamic reshaping of the processes' layout. Following, the new functions are described:  
* MPI_Init_adapt: Initializes MPI likewise MPI_Init does, but it has an additional output parameter that informs about how processes are created. If the parameter contains joining, it means that the process was created during the execution. While if it contains new, it means that the process was created when the execution started. * MPI_Probe_adapt: With this routine, processes can ask the RMS whether a reconfiguration is required when a synchronization point is reached. There is an alternative interface of this routine, which is useful for fault tolerance applications. * MPI_Comm_adapt_begin: This routine starts the adaptation phase. It retrieves the inter-communicator, equivalent to the generated by spawn routines, and a flat intra-communicator, which will be _MPI_COMM_WORLD_ when the adaptation is completed. When shrinking, the intra-communicator of the processes that have to be terminated is set to _MPI_COMM_NULL_. They still are connected to the other processes with the inter-communicator. Management of these two communicators will allow data redistribution required during the adaptation. * MPI_Comm_adapt_commit: This routine notifies the RMS that the reconfiguration has ended. MPI_COMM_WORLD is changed by the new_comm_world obtained in the previous routine, in which the processes to be destroyed are excluded. An example of how to implement malleability using these functions is showcased in Listing 3. ```
1 intmain(intargc,char**argv){
2 MPI_Init_adapt(&argc,&argv,&local_status);
3 for(...){
4 MPI_Probe_adapt(&adapt,...);
5 if(localstatus==MPI_ADAPT_STATUS_JOINING
6||adapt==MPI_ADAPT_TRUE){
7 MPI_Comm_adapt_begin(...);
8//Redistributioncode
9 MPI_Comm_adapt_commit(...);
10}
11//ComputeandMPIcode
12}
13}
```  
Listing 3: Elastic MPI-Structure of a simple malleable application using the elastic execution [19]. Listing 3. SLURM Integration  
Despite the different alternatives when it comes to RMS, the proposed framework is integrated into SLURM [20]. Since other malleability solutions leverage SLURM as well, at this point, it is worth it to describe its architecture:  
* The SLURMCTLD daemon is a single centralized component (running on the management node), which is responsible for monitoring each compute node state and for allocating resources to jobs. The rest of the nodes (compute nodes) run the SLURMD daemon, on which the SLURMCTLD notifications arrive. These daemons periodically communicate to the SLURMCTLD to exchange node and job state information. * Jobs are included in a priority-ordered queue, concerning the criteria of the defined scheduling policy. When a job is selected to be executed, the job is initiated in a compute node, executing an SRUN instance. This instance notifies the other compute nodes that a SLURMSTEPD daemon has to be launched in each node. SLURMSTEPD daemons launch the local processes and interact with them via the Process Management Interface (PMI). Figure 2 depicts the interactions among SLURM daemons. To incorporate elasticity, some modifications to the design and functionality of SLURM are proposed in [19] to support its malleability framework [21,22]. * The batch script language is extended by the following parameters:
* - min-nodes-invasic defines the minimum number of nodes, such that the job is properly executed. * - max-nodes-invasic defines the maximum number of nodes for the job. * - node-constraints includes additional constraint on the number of cores. Some values are odd, even, power-of-two, cubic,...
* SLURMCTLD is divided in two components:
* Elastic Runtime Scheduler (ERS) manages expand/shrink operations for malleable jobs.