# A Survey on Malleability Solutions for High-Performance Distributed Computing
## 5 Charm++ Malleability
### Adaptive MPI  
Adaptive MPI (AMPI) [49, 50] is presented as a regular MPI implementation (such as MPICH or OpenMPI) that allows MPI developers to leverage CHARM++'s dynamic runtime system. In this regard, MPI applications, with zero to minimal changes, are granted support for process virtualization, the overlap of communication and computation, load balancing, and fault tolerance [51, 52]. There are different options to implement CHARM++ chares: processes, kernel-threads, and user-threads. AMPI has chosen the latter because it provides full control over scheduling. Moreover, with user threads, it is also possible to maintain information on the communication pattern among chares, and their computational and memory requirements. As has been introduced before, each chare has local data and references to global memory that are stored in the stack of the thread. The success of the migration is achieved by copying data and references from one thread to another and keeping consistent the  
Figure 8: Charm++ and RMS—shrinking a job by removing nodes 2 and 3 [47]. Based on: [[http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf](http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf)]([http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf](http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf)) (accessed on 7 May 2022). Adapted with permission from [47]. Copyright 2014, [PERSON]. Figure 7: Expanding a job by adding nodes 2 and 3 [47]. Based on: [[http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf](http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf)]([http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf](http://charm.cs.illinois.edu/newPapers/15-12/paper.pdf)) (accessed on 7 May 2022). Adapted with permission from [47]. Copyright 2014, [PERSON]. allocations in the new mapping. In order to convert an MPI code to AMPI, global and static variables have to be transformed as private variables in each chare [49; 50]. AMPI proposes the use of an scalable variant of the _isomalloc_ functionality PM2[53]. This technique consists of allocating the thread's stack, in such a way that it spans the same reserved virtual addresses across all the processors. Thus, the total virtual address space is split among physical processors with the same virtual addressing. For this reason, during migrations, memory locations do not change among threads. Finally, it should be noted that the management of the virtual address space is based on the use of _mmap_ and _mummap_ functionality of Linux. Thus, the allocation routines are intercepted and handled by the runtime which locates the memory reservation in the corresponding virtual address. Natively, CHARM++ operations are non-blocking. To enable blocking, operations are expected to be split into two different chares. The first one implements the same operation in a non-blocking manner, whereas the second contains the task after blocking, thus, when the first is completed, the next starts. The use of blocking MPI operations could considerably alter the structure of the program, therefore, to avoid this problem,  
AMPI includes specific implementations of these routines to facilitate their use. AMPI includes several routines as extensions to the MPI standard, in which AMPI_Migrate is the most relevant for load balancing and malleability. This is a collective operation where all chares have to execute with the same parameters. Then, the runtime analyzes the load of the physical processors, and the corresponding load balancing strategy is applied. As a result, chares could be scheduled for migration, and consequently, the job rescaled. AMPI_Migrate has a single parameter (MPI_Info), which determines the operation. There are four different possibilities. Thus, two built-in MPI_Info objects, called AMPI_INFO_LB_SYNC and AMPI_INFO_LB_ASYNC, which, respectively, determine whether the load balancing is at a synchronization point (see Listing 7). Moreover, MPI_INFO_CHKPT_IN_MEMORY is another built-in MPI_Info object that forces to perform a checkpointing in memory.