# A Survey on Malleability Solutions for High-Performance Distributed Computing
## 3 Fault Tolerance for Malleability
### SRS Library and RSS
Performed by two functions to know the rank and state of each process, one for the state of the application and another one to simplify the data redistribution task. 2. C/R techniques. Performed by two functions to store or load checkpoints to redistribute data. Data are stored with proximity to their destination. 3. Initialization and finalization. Performed by one function to initialize the environment and another one to finalize it. 4. Collective reconfiguration. Performed by four different functions to allow different kinds of reconfigurations using some or all the processes of the application

It also supports data distributions such as block, cyclic, and block-cyclic; however, it is limited to data-parallel programs with a 2D data structure and a linear communication structure. Users can develop their malleable applications through both: PCM API calls, such as MPI_PCM_Init that initializes internal data structures; a configuration file that has information about the IOS runtime configuration as the port number where accept connections and the location of the PCM daemon. PCM functions wrap many of the MPI functions/variables. In this regard, coders are expected to use the API following this workflow:

* Check if there is a migration in progress. If so, determine which type of reconfiguration has been scheduled (expansion or shrinkage). * Spawn new processes and configure them to load data according to the new processes layout. PCM explores how malleability can be used in C/R applications [12]. Especially, PCM reconfiguration has been implemented and evaluated with a fluid dynamic problem that solves heat diffusion in a solid for testing purposes. The C/R mechanism is leveraged to restart applications with a different number of processes from data stored in checkpoint files. ### Scalable Checkpoint /Restart Extended

Authors in [13] present the scalable C/R (SCR) library, which consists of a novel multi-level checkpoint system that allows us to write checkpoints to RAM memory, flash, or disk on the compute nodes in addition to the parallel file system. This library enables MPI applications to use storage distributed on a system's compute nodes to attain high checkpoint and restart bandwidth. This technique uses multiple types of checkpoints that have different levels of resiliency and cost in a single application run. The slowest but most resilient level writes the checkpoint file to the parallel file system. Faster but less resilient checkpoint levels utilize local node memory, such as RAM, Flash, or disk to cache these files. So, applications can take frequent inexpensive checkpoints and less frequent and more resilient ones, resulting in better efficiency and reduced load on the parallel system. It also reduces the input/output bandwidth. SCR library implementation is focused on two key issues:

1. Jobs only need their most recent checkpoint. As soon as the next one is saved in the local storage of the compute node, the previous can be discarded. SCR can also apply a cross-redundancy scheme by caching a local checkpoint file on others nodes. This helps to recover from a failure of a small part of the system. In addition, SCR periodically copies the local checkpoint file to the parallel file system to support failures that disable larger parts of the system. 2. SCR depends on an external service (such as an RMS) to cancel a job that presents a failure. Upon failure, it attempts to recover the most recent checkpoint from a local cache. Assuming it succeeds, SCR can copy that file to the parallel file system and try to restart the execution from the checkpoint. If it fails, it looks for the last file cached in the parallel file system and attempts to restart execution from there. In [14], authors extend SCR to enable malleability in MPI applications, leveraging the multi-level C/R. In addition, it also permits data to be cached at a specific position in the checkpoint files providing their further redistribution.