Load-Balance and Fault-Tolerance for Massively Parallel Phylogenetic Inference

Upcoming exascale supercomputers will comprise hundreds of thousands of CPUs. Scienti c applications on these supercomputers will face two major challenges: Hardware failures, and parallelization e ciency. We extend RAxML-ng, a widely used tool to build phylogenetic trees, to mitigate hardware failures without user intervention. For this, we increase the checkpointing frequency. We also detect failures, redistribute the work among the surviving ranks, restore a consistent search state, and restart the tree search automatically. RAxML-ng now supports fault tolerance in the tree search mode, using multiple starting trees, and multiple alignment data partitions. RAxML-ng can handle multiple failures at once as well as multiple successive failures. There is no limit on the number of failures that can occur simultaneously or sequentially. We also support mitigating failures which occur during the recovery of a previous failure or during checkpointing. In contrast to the previously available manual recovery scheme, a recovery is initiated automatically after a failure, that is, the user does not have to take any action. We benchmark our algorithms for checkpointing and recovery. In our experiments, creating a checkpoint of the model parameters requires at most 72.0 ± 0.9ms (400 ranks, 4,116 partitions). Creating a checkpoint of the tree topology requires at most 0.575 ± 0.006ms (1,879 taxa). The overall runtime of RAxML-ng increases by a factor of 1.02 ± 0.02 when using the new checkpointing scheme and by a factor of 1.08 ± 0.07 when using the new checkpointing scheme and ULFM v4.0.2u1 as the MPI implementation. Restoring the search state after a failure requires at most 535 ± 19ms. We simulated up to ten failures, which causes the overall runtime to increase by a factor of 1.3 ± 0.2. We also describe multiple approaches on how to store the MSA data, which has to be re-read after a failure, redundantly in memory to avoid disc-accesses after a failure. RAxML-ng synchronizes thousands of times per second. How equally the load balancer distributes the work across the CPUs therefore directly in uences the overall runtime. We nd that some ranks require up to 30 % more time to process their portion of the work than the average rank does. We also nd that a single rank sometimes requires the most time to process the current portion of work in 30 % of all iterations. We identify the site-repeats feature (an algorithmic optimization that avoids redundant computations) as the cause of this imbalance. We also present algorithms to solve the multi-sender h-relation problem and the unilaterally-saturating 1-matching problem. The multi-sender h-relation problem is a variant of the h-relation problem in which each package can be received by any CPU in a set of valid sources. The unilaterally-saturating 1-matching problem is a variant of the 1-matching problem in bipartite graphs. In the 1matching problem, a function 1 (E) de nes an upper-bound for the number of matching edges each vertex E might be incident to. The matching is called unilaterally-saturating, if for one of the two sets of the bipartite graph, each vertex is incident to at least one matching edge.