# Bridging the Gap Between Genericity and Programmability of Dynamic Resources in HPC

Dominik Huber, Sergio Iserte, Martin Schreiber, Antonio J. Peña, Martin Schulz

► To cite this version:

Dominik Huber, Sergio Iserte, Martin Schreiber, Antonio J. Peña, Martin Schulz. Bridging the Gap Between Genericity and Programmability of Dynamic Resources in HPC. ISC High Performance 2025 - 40th ISC High Performance International Conference, Jun 2025, Hamburg, Germany. pp.1-11. hal-04994828

HAL Id: hal-04994828

https://inria.hal.science/hal-04994828v1

Submitted on 17 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Bridging the Gap Between Genericity and Programmability of Dynamic Resources in HPC

Dominik Huber

Sergio Iserte

Martin Schreiber

Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG
Grenoble, France

Barcelona Supercomputing Center
(BSC-CNS)

Université Grenoble Alpes
Inria

Technical University Munich

Barcelona, Spain

CNRS

TUM School of Computation, Information and Technology
Garching, Germany

0000-0003-3654-7924

Laboratoire Jean Kuntzmann
Grenoble, France

0000-0001-9696-9382

0000-0002-2390-6716

Antonio J. Peña

Martin Schulz

Barcelona Supercomputing Center
(BSC-CNS)

Technical University Munich

Barcelona, Spain

TUM School of Computation, Information and Technology
Garching, Germany

0000-0002-3575-4617

0000-0001-9013-435X

## Abstract—

Dynamic Resource Management (DRM) allows dynamically changing the resources assigned to a job during its execution. This flexibility can improve productivity in completed jobs per unit of time, resource utilization rate, and energy consumption, among other metrics.

Despite its benefits, DRM remains complex to implement, which is one of the reasons it has not yet been widely adopted in production HPC systems. To address this challenge, we design, implement, and evaluate a methodology aimed at improving the programmability of DRM for iterative algorithm-based applications, while maintaining a generic and flexible foundation. Specifically, we interface the Dynamic Management of Resources API (DMR-API), an application-level abstraction layer that simplifies the adoption of dynamic resources—particularly in classical iteration-based HPC applications—with the Dynamic Processes with PSets (DPP) approach, which provides a set of generic design principles for dynamic resource management in high-performance parallel programming models. This integration makes DRM more accessible for iteration-based HPC applications through the DMR-API, while preserving the generality and flexibility of DPP at the system software level to support a wider range of HPC application types.

Our results show that DRM can be effectively leveraged in HPC environments with minimal coding effort, unlocking the benefits of dynamic resource allocation for job throughput and system utilization.

## I. INTRODUCTION

The rapid growth in the number and scale of High-Performance Computing (HPC) systems highlights the importance of efficient resource management. Currently, most systems rely on static resource management, where resources allocated to a workload remain fixed throughout execution. In contrast, Dynamic Resource Management (DRM) allows for the real-time adjustment of resources assigned to workloads during their execution. Over recent years, DRM has

garnered significant attention due to its potential to offer numerous advantages to HPC system providers and users. For instance, many HPC applications, such as Adaptive Mesh Refinement methods and in-situ workloads, exhibit varying resource requirements over their runtime time. DRM enables tailoring the resource assignments to the dynamic behavior of such applications [1], [2], possibly increasing resource utilization efficiency and reducing the amount of charged core-hours. From the system perspective, DRM increases the scheduling flexibility to optimize global system metrics such as throughput, utilization rate, and energy efficiency [3]–[5].

Despite the tangible advantages, DRM has not yet crossed the border toward deployment on production systems. A major challenge is the trade-off between a flexible and generic design on the one hand and a user-friendly API on the other hand. A flexible and generic design is required to cover all different kinds of dynamic HPC patterns (iterative workloads, in-situ workloads, coupled workloads, etc.). Unifying DRM for these use cases into a single generic mechanism enables 1) interoperability across different HPC system software components and implementations and 2) a comprehensive formulation of the optimization problem for resource scheduling. Hence, it is the basis for its adoption in production HPC systems. However, such a generic design often comes at the price of a more complex and hard-to-use interface for application developers. This equates to longer application development cycles and higher costs, harming the willingness of application developers to adopt dynamic resources in their applications. Lowering this bar is, therefore, another crucial factor for the success of DRM.

In this work, we address the challenge of bridging the gap between genericity and programmability in DRM for HPC systems. To achieve this, we propose a layered approach that employs a generic abstraction at the lower layers of
the HPC software stack—offering a high-dimensional feature
space—while exposing a specialized, user-friendly API to ap-
plication developers, providing a simplified, low-dimensional
feature space. Our framework integrates the Dynamic Pro-
cesses with PSets (DPP) design principles [6] with the Dy-
namic Management of Resources API (DMR-API) [7], making
DRM more accessible for common HPC workloads.

The DPP model follows generic design principles for man-
aging dynamic resources in parallel programming models.
It uses Process Sets (PSets) and Process Set Operations
(PSetOps) as a unified, graph-based abstraction layer between
applications and the resource manager, enabling flexible and
scalable dynamic resource assignments.

On the other hand, the DMR-API is a high-level interface
designed to facilitate the integration of dynamic resource
management into HPC codes. It provides a user-friendly API
that abstracts the complexities of low-level DRM mechanisms,
making it easier for application developers to adopt dynamic
resource usage without being burdened by underlying technical
intricacies.

By embedding the DMR-API into the DPP framework, we
offer a DRM solution that is both generic and easy to use,
ensuring flexibility while reducing the entry-level effort for
application developers.

In summary, we present, for the first time, a highly usable,
scalable, and reliable DRM solution for supercomputers. This
approach combines a generic and flexible mechanism, capable
of covering diverse application use cases and scheduling
strategies, with practical, easy-to-use interfaces for application
developers.

The remainder of this document is structured as follows.
In Sec. II, we review other approaches for dynamic re-
sources in the literature and provide background for our work.
Sec. III introduces DPP and DMR-API in detail and states
the contributions of this work. Subsequently, we describe
our approach to map DMR-API and DPP and illustrate its
implementation using MPI Sessions in Sec. IV. Finally, we
provide an evaluation of a demonstrator implementation in
Sec. V and a conclusion in Sec. VI.

II. RELATED WORK

In this section, we review efforts in dynamic resource
management from the perspective of our contribution. In this
regard, we limit the review of the state-of-the-art to in-
memory solutions based on the MPI paradigm, in other words,
distributed dynamic resource managers capable of reassigning
data on-the-fly, with no need to store them on disk.

A comprehensive survey and comparison of spawn-based
process malleability are given by [8] and a more general study
and comparison of dynamic resource management state-of-
the-art are provided in [9].

At the moment of writing this paper, the vast majority of
DRM solutions are based on the *MPI World model* and use
*MPI.Comm_spawn*, a mechanism provided by MPI since the
standard version 2, which creates new ranks and communicates

them with the invoking ranks via an inter-communicator
between the original and spawned communicators.

Frameworks such as FlexMPI [10], AMPI [11], Elas-
ticMPI [12], OmpSs2@Cluster [13], and Proteo [14] rely on
MPI.Comm_spawn to create new processes. Although they
all share the same philosophy of spawning processes, each
tackles job reconfiguration differently.

FlexMPI has extended MPICH by providing functions for
registering data structures to assist data redistribution op-
erations. FlexMPI leverages MPI.Comm_split to remove
ranks from communicators for finally terminating them when
shrinking jobs.

Adaptive MPI (AMPI) implements MPI on top of
CHARM++'s adaptive runtime system [15]. CHARM++ mi-
gratable objects, named *chares*, are redistributed among the
new number of processes to resume the execution.

ElasticMPI proposes an extension to the MPICH library
to allow the dynamic reshaping of the processes’ layout on
large-scale distributed memory systems. Functions such as
MPI_Init_adapt or MPI_Probe_adapt are included in the new
definition of MPI to support resizing the MPI_COMM_WORLD
communicator. ElasticMPI is integrated with Slurm enabling
system-driven job reconfiguration decisions.

OmpSs-2@Cluster [16] is a task-based programming model
on top of an MPI-(spawn)-based dynamic runtime that auto-
matically redistributes the application’s data and subsequently
schedules OmpSs tasks across the new number of nodes. While
task-based programming models provide an easy route for
new applications to support malleability as the application is
written in a way that is generally independent of the underlying
compute resources, porting existing MPI applications to task-
based programming models requires rewrites of entire appli-
cations.

However, the *MPI World model* and *MPI.Comm_spawn* have limitations in their flexibility to support dynamic process reconfiguration for more complex applications [6], [12]. Moreover, past approaches used a vertical, end-to-end design that is specialized on only particular application types, therefore lacking genericity in the lower-level system software. The objective of our work is to overcome these limitations by embedding a specialized interface into a generic dynamic resource management approach.

The MPI Standard 4.0 presents the *MPI Sessions model* [17], which enhances resource flexibility by eliminating the global *MPI COMM WORLD* communicator and providing closer integration with runtime information. It introduces MPI process sets (PSets) as a way to establish communicators in a more modular and scalable manner. PSets can be referenced in a dictionary using URIs corresponding to specific sets of processes. However, the standard currently does not specify further details on dynamic process management with PSets beyond this point.

These new MPI PSets motivated an approach called Dy-
namic Processes with Process Sets (DPP) [18], [19], an effort
to realize generic design principles for dynamic resources in
parallel programming models [6].
A mandatory drawback of using a generic dynamic re-source interface is its increased complexity, hence, higher entry barrier. Contrary, DMR-API is an application-level layer that abstracts the distributed parallel runtime methods and the resource management system (RMS) logic that provide dynamic resource management capabilities [20].

In this paper, we combine the DMR-API with DPP in a layered approach to leverage the best from both sides: Due to its high level of abstraction, the DMR-API provides better programmability for iteration-based algorithms, which represent a large share among HPC applications. On the contrary, the DPP approach supports a variety of parallel algorithmic patterns but through a programming layer of increased complexity. The combination of both approaches will be part of the present paper leading to high programmability while avoiding the specialization dead end on the backend side. We approach this challenge by embedding the DMR-API features into the graph-based abstraction of DPP, and implement the dynamic graph execution in the DMR-API backend. Therefore, in the next section, we provide the necessary background about DPP and DMR-API.

### III. SOFTWARE STACK

Part of the research presented in this paper involves integrating the backend DPP and the frontend DMR-API into the same software stack.

#### A. DPP

Dynamic Processes with PSets (DPP) [6], [18], [19] is a set of design principles for dynamic resources in parallel programming models. It is based on a co-design between applications and resource management software to provide the required flexibility for application developers to express their applications' particular dynamic resource patterns (iterative [19], in-situ [21], graph-based, etc.) while allowing the resource management software to optimize the resource assignments globally. For the sake of brevity, here we only briefly summarize the most important aspects of DPP and refer the readers to [6] for more details.

* DPP uses a separation of concerns between dynamic process management (DPM) from the application perspective and Dynamic Resource Allocation from the resource manager perspective.

* DPP uses Process Sets (PSets) and Process Set Operations (PSetOps) for DPM as a common abstraction between the application and the resource manager to express dynamic process reconfigurations. PSetOps are set operations (such as ADD, SUB, UNION, DIFFERENCE, ...) defined on a list of input PSets and can be executed by the resource manager to create a list of output PSets. This leads to a graph-based representation of process/resource (re-)assignments, where vertices of the graph represent PSets and directed hyperedges represent PSetOps. Applications define set operations to express their complex dynamic resource requirements generically while enabling the resource manager to optimize the

corresponding dynamic resource allocation based on a global optimization objective.

* Optimization information is expressed in a Cooperative Optimization Language (COL) and associated with PSet-s/PSetOps. The resource manager uses this optimization information to decide on the execution of PSetOps and the mapping of PSets on system resources. The development and standardization of the COL as a Domain-Specific Language for Dynamic Resource Management is considered future work.

Based on the DPP design principles corresponding extensions to MPI (Sessions) and PMIx have been implemented as a demonstrator software stack [22] based on Open-MPI, OpenPMIx, and PRRTE. The extended interface adds two basic functionalities to the MPI Sessions routines defined in the MPI Standard 4.0. First, routines for interacting with the resource manager by sending (MPI_Session_psetop) and querying (MPI_Session_psetop_query) process set operations which express resource reconfigurations. Second, routines to access a global dictionary (MPI_Session publish/lookup_data) where key-value pairs can be associated with a particular PSet URI for sharing application-specific information across PSets. The software stack is integrated with DynRM [23], a modular, python-based dynamic resource manager implementation supporting the DPP design principles to facilitate the exploration of new dynamic scheduling strategies. In this work, we use this software stack for our demonstrator implementation and DynRM as the resource management software for our experiments.

As the DPP interface is not specialized on this particular application type, but offers generic mechanisms for expressing dynamic resource changes for a large spectrum of HPC applications, its usage requires substantial efforts from application developers. Therefore, providing a convenience programming layer to DPP specialized on particular parallelization patterns of applications would strongly ease its adoption while maintaining a generic mechanism to interact with the resource management software.

#### B. DMR-API

The Dynamic Management of Resources Application Programming Interface (DMR-API) is a high-level API that facilitates the adoption of dynamicity in HPC codes. Particularly, DMR-API can abstract the use of different MPI dynamic solutions into the same syntax. DMR-API provides users with a series of variables, functions, and macros to enable dynamicity in MPI codes. Users can leverage variables to handle MPI communicators or enable predefined data redistribution patterns. DMR-API also allows users to parameterize dynamicity behavior via the functions for configuring reconfiguration periods and limits. An illustration of the usage of DMR-API in an iterative workload is given in Fig 5.

The DMR-API core is its preprocessor macros that abstract all the complexity when implementing job reconfigurations.
* **DMR_INIT**: This macro initializes the DMR environment and expects user functions to initialize the application's data and receive data after a reconfiguration. The macro expects two user functions: 1) To initialize the environment, users must provide a function for the initial processes to set up the application data; 2) To receive the application data provided by the running processes.

* **DMR_RECONFIGURATION**: This triggers the dynamicity and is placed wherever a reconfiguration may occur. It communicates with the RMS, reshapes the process layout, and redistributes the application's data by leveraging user functions. In this case, the macro expects three functions to handle sending and receiving data among the joining, staying and leaving processes.

* **DMR_FINALIZE**: It is invoked at the end of the execution to deallocate memory and collect garbage. It expects a function that will be invoked before finalizing MPI and will be used for any user-level finalization steps, in particular those requiring MPI communicators (such as parallel I/O).

DMR-API provides users with special MPI communicators and groups for use during the program execution:

* **DMR_comm**: The main communicator provided by DMR-API. This communicator contains the current application worker processes. It is created in `DMR_INIT` and destroyed in `DMR_FINALIZE`. The size of `DMR_comm` can change in `DMR_RECONFIGURATION`.

* **DMR_group_world**: Contains a group of processes that are launched together (either during initial job launch or during runtime). It is created in `DMR_INIT` and destroyed in `DMR_FINALIZE`. It remains unchanged throughout the execution.

* **DMR_comm_old/DMR_comm_new**: These communicators are accessible from within the user functions provided to DMR-API macros. They represent the group of processes contained in `DMR_comm` before and after a reconfiguration is triggered, respectively. `DMR_comm_old` and `DMR_comm_new`. For instance, if an expansion takes place in `DMR_RECONFIGURATION`, `DMR_comm_old` only contains the original processes, while `DMR_comm_new` contains both the original processes and the new processes. Users can use these communicators, e.g., to determine the necessary communication for the data redistribution.

Redistribution of application data during process reconfiguration is usually highly application-specific. Thus, DMR-API macros generally call user-provided functions when necessary to redistribute application data from `DMR_comm_old` to `DMR_comm_new` during a reconfiguration. To this end, users can leverage various DMR-API helper variables and functions or external libraries to implement their custom redistribution logic.

# IV. EMBEDDING DMR-API IN DPP

This section describes our methodology for embedding the lower-dimensional feature space of DMR-API into the

<figure><img src="image_2.png" /><figcaption>Fig. 1: DMR-API PSet Graph for an example application with two reconfigurations.</figcaption></figure>

higher-dimensional feature space of DPP. We start from an abstract point of view in Sec. IV-A, where we use the graph-based perspective of DPP to describe the dynamic behavior of a DMR-API-enabled application. Subsequently, we illustrate how this graph description can be implemented in the DMR-API backend using DPP. Finally, we provide a direct comparison of a dynamic iterative code using plain DPP and DMR-API+DPP respectively. Although, in this work, we focus on embedding DMR-API, a similar approach can be used to create new interfaces for other application types embedded into the DPP feature space.

## A. DMR-API PSet Graph

As a first step, we map the dynamic resource behavior of an DMR-API-enabled application to the *PSet* and *PSet Operation* abstraction of DPP (cf. III-A). This leads to a characteristic *PSet Graph* for such applications. The *PSet* graph of an application is a generic representation of its (dynamic) resource assignments over time. Such a representation is mandatory for the resource manager to be able to include different application types in a unified dynamic resource optimization problem.

Fig. 1 illustrates the *PSet* graph for an example execution of an DMR-API-enabled application. The application 1) starts with two processes, 2) grows to 4 processes, 3) shrinks back to two processes, and 4) terminates. Note that a DMR-API application generally has a sequence of such grow/shrink reconfigurations, which depends on scheduling decisions. However, the basic structure of the *PSet* graph is similar for all DMR-API applications. These four steps are represented by four *PSet* operations (hyperedges between vertices in the *PSet Graph*):

* **Application start**: The application start is modeled by an ADD operation on the empty *PSet* (a *PSet* with zero processes). The output of this operation is a single *PSet* (`pset://main/0`) in Fig. 1 containing the initial processes `p0` and `p1`.

* **Expansion**: The application expansion is modelled by a GROINK operation on `pset://main/0`. A GROINK operation indicates the application's willingness to add and/or remove processes for the given input *PSet*. It has three outputs *PSets*: 1) a $\delta_{add}$ *PSet* containing the processes to be added (`PSet pset://add/0` containing processes `p2` and `p3`), 2) a $\delta_{sub}$ *PSet* containing the processes
to be removed (*pset://empty* as in case of expansion
no processes are removed) and a PSet representing the
state after the PSetOp, i.e., *PSet<in> \ delta<sub>sub</sub> ∪ delta<sub>add</sub>*
(*pset://main/1* containing processes *p0*, *p1*, *p2*, *p3*).

* **Shrinkage:** Shrinkage is modeled analogously to the expansion using a GROINK operation on (*pset://main/1*). In this case *delta<sub>add</sub>* is *pset://empty* as no processes are added and *delta<sub>sub</sub>* (*pset://sub/0*) contains the processes to be removed (*p1* and *p2*). The resulting PSet *pset://main/2* contains the staying processes (*p0* and *p3*).

* Termination: Termination is modeled by a SUB operation on PSet *pset://main/2*. The output PSet is *pset://empty* i.e., *p0* and *p3* terminate.

Such a PSet graph describes the execution of a DMR-API-
enabled application, however, its concrete instantiation during
runtime is based on interactions between the application and
the resource manager:

* The application provides the input PSets and the type of a PSetOp (possibly with additional optimization information attached) to the resource manager.

* The resource manager considers PSetOps as a potential reconfiguration action in its scheduling policy and eventually provides the concrete output PSets to the application (depending on resource availability and optimization objectives).

* The application can then reconfigure itself according to the output PSets of the operation.

To implement this mechanism in the DMR-API backend
(and therefore hide its complexities from users) we used
the DPP-based MPI Sessions extensions as described in the
following subsection.

B. DMR-API embedding into DPP

In this subsection, we describe how the mapping of DMR-
API to DPP was implemented using the extended functional-
ities of the MPI Sessions interface (see Sec. III-A) and PSet
graph (see Sec. IV-A). For clarity, we omit some implemen-
tation details in this subsection.

Fig. 2 gives an overview of the basic application structure
(left column), the usage of the DMR-API interface and its
internal implementation (middle column), and the correspond-
ing DPP mapping (right column). The details for expanding
and shrinking the DMR-API communicator are illustrated
separately in Fig. 3.

1) *Basic Application Structure:* We start by describing the
3 basic phases of the DMR-API application:

**Application Initialization:** In the initialization phase, the application runs the *DMR_INIT* macro passing the *user_init()* and *expand_recv()* functions. The *DMR_INIT* macro first initializes a new MPI Session. Subsequently, it creates the global *DMR_comm* communicator and *DMR_Group_world* from the *mpi://WORLD* PSet. The *mpi://WORLD* PSet is mandated by the MPI Standard and contains all the processes created by a common process launch. Hence, for the initial application processes, the *mpi://WORLD* URI is an alias for the PSet *main/0*, which is the output of

<figure><img src="image_3.png" /><figcaption>Fig. 2: DMR-API implementation using MPI Sessions and DPP. Orange boxes indicate standard MPI functions, green boxes indicate functions of the extended MPI Sessions interface, and blue boxes indicate user-provided functions.</figcaption></figure>

the initial ADD operation in the DPP PSet graph. Finally, the
provided *user_init()* function is called to initialize the
user data.

**Main Loop:** In the main loop, the application executes some application-specific iterative computations (indicated by *compute* in Fig. 2) using the *DMR_comm*. When a synchronization point is reached (typically at the end of an iteration), the *DMR_RECONFIGURATION* macro is executed. This macro behaves differently depending on the current state of the DPP PSet Graph:

* **No PSetOp:** If DMR-API has not yet indicated support for a PSetOp on the current main PSet, it sends a new PSetOp of type GROINK to the resource manager using the *MPI Session psetop* function. The input of the PSetOp is the PSet that was used to create the current *DMR_comm*. Optionally, additional optimization information could be associated with this operation. When receiving the PSetOp, the resource manager includes it in its scheduling optimization.

* **Pending PSetOp:** If DMR-API has already sent a PSetOp to the resource manager, it uses the *MPI Session psetop_query* function to query for the list of output sets of the set operation. If the resource manager has not yet decided to execute the PSetOp (i.e., the PSetOp is pending), the returned list of output PSets will be empty, and no resizing action is required in this iteration.

* **Executed PSetOp:** If the resource manager has decided to execute a PSetOp, *MPI Session psetop_query* will return a non-empty list of output PSets, and a resizing action is required in this iteration. The details of the
resizing action are provided in subsections IV-B2 and IV-B3.

**Application Finalization:** When exiting the main loop, the application finalizes by calling `DMR_FINALIZE`. This macro first executes the provided `user_finalize()` function, calls `MPI_Comms_disconnect` on the `DMR_comm` and finalizes the MPI Session. The termination of the last DMR-API processes (i.e., processes not terminating due to a resize decision) is modeled by a SUB operation on the current DMR-API PSet.

<figure><img src="image_4.png" /><figcaption>Fig. 3: **DMR-API implementation of expansion (top) and shrinkage (bottom) using MPI Sessions and DPP.** Orange boxes indicate standard MPI functions, green boxes indicate functions of the extended MPI Sessions interface, blue boxes indicate user-provided functions, pastel boxes represent MPI communicators. For the sake of visualization, the overlap `DMR_comm_old` and `DMR_comm_new` during the data redistribution is not illustrated in the figures.</figcaption></figure>

2) **DMR-API Expansion:** In this subsection, we describe the details of the case when the resource manager executed an expansion GROINK operation. In this scenario, the runtime has launched new application processes and executes `DMR_INIT`. After initializing an MPI Session, these processes need to join the currently running processes in the `DMR_RECONFIGURATION` call. For this, the currently running processes query the output PSets of the PSetOp. Then rank #0 calls `MPISession_pset publish` to publish

the name of the new main PSet (`main/1`) with the key `main` in the dictionary associated with PSet `add/0` (which is the `mpi://WORLD` PSet for the joining processes). The joining processes use `MPISession_pset_lookup` to look up the key-value pair. Subsequently, all processes (current and joining) create a new `DMR_comm` from PSet `main/1` and synchronize the `DMR_iter` counter among all processes using `MPI_Bcast`. Finally, the data redistribution is executed using the provided `send侯 expand`/`send_recv` user functions and the original processes call `MPI_Comms_disconnect` on the old `DMR_comm`.

3) **DMR-API Shrinkage:** Similar to the expansion, the processes in `DMR_RECONFIGURATION` query the list of output PSets of the GROINK operation. This allows the processes to determine if they are included in the new main PSet `main/2` (staying processes) or not (leaving processes). Only the staying processes create the new `DMR_comm` from the new main PSet `main/2`. Subsequently, the application data is redistributed using the user-provided functions `shrink_recv` (staying processes) and `shrink_send` (leaving processes). The leaving processes then break out of the main loop while the staying processes call `MPI_Comms_disconnect` on the current DMR-API communicator. This disconnect call synchronizes with the disconnect call in `DMR_FINALIZE` called by the leaving processes after breaking out of the main loop.

### C. Code comparison

To give an impression of the usability enhancement that DMR-API adds compared to using DPP directly we provide a code comparison for a simple iterative application.

For the sake of readability, we show pseudo-code for the plain DPP example in Fig. 4, while we show actual C code for the DMR example in Fig. 5. The equivalent code regions are colored with the same color. Comparing both codes, with DMR-API we use 85% fewer (9 vs. 61) lines of code (LoC). In reality, the C-based backend DPP- implementation for DMR-API needs 245 LoC. Moreover, DMR-API is easier to use and less prone to errors compared to the plain DPP code. Therefore, DMR-API provides a high degree of programmability for the iteration-based application, while relying on a generic representation in the backend as shown in Sec. IV-A.

## V. DEMONSTRATOR AND EVALUATION

While we have demonstrated the benefits of our approach in terms of programmability in Sec. IV-C, this section is intended to evaluate the applicability and performance of the approach in a dynamic resource management scenario on a modern HPC system.

### A. Demonstrator Implementation

We implemented a demonstrator of the DPP-DMR-API mapping [24] as described in the previous section. For our demonstrator evaluation, we also extended the DynRM resource manager with a new policy module implementing the dynamic min-max-pref reconfiguration strategy [25] which was previously used to support DMR-API in SLURM.
```c
MPI Session init();
info = MPI Session get pset info("mpi://WORLD");
if (info.dpp_dynamic)
    pset = MPI_DPP_pset lookup("mpi://WORLD", "main");
if (pset == "cancel") {
    MPI Session finalize();
    return;
}
} else {
    pset = "mpi://WORLD"
}
group = MPI Group from session pset (pset);
comm = MPI Comm create from group (group);
if (info.dpp_dynamic) {
    expand_recv();
} else {
    user_init();
}
params = {min, max, pref};
```

```c
for (int it = 0; it < ITERATIONS; it++) {
    compute();
    if (comm_rank == 0 && !pending &&
        it % num_iters == 0) {
        MPI_DPP_psetop_nb(GROINK, pset, params );
        pending = 1;
    }
    op, out_psets = MPI_DPP_psetop_query(pset);
    if (op != OP_NULL) {
        if (out_psets[0] != "") {
            if (out_psets[0].member) {
                shrink_send();
                break;
            } else {
                shrink_recv();
            }
        } else if (out_psets[1] != "" && comm_rank == 0) {
            MPI_DPP_pset_pub(out_psets[1], "main", pset);
            MPI_DPP_psetop_finalize(pset);
        }
        pending = 0;
        pset = out_psets[2];
        MPI Comm disconnet (comm);
        group = MPI Group from session pset (pset);
        comm = MPI Comm create from group (group);
        if (out_psets[1] != "") {
            expand_send();
        }
    }
}
user_finalize();
if (pending && comm_rank == 0) {
    op, _ = MPI_DPP_psetop(CANCEL, pset);
    if (op == MPI PSETOP_NULL) {
        op, out_psets = MPI_DPP_psetop_query(pset);
        MPI_DPP_pset_pub(out_psets[2], main, "cancel");
        MPI_DPP_psetop_finalize(pset);
    }
}
MPI Comm disconnet (comm);
MPI Session finalize();
```

<caption>Fig. 4: Pseudo-code for a resource-dynamic application with an iterative algorithmic pattern using plain MPI+DPP. For the sake of clarity, we use the MPI_DPP prefix to differentiate the DPP MPI-extensions from standard MPI Sessions routines.</caption>

```c
DMR_INIT(n_iter, user_init(), expand_recv());
DMR_Set_parameters(min, max, pref);
DMR Inhibit_iter (num_iters);
for (int it = 0; it < ITERATIONS; it++) {
    compute();
    DMR_RECONFIGURATION (expand_send(),
                        shrink_send(), shrink_recv());
}
DMR_FINALIZE (user_finalize());
```

<caption>Fig. 5: Code for a resource-dynamic application with an iterative algorithmic pattern using DMR-API.</caption>

With the demonstrator, we have implemented several dynamic codes that represent typical iterative patterns of HPC applications. They are based on the DMR-API interface to interface the DPP software stack, as shown in Fig. 6:

* **Jacobi:** The Jacobi method is an iterative algorithm for determining the solutions of a strictly diagonally dominant system of linear equations $Ax = b$ [26]. We use a randomly generated system matrix $A$ stored in a dense format, a right-hand-side vector $b_i = i$ and an initial solution guess $x_i = 1$.

* **Conjugate Gradient (CG)** The Conjugate Gradient (CG) solver is the most prominent Krylov-subspace-based iterative method for solving sparse systems of linear equations which is also included in our benchmarks. It is based on Krylov subspaces ($A$, $A^2$, $A^3$, ...) spanned up iteratively with matrix-vector multiplications and vector-based reduce operations. Further information can be found in [27].

* **Laplace:** The last example solves the 2D Laplace equation

$$\nabla^2 f(x, y) = 0$$

representing the steady-state heat equation [28] with a linearly varying Dirichlet boundary condition. A Jacobi iterative solver is used in each iteration to update the grid.

The implementations [29] of Jacobi, CG and Laplace are 2D MPI-based iterative with a single communicator. Their data are stored vectorized in 1D arrays to facilitate data redistribution operations. These codes have also been leveraged in other dynamicity studies with DMR and other reconfiguration engines [7], [30] and represent typical iterative kernels frequently used in PDE solver frameworks.

## B. Target Platform and Software

The benchmarks were executed on the LRZ SuperMUC-NG Phase 2 [31]. We used its Thin Nodes consisting of 2 sockets, each featuring an Intel Skylake Xeon Platinum 8174 processor with 24 processes and two hardware threads per process. The total RAM per node is 96 GB, and nodes are connected by an OmniPath network with up to 100 Gbit/s using a FatTree topology inside each island (all tests were performed inside a single island).
<figure><img src="image_5.png" /><figcaption>Fig. 6: Overview of the demonstrator software stack.</figcaption></figure>

<table><caption>TABLE I: Application parameters for Benchmark #2.</caption><thead><tr><th>Application</th><th>Parameter</th><th>Value</th><th>Description</th></tr></thead><tbody><tr><td rowspan="6">Jacobi</td><td><i>N</i></td><td>2<sup>16</sup></td><td>Number of gridpoints</td></tr><tr><td><i>iters</i></td><td>4500</td><td>Number of iterations</td></tr><tr><td>min</td><td>32</td><td>Min. num. of processes</td></tr><tr><td>max</td><td>512</td><td>Max num. of processes</td></tr><tr><td>pref</td><td>256</td><td>Pref num. of processes</td></tr><tr><td>pow2</td><td>False</td><td>only power of 2 processes</td></tr><tr><td rowspan="6">Laplace</td><td><i>N</i></td><td>2<sup>15</sup> × 2<sup>15</sup></td><td>Number of gridpoints</td></tr><tr><td><i>iters</i></td><td>5000</td><td>Number of iterations</td></tr><tr><td>min</td><td>32</td><td>Min. num. of processes</td></tr><tr><td>max</td><td>256</td><td>Max num. of processes</td></tr><tr><td>pref</td><td>128</td><td>Pref num. of processes</td></tr><tr><td>pow2</td><td>True</td><td>only power of 2 processes</td></tr><tr><td rowspan="6">CG</td><td><i>N</i></td><td>2<sup>15</sup></td><td>Number of gridpoints</td></tr><tr><td><i>iters</i></td><td>5000</td><td>Number of iterations</td></tr><tr><td>min</td><td>32</td><td>Min. num. of processes</td></tr><tr><td>max</td><td>512</td><td>Max num. of processes</td></tr><tr><td>pref</td><td>128</td><td>Pref num. of processes</td></tr><tr><td>pow2</td><td>True</td><td>only power of 2 processes</td></tr><tr><td></td><td>exclude</td><td>256</td><td>prohibited num. of processes</td></tr></tbody></table>

We used the Spack Package Manager [32] to install DMR-API, DynRM, Open-MPI, OpenPMIx, and PRRTE in a dedicated spack environment. The used spack packages and spack environments are publicly available [33].

All of the tests were performed inside of SLURM allocations in the micro/general partition with Energy Aware Runtime (EAR) [34] disabled.

### C. Benchmark Description

We used two different benchmarks to assess the impact of our approach on individual application performance and global scheduling productivity.

1) **Benchmark #1: Application Scalability and Reconfiguration Overhead:** The goal of this benchmark is to characterize application behavior when increasing or decreasing the number of processes.

For the expansion, the application starts with 32 processes (1 node), and the number of processes is doubled every 200 iterations up to 512 processes (16 nodes). For shrinkage,

the application starts with 512 processes (16 nodes), and the number of processes is halved every 200 iterations down to 32 processes (1 node).

For each iteration, we measure the time required for compute(). For each reconfiguration (after 200 iterations), we measure the time required for DMR_RECONFIGURATION, where we differentiate between the *data redistribution* and the *MPI reconfiguration* overhead. Note that the *data redistribution* overhead is application-specific and depends on the user-provided redistribution functions.

2) **Benchmark #2: Performance of Dynamic Scheduling:** To assess the impact of our approach on scheduling performance, we compare a dynamic scheduling policy (min-max-pref DMR Policy as introduced in [25]) with a static scheduling policy (EasyBackfilling). To this end, we run our demonstrator implementation inside a 64-node SLURM allocation where our scheduler manages the assignment of nodes to workloads inside this allocation.

We use the same 20-job mix in the static and dynamic cases. Jobs are randomly chosen from our three adapted applications. With the Feitelson model [35], job inter-arrival time is determined with a factor of 10, representing a scenario where jobs are rapidly filling up the queue with a short delay that fits the Poisson distribution of the model. The applications are executed with the parameters listed in Table I.

For the static scheduling, the applications are started with either the *pref* number of processes (static_pref) or the *max* number of processes (static_max). For the dynamic scheduling, the jobs are started with *pref* processes (dynamic_pref), i.e., the applications are not moldable at launch but could be reconfigured inside of the min-max range during runtime based on scheduling decisions.

For each of these cases, we measure the makespan, average pending time, average runtime, average turnaround time, and average node utilization.

### D. Results

This section presents the results for the two benchmarks described above.

1) **Benchmark #1:** Fig. 7 summarizes the results of Benchmark #1 for Jacobi, CG, and Laplace solver.

It can be seen that the applications exhibit different strong scaling behavior. We used these scaling results to determine appropriate DMR parameters min, max and pref as listed in Table I.

The data redistribution overhead varies significantly among the applications, as illustrated on the logarithmic scale in Fig. 7. The Laplace application exhibits a notable data redistribution overhead, which increases as the number of processes rises. However, it is important to clarify that this overhead is not necessarily higher in the absolute sense; rather, it is a consequence of the choice of a much larger solution domain, as no $N \times N$ system matrix needs to be stored. In this case, the entire solution domain is redistributed across processes. Additionally, an extra halo exchange is needed to ensure that neighboring processes have the necessary boundary data.
<figure><img src="image_6.png" /><figcaption>Fig. 7: Results for Benchmark #1 for expansions (left) and shrinkage (right). The top row shows the median time required for 25 (Jacobi), 165 (Laplace), and 100 (CG) iterations of the user workload with the given number of processes. The middle row shows the time required for data redistribution when doubling, halving the given number of processes. The bottom row shows the corresponding overhead due to MPI reconfigurations.</figcaption></figure>

In contrast, for CG and Jacobi, the redistribution overhead is considerably lower and tends to decrease with a greater number of processes. In these cases, only the solution vector is redistributed among the processes, while the matrix itself can be regenerated locally, as it remains constant throughout the execution. This regeneration process appears to dominate the total overhead. As the size of the matrix slice per process decreases with an increasing number of processes, the data redistribution overhead decreases accordingly. These results underline the importance of efficient data redistribution algorithms for dynamic resource management and its consideration in reconfiguration decisions.

The overhead of the MPI reconfiguration is similar for the three applications and slightly increases with an increasing number of processes. Overall, the overhead is low (in the range of 50–130 ms) compared to the data redistribution overheads in most cases. Moreover, there is no limitation on the granularity of adding/removing processes. This clearly shows the benefits of using an MPI Sessions-based approach compared to previous blocking `MPIComm spawn` based approaches, where high reconfiguration overheads were reported [14].

2) **Benchmark #2:** Fig. 8 presents the results of Benchmark #2. We use `static_pref` as the baseline and compare it to `static_max` and `dynamic_pref`. The results are shown for the following metrics:

$$ makespan = \max_{i} t_{end}^{i} - \min_{i} t_{sub}^{i}, $$

$$ avg_queueing_time = \frac{1}{J} \sum_{i=0}^{J-1} (t_{start}^{i} - t_{sub}^{i}), $$

<figure><img src="image_7.png" /><figcaption>Fig. 8: Results for Benchmark #2. Results are shown relative to `static_pref`. The lower, the better for all metrics except for the average utilization rate.</figcaption></figure>

$$ avg_run_time = \frac{1}{J} \sum_{i=0}^{J-1} (t_{end}^{i} - t_{start}^{i}), $$

$$ avg Clerk's time = \frac{1}{J} \sum_{i=0}^{J-1} (t_{end}^{i} - t_{sub}^{i}), $$

$$ avg\_utilization = \frac{\sum_{i=0}^{N-1} t_{util}^{i}}{\sum_{i=0}^{N-1} t_{util}^{i} + \sum_{i=0}^{N-1} t_{idle}^{i}}, $$

where $t_{sub}^i$, $t_{start}^i$, $t_{end}^i$ are the submission, start and end times of job i respectively, J is the number of jobs, $t_{util}^i$ and $t_{idle}^i$ are the utilized and idle times of node i respectively, and N is the number of nodes.
Comparing *static_max* to *static_pref*, it can be seen that the average runtime of the applications is reduced and the utilization rate increased. However, the average pending time is significantly higher, counteracting these benefits. This results in a higher average turnaround time and longer makespan.

For *dynamic_pref*, the average runtime of the applications is slightly higher compared to *static_pref*. However, the average pending time is significantly shorter than static scheduling, reducing average turnaround time and makespan. Moreover, the average utilization rate is higher than with static scheduling.

These results indicate the potential for more flexible, dynamic scheduling of reconfigurable jobs. The reconfiguration of jobs at runtime allows the scheduler to start more jobs from the job queue by shrinking running jobs if the job queue is crowded (reducing average pending time). On the other hand, it allows jobs to expand if there are free resources in the system or when fragmentation prevents more jobs from being started (improving utilization rate). In our experiment, we used a simple heuristic [25] based on *min*, *max*, and *pref* values, without considering the exact scalability, redistribution overhead, or frequency of reaching the synchronization point for each application. We expect more sophisticated dynamic scheduling strategies to achieve even better performance.

# VI. CONCLUSION

The increasing number and scale of HPC systems and the associated increasing energy consumption require research on new techniques to improve resource utilization efficiency on these systems. While research on Dynamic Resource Management is promising, its integration into production systems remains challenging. One major challenge is bridging the gap between the programmability at the application layer and the genericity in the lower system software layers.

In this work, we addressed this challenge by combining two dynamic resource approaches - DMR-API and DPP. We embedded the high-level abstractions of the DMR-API interface into the lower-level abstractions of DPP. To this end, we used the graph-based perspective of DPP to describe the dynamic reconfiguration pattern of DMR-API-based applications. The dynamic instantiation of this graph during runtime is based on interactions with the resource manager and implemented in the DMR-API backend using the generic interface of DPP. With this approach, we achieve high programmability from the application developer's perspective while relying on a generic mechanism to interact with lower-level system software.

As a proof of concept, we developed a demonstrator implementation and integrated it into a dynamic resources software stack. Based on a single-application benchmark and a multi-application scheduling benchmark, we assessed the overheads of resource reconfigurations and the impact of our approach on scheduling performance. The results confirm the applicability of our approach for dynamic resource management in HPC and highlight the benefits of combining the genericity of the DPP approach with the programmability of DMR-API.

Overall, this work represents one of many crucial, major steps to make dynamic resources more accessible and suitable for production systems.

# ACKNOWLEDGEMENTS

This project has received funding from the Federal Ministry of Education and Research and the European HPC Joint Undertaking (JU) under grant agreement No 955701, Time-X, No 955606, DEEP-SEA and No 956560, REGALE. The JU receives support from the European Unions Horizon 2020 research and innovation programme and Belgium, France, Germany, Switzerland. This work was supported by a French government grant managed by the Agence Nationale de la Recherche under the France 2030 program, reference "ANR-23-PECL-0008. The researchers from BSC are involved in the project The European PILOT, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreements No. 101034126 and No. PCI2021-122090-2A under the MCIN/AEI and the EU NextGenerationEU/PRTR. They are also grateful for the support from the Department of Research and Universities of the Government of Catalonia to the AccMem project (Code: 2021 SGR 00807). Antonio J. Peña was partially supported by the Ramón y Cajal fellowship RYC2020-030054-I funded by MCIN/AEI/ 10.13039/501100011033 and by "ESF Investing in your future".

# REFERENCES

[1] M. Schreiber, C. Riesinger, T. Neckel, H. J. Bungartz, and A. Breuer, "Invasive compute balancing for applications with shared and hybrid parallelization," *International Journal of Parallel Programming*, vol. 43, no. 6, pp. 1004-1027, 2015.

[2] T. Jin, F. Zhang, Q. Sun, M. Romanus, H. Bui, and M. Parashar, "Towards autonomic data management for staging-based coupled scientific workflows," *Journal of Parallel and Distributed Computing*, vol. 146, pp. 35-51, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0743731520303312

[3] S. Prabhakaran, "Dynamic resource management and job scheduling for high performance computing," 2016. [Online]. Available: https://api semanticscholar org/CorpusID 34068049

[4] T. Özden, T. Beringer, A. Mazaheri, H. M. Fard, and F. Wolf, "Elastisim: A batch-system simulator for malleable workloads," in *Proceedings of the 51st International Conference on Parallel Processing*, ser. ICPP '22. New York, NY, USA: ACM, 2023.

[5] P. Finnerty, J. Posner, J. Bürger, L. Takaoka, and T. Kanzaki, "On the performance of malleable APGAS programs and batch job schedulers," *SN Computer Science*, vol. 5, 03 2024.

[6] D. Huber, M. Schreiber, M. Schulz, H. Pritchard, and D. Holmes, "Design principles of dynamic resource management for high-performance parallel programming models," 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.17107

[7] S. Iserte, "High-throughput computation through efficient resource management," PhD Thesis, Universitat Jaume I, Castelló de la Plana, Nov. 2018. [Online]. Available: http://hdl.handle.net/10803/664128

[8] J. I. Aliaga, M. Castillo, S. Iserte, I. Martín-Álvarez, and R. Mayo, "A survey on malleability solutions for high-performance distributed computing," vol. 12, pp. 1-32.

[9] A. Tarraf, M. Schreiber, A. Cascajo, J.-B. Besnard, M.-A. Vef, D. Huber, S. Happ, A. Brinkmann, D. E. Singh, H.-C. Hoppe, A. Miranda, A. J. Peña, R. Machado, M. G. Gasulla, M. Schulz, P. Carpenter, S. Pickartz, T. Rotaru, S. Iserte, V. Lopez, J. Ejarque, H. Sirwani, and F. Wolf, "Malleability in modern HPC systems: Current experiences, challenges, and future opportunities," pp. 1-14, conference Name: IEEE Transactions on Parallel and Distributed Systems. [Online]. Available: https://ieeexplore.ieee.org/document/10541114
[10] G. Martín, M.-C. Marinescu, D. E. Singh, and J. Carretero, “FLEX-MPI: An MPI extension for supporting dynamic load balancing on heterogeneous non-dedicated systems,” in *Euro-Par Parallel Processing*, Aug. 2013, pp. 138–149.

[11] A. Gupta, B. Acun, O. Sarood, and L. V. Kalé, “Towards realizing the potential of malleable jobs,” in *21st HiPC*, 2014.

[12] I. Comprés, A. Mo-Hellenbrand, M. Gerndt, and H.-J. Bungartz, “Infras-structure and API extensions for elastic execution of MPI applications,” in *23rd EuroMPI*, 2016, pp. 82–97.

[13] J. Aguilar Mena, “Methodology for malleable applications on distributed memory systems,” Ph.D. dissertation, Universitat Politècnica de Catalunya, 11 2022. [Online]. Available: http://hdl.handle.net/2117/380814

[14] I. Martín-Álvarez, J. I. Aliaga, M. Castillo, and S. Iserte, “Proteo: A framework for the generation and evaluation of malleable MPI applications.” [Online]. Available: https://doi.org/10.1007/s11227-024-06277-5

[15] B. Acun et al., “Parallel programming with migratable objects: CHARM++ in practice,” in *SC14*. IEEE, Nov 2014, pp. 647–658.

[16] J. Aguilar Mena, O. Shaaban, V. Beltran, P. Carpenter, E. Ayguade, and J. Labarta Mancho, “Ompss-2@cluster: Distributed memory execution of nested openmp-style tasks,” in *Euro-Par 2022: Parallel Processing: 28th International Conference on Parallel and Distributed Computing, Glasgow, UK, August 22–26, 2022, Proceedings*. Berlin, Heidelberg: Springer-Verlag, 2022, p. 319–334. [Online]. Available: https://doi.org/10.1007/978-3-031-12597-3_20

[17] D. Holmes, K. Mohror, R. Grant, A. Skjellum, M. Schulz, W. Bland, and J. Squyres, “MPI Sessions: Leveraging runtime infrastructure to increase scalability of applications at exascale,” in *Proceedings of the 23rd European MPI Users’ Group Meeting*, 2016, pp. 121–129.

[18] J. Fecht, M. Schreiber, M. Schulz, H. Pritchard, and D. J. Holmes, “An emulation layer for dynamic resources with MPI sessions,” in *HPCMALL 2022 - Malleability Techniques Applications in High-Performance Computing*. Cham: Springer International Publishing, Jun. 2022, pp. 147–161. [Online]. Available: https://hal.archives-ouvertes.fr/hal-03856702

[19] D. Huber, M. Streubel, I. Comprés, M. Schulz, M. Schreiber, and H. Pritchard, “Towards dynamic resource management with MPI sessions and PMIx,” in *Proceedings of the 29th European MPI Users’ Group Meeting*, ser. EuroMPI/USA’22. New York, NY, USA: ACM, 2022, p. 57–67. [Online]. Available: https://doi.org/10.1145/3555819.3555856

[20] S. Iserte, R. Mayo, E. S. Quintana-Ortí, V. Beltran, and A. J. Peña, “DMR API: Improving cluster productivity by turning applications into malleable,” *Parallel Computing*, Oct 2018.

[21] Y. Ju, D. Huber, A. Perez, P. Ulbl, S. Markidis, P. Schlatter, M. Schulz, M. Schreiber, and E. Laure, “Dynamic resource management for in-situ techniques using mpi-sessions,” in *Recent Advances in the Message Passing Interface*, C. Blaas-Schenner, C. Niethammer, and T. Haas, Eds. Cham: Springer Nature Switzerland, 2025, pp. 105–120.

[22] D. Huber. (2023) DPP Prototype. [Online]. Available: https://gitlab.inria.fr/dynres/dyn-procs

[23] ——. (2023) DynRM. [Online]. Available: https://gitlab.inria.fr/dynres/dyn-procs/dynqm/-/tree/main/dynqm

[24] S. I. Dominik Huber. DPP-DMR-API. [Online]. Available: https://gitlab.bsc.es/siserte/dmr/-/commit/2f7caa32186bc0dc7830344cd309a1a91b80e16c

[25] S. Iserte, R. Mayo, E. S. Quintana-Ortí, and A. J. Peña, “DMRlib: Easy-coding and efficient resource management for job malleability,” *IEEE Transactions on Computers*, vol. 70, no. 9, pp. 1443–1457, Sep. 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9190024

[26] M. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” *Journal of Research of the National Bureau of Standards*, vol. 49, no. 6, p. 409, Dec 1952.

[27] Y. Saad, *Iterative Methods for Sparse Linear Systems*. Society for Industrial and Applied Mathematics, Jan 2003.

[28] G. Mitalas, “Calculation of transient heat flow through walls and roofs,” *ASHRAE Trans.*, (United States), vol. 74, 1968.

[29] S. I. Dominik Huber. Implementation of dmr applications. [Online]. Available: https://gitlab.inria.fr/dynres/applications/dmr_examples

[30] J. Reina León, “Implementación distribuida maleable del método Laplace,” Jan. 2024, publisher: Universitat Oberta de Catalunya (UOC). [Online]. Available: https://openaccess.uoc.edu/handle/10609/149763

[31] Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities. SuperMUC-NG Supercomputer. [Online]. Available: https://doku.lrz.de/supermuc-ng-10745965.html

[32] T. Gamblin, M. LeGendre, M. R. Collette, G. L. Lee, A. Moody, B. R. de Supinski, and S. Futral, “The Spack package manager: Bringing order to HPC software chaos,” in *SC '15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, 2015, pp. 1–12.

[33] S. I. Dominik Huber. Demonstrator Spack Setup. [Online]. Available: https://gitlab.inria.fr/dynres/dyn-procs/dyn_prcs_setup/-/tree/hipc_spack_setup?ref_type=heads

[34] J. Corbalan, L. Alonso, J. Aneas, and L. Brochard, “Energy optimization and analysis with EAR,” in *IEEE International Conference on Cluster Computing (CLUSTER)*, 2020, pp. 464–472.

[35] D. G. Feitelson and L. Rudolph, “Toward convergence in job schedulers for parallel supercomputers,” in *Job Scheduling Strategies for Parallel Processing*, D. G. Feitelson and L. Rudolph, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1996, pp. 1–26.