<figure><img src="image_1.png" /></figure>

# MPI malleability validation under replayed real-world HPC conditions

Sergio Iserte ID a, Maël Madon ID b, Georges Da Costa ID b, Jean-Marc Pierson ID b,
Antonio J. Peña ID a,*

## ARTICLE INFO

**Keywords:**

Dynamic resource management

Replay with feedback

Malleability

Workload characterization

Cluster computing

## ABSTRACT

Dynamic Resource Management (DRM) techniques can be leveraged to maximize throughput and resource utilization in computational clusters. Although DRM has been extensively studied through analytical workloads and simulations, skepticism persists among end administrators and users regarding their feasibility under real-world conditions. To address this problem, we propose a novel methodology for validating DRM techniques, such as malleability, in realistic scenarios that reproduce actual cluster conditions of jobs and users by replaying workload logs on a High-performance Computing (HPC) infrastructure. Our methodology is capable of adapting the workload to the target cluster. We evaluate our methodology in a malleability-enabled 125-node partition of the Marenostrum 5 supercomputer. Our results validate the proposed method and assess the benefits of MPI malleability on a novel use case of a pioneer user of malleability (our “PhD Student”): parallel-efficiency-aware malleability reduced a malleable workload time by 27% without delaying the baseline workload, although introducing queueing delays for individual jobs, but maintaining the resource utilization rate.

## 1. Introduction

High-performance computing (HPC) facilities are critical for advancing scientific research, engineering, and data analysis across various domains, from genomic research [1,2] to multi-physics simulations [3–5] through artificial intelligence [6–8]. These computational clusters rely on efficiently utilizing resources to maximize their productivity. Dynamic resource management (DRM) techniques have emerged as key strategies to improve the utilization of these systems [9–11]. These techniques enable the flexible allocation and reallocation of computational resources and reconfiguring jobs accordingly to adapt to the dynamic nature of HPC workloads. Among others, malleability based on the Message-Passing Interface (MPI) is one of the most extended DRM approaches to process layout reshape and data redistribution [12–15].

Despite their potential, DRM techniques face several challenges that hinder their adoption in production environments [16]. Particularly, MPI malleability requires 1) a malleability-ready resource management system (RMS), which many existing facilities do not possess, and 2) a malleability framework to develop malleable applications compatible with the RMS and the available distributed parallel runtime systems in the cluster. These kinds of limitations increase the complexity of evaluating DRM techniques with realistic workloads and foster skepticism among end users and system administrators regarding DRM's practical

benefits and feasibility. In this regard, previous studies evaluating DRM techniques have relied on simplistic simulations, biased benchmarks, or synthetic traces, often failing to accurately represent the conditions of real-world systems. For example, some works limit their evaluations to small-scale testbeds [17–19], which struggle to provide a nice picture of the queuing dynamics or contention effects of production supercomputers. Others employ synthetic workloads or with uniform job sizes or idealized submission patterns [17,20], overlooking the variability and burstiness of user behavior. Still others focus only on single-application benchmarks or simulators [18,21–23], which, while useful to demonstrate mechanisms, do not provide insight into system-wide interactions with the RMS. These shortcomings have limited the impact of prior results, making it difficult for the community to assess the practical benefits of DRM techniques. A more detailed analysis of these studies is provided in Section 2.

To address the challenges of evaluating DRM techniques, we propose a novel methodology that leverages supercomputer logs to generate realistic workloads based on actual user submission behaviors, including temporal patterns across hours and days of the week. Our approach introduces a mechanism called the *User-Based Submitter*, designed to scale across different computing clusters. Rather than relying on synthetic simulations, this method replays real user interactions from production supercomputers, enabling a more accurate and comprehensive evaluation of DRM techniques.

<footnote>a Barcelona Supercomputing Center (BSC), Barcelona, Spain</footnote>

<footnote>b University of Toulouse, CNRS, Toulouse INP, IRIT, Toulouse, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: sergio.iserte@bsc.es, antonio.pena@bsc.es (A.J. Peña).</footnote>
We further incorporate malleable jobs into these replayed workloads to assess the impact of DRM on job execution and system performance. This allows us to observe the effects of job malleability and resource reallocation in a controlled yet realistic environment. Notably, the study includes the perspective of a pioneering user—submitting malleable jobs in a 125-node partition—demonstrating the potential of our methodology to validate previous findings and uncover new insights in real HPC settings. It is worth noting that the cluster size used in our experiments ranks among the largest evaluated in state-of-the-art malleability studies (see Section 2).

In summary, this paper presents 1) a methodology to replay logs in any computational cluster keeping the logic behind user submissions. Instead of a mathematical model, this methodology uses the techniques of user sampling from a recorded log and replay with feedback to reproduce the workload in a target system (see Section 3). This paper particularly 2) designs and analyzes a novel case study of the adoption of malleability in a production system. For this purpose, the authors exemplify this event with a new user ("the student") who is the pioneer in submitting malleable jobs to the cluster. Among all the possible scenarios, the authors have decided to develop a case that paves the way to more complex malleable workloads (see Section 4). Similarly to hardware simulators that have to be validated in actual hardware to be published, this paper presents, for the first time in the literature, 3) the validation of MPI malleability using a real workload in a malleability-enabled supercomputer, without simulations or synthetic benchmarks (see Section 5).

The paper ends with a discussion of the applicability of the presented technique and the meaningful results obtained in the evaluation of the validation methodology (see Section 6). It concludes with a summary of our contributions and findings (see Section 7).

At the end of the day, we aim to bridge the gap between simulation-based studies and practical deployment in HPC infrastructures.

## 2. Related work

This section reviews the previous efforts done in the fields of workload replay and dynamic resource management.

### 2.1. Workload replay

A classical approach to evaluate resource management policies consists of replaying a historical workload from its log. However, modifying the target infrastructure or scheduling policies impacts the system's performance (e.g., response time, computing speed, resource availability), which has a crucial impact on the submission behavior of its users [24,25]. As a result, many previous works in the field have generalization issues, since they do not account for these behaviors. A solution is to perform closed-loop simulations [26], where the submitted workload adapts to the simulated performance of the system. To do so, the authors in [27] extract relevant submission patterns from historical workloads to replay them in simulations. More recently, the authors in [28] provide an in-depth study of this technique, named "replay with feedback". In [29], the authors go one step further and suggest using re-sampling with feedback for performance evaluation where the historical workload is scaled to the target infrastructure through user sampling.

### 2.2. Dynamic resource management

The MPI paradigm is the de facto standard for implementing distributed parallel applications designed to run on computational clusters. MPI provides efficient and straightforward mechanisms for communication among different processes, known as ranks, each with its own memory address space. In a traditional MPI job, a fixed number of ranks are initiated at the start and remain active until the completion of the job. This static allocation of resources may lead to inefficiencies, especially in dynamic and heterogeneous computing environments.

MPI malleability addresses these inefficiencies through DRM. This technique allows the number of MPI ranks to be modified during the execution of a program, enabling the application to resize on the fly [30]. Malleable MPI applications may adapt to changing computational resources, improving overall system utilization and application performance. Although MPI malleability could be seen as a variant of checkpoint/restart, most modern solutions implement on-memory data redistribution, reducing overheads from accessing storage media.

Closely related to malleability is the concept of moldability. While malleability involves adjusting the number of MPI ranks during the execution of a job, moldability determines the number of ranks at the time of job submission before its initialization [31]. In this regard, jobs specify a range of sizes, allowing the job scheduler to choose the most suitable configuration based on current system availability and load. This approach provides an extra degree of flexibility by enabling better resource allocation at job scheduling time, which can lead to improved system efficiency and potentially shorter wait times for job initiation. However, unlike malleable applications, moldable applications do not adjust their resource usage dynamically during runtime.

Surveys in [32,33] extensively review the state-of-the-art of malleability in HPC systems. Instead, we focus on **how malleability frameworks and actual RMS have been evaluated with workloads** to demonstrate their usefulness of DRM in high-performance clusters.

ReSHAPE is a coupled solution for adaptive workloads, including its specific reconfiguration libraries, scheduler, and runtime system. This strong integration forces ReSHAPE users to develop applications that are compatible with this system. In [17], ReSHAPE is evaluated in a 50-node cluster with a 5-job workload composed of the benchmarks LU, MM, Master-worker, Jacobi, and FFT, submitted simultaneously. A subsequent publication added to the workload job instances of a malleable version of LAMMPS [21].

PARM (acronym of Power-Aware Resource Manager) relies on over-provisioning, power capping, and job malleability, based on CHARM++ and Slurm [18]. PARM is evaluated in a 38-node cluster with a 5-job workload comprising the benchmarks and applications Wave2D, Jacobi2D, LeanMD, Lulesh, and AMR. Larger experiments relied on the Slurm simulator with a log of 68,936 jobs submitted to a 40,960-node cluster. The authors assume that all the jobs in the workload are malleable and assign them random scalability from a set of samples.

The solution developed by Prabhakaran et al. combines AMPI (based on CHARM++) with the RMS Torque/Maui to tackle malleable jobs [19]. The authors evaluate in a 15-node cluster a workload based on a modified version of the ESP benchmark to contain various percentages of rigid, malleable, and evolving jobs composed of 230 instances. While the rigid jobs followed the benchmark directives, the flexible jobs were a malleable version of LeanMD and an evolving implementation of the synthetic Quadflow.

The Dynamic Management of Resources Library (DMRlib) (detailed in Section 3.3) was evaluated in a 129-node cluster with four benchmarks and applications (CG, Jacobi, N-Body, HPG-Aligner [23]) instantiated in 2,000-job workloads with various percentages of malleable jobs and turning on and off malleability in the applications of the workloads [34]. The authors leveraged the Feitelson synthetic workload generator [35] to evaluate job scheduling algorithms in HPC. This generator provides a way to simulate job arrival patterns, job sizes, and resource demands based on empirical data from real-world systems.

While prior research has explored the potential of DRM through benchmarks and simulations, these studies often rely on synthetic workloads with arbitrarily defined job sizes and submission patterns such as steady or bursty arrivals that fail to capture the complexity and variability of real-world HPC usage. These simplified models limit the generalizability and applicability of the findings to production systems.

In contrast, our work introduces a methodology that replays actual user behavior extracted from supercomputer logs, preserving the
nuanced temporal and structural characteristics of real workloads. This approach enables the validation of past and future synthetic setups and their results, overcoming the limitations of synthetic modeling where user behavior is too complex to be accurately captured by mathematical functions.

### 3. Methodology

In order to study DRM under real-world conditions, our method consists of reproducing the nominal activity of a cluster computing infrastructure. We use a workload submitter based on the logic of users (Section 3.1). The nominal activity is represented by submitting non-malleable jobs. These jobs, later called “baseline workload”, are reproduced from historical workload logs and sized to our testbed through user sampling (Section 3.2). To study the effects of DRM on top of the baseline workload, the submitter includes an additional user leveraging moldability and malleability in her jobs, thanks to a malleable runtime (Section 3.3).

#### 3.1. User-Based workload submitter

The core of the experiments presented in this paper relies on a piece of software running throughout the whole experiment duration: the User-Based Submitter (UBS)<fnref n="1" />. UBS reproduces HPC users issuing job submissions to the Slurm resource manager at certain timestamps. There are two types of users as described below:

* *Traditional users* submit jobs whose execution time, number of parallel resources, and submission time are given in an input file, based on a historical workload. These are used to replay the baseline workload. Consequently, the submitted jobs replicate the traditional rigid workload, allocating the requested resources during the desired time.

* *Generative users* submit malleable jobs, using feedback on the status of previous submissions to decide on the following job submissions (see [28]). More precisely, a generative user takes as input a triplet $(t_0, \Delta t, N)$ with $t_0$ the timestamp of their first submission, $\Delta t$ the think time between the end of the previous job and the submission of the new job and $N$ the total number of submissions from this user.

In the UBS, traditional and generative users are managed in two dedicated threads. The thread managing the traditional users sleeps until the next submission time, then calls a shell script issuing the job submission, passing it all relevant parameters about the job (i.e.: type, duration, or number of resources). The thread of the generative users uses a socket to be informed of the completion of previous jobs. When a job owned by a generative user terminates, a new thread is started, sleeping for $\Delta t$ seconds before calling the submission script.

#### 3.2. User sampling

To create a realistic baseline workload of the actual activity in a cluster platform, we leverage historical workload logs. The challenge is to adapt this workload to the size of our testbed *without losing the logic of submission of the original users*. Thus, instead of randomly sampling jobs from the original log, we rather perform a sampling of *users*, proceeding as described below (see Algorithm 1).

First, we choose a target level of activity for the platform, in terms of average or median load submitted per day. For example, if the testbed has 100 nodes and we target 85% activity, we want the baseline workload to have approximately 85 × 24 node-hours per day. To reach the level, we proceed iteratively by randomly adding users to a user pool. At each step, we calculate the load submitted daily by the users in the pool by summing the product of execution time and number of nodes

**Algorithm 1** User sampling from workload.

Input: $\bar{m}$ = the target median/average load per day

Input: $M$ = the number of nodes in the target platform

1: $U \leftarrow$ list of users in the workload

2: $pool \leftarrow \{\}, m \leftarrow 0$

3: **while** $m \notin [.95\bar{m}, 1.05\bar{m}]$ **do**

4: $u \leftarrow$ draw a random user from $U$, without replacement

5: **if** $max(nodes \text{ required by } u) < M$ **then**

6: $pool.add(u)$

7: $m' \leftarrow$ load per day from users in $pool$

8: **if** $m' > 1.05\bar{m}$ **then**

9: $pool.remove(u)$

10: **else**

11: $m \leftarrow m'$

12: **end if**

13: **end if**

14: **end while**

15: **if** $m \notin [.95\bar{m}, 1.05\bar{m}]$ **then**

16: print("SAMPLING FAILED!")

17: **end if**

Output: $pool$

for all jobs submitted by these users in the original log. If the activity reaches the desired level (with a margin of plus or minus 5%), the process is stopped; otherwise, we add a new user. The last added user may overshoot the target by more than 5%; in this case, that user is removed from the pool before adding another.

#### 3.3. Dynamic resource management

This research enables DRM by leveraging moldability and malleability thanks to the Dynamic Management of Resources Library (DMR-lib) [34]. DMRlib is a high-level API that facilitates the adoption of malleability in HPC codes. DMRlib implements a communication layer between the parallel distributed runtime (PDR) and the RMS, driving the management of processes and resources transparently to the user while providing flexibility to increase productivity and resource utilization of HPC facilities.

Fig. 1 depicts the operation of dynamic resource management in DMRlib, which is described as follows:

* During execution, jobs periodically expose their readiness for reconfiguration to the RMS.

* This communication occurs at a synchronization point specified in the code, where the reconfiguration process may commence. In iterative applications, the end of an iteration often serves as an ideal synchronization point.

* The RMS, with its cluster-wise information and the performance metrics provided by a monitor, determines reconfiguration actions following its defined policies, and informs the PDR.

* If this results in a change to the job size, the RMS reallocates resources and returns the new number of processes, which may involve either expanding or shrinking the job.

* Finally, the PDR redistributes the data among the processes according to the application's guidelines, allowing jobs to continue execution with the new process layout from the point where the reconfiguration was triggered.

DMRlib supports multiple Process Dynamic Runtimes (PDRs), including Nanos++ [20] and several MPI implementations [36], and can seamlessly interface with different malleability backends such as Proteo [37] and DPP [38].

<footnote n="1">¹ Code available at https://gitlab.bsc.es/siserte/ubs</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. DMRlib Application–MPI–Slurm communication.</figcaption></figure>

In this work, DMRlib has been configured with MPICH<fnref n="2" />, as the PDR, and Slurm<fnref n="3" /> as the RMS [39]. DMRlib integrates a monitoring module to track performance metrics during job execution and react accordingly by requesting/suggesting expansion or shrinking actions to the RMS. Metrics are gathered by TALP [40], which collects “POP metrics”<fnref n="4" />. This set of metrics is organized hierarchically and is multiplicative; that is, the value of the parent metric is equal to the product of the child metrics. Among others, Parallel Efficiency (PE) reveals the inefficiency in splitting computation over processes and then communicating data among processes. PE is a compound metric whose components reflect two critical factors to attain good parallel performance by ensuring even distribution of computational work across processes (load balance) and minimizing time communicating data among processes (communication efficiency).

# 4. Experimental setup

We implement the method described above to evaluate the impact on the job submissions of a user enabling DRM in a computing cluster. For the purpose of this narrative, we portray this user as a PhD student working under a tight deadline, submitting malleable instances of a scientific application. This section describes the testbed, workload, and specific parameters of the experimental campaign.

## 4.1. Cluster computing infrastructure

The evaluation is performed in the general-purpose partition of the Marenostrum 5 (MN5) supercomputer, a pre-exascale machine integrated into the EuroHPC-JU European supercomputing infrastructure<fnref n="5" />. The nodes of this partition are based on Intel Sapphire Rapids (4th Generation Intel Xeon Scalable Processors), each equipped with two Intel Xeon Platinum 8480 + with 56 cores running at 2 GHz of base frequency, for a total of 112 cores and 256 GB of DDR5 memory. The nodes are interconnected through a 100 Gbit/s ConnectX-7 NDR200 InfiniBand network.

Regarding the software stack, the executions rely on the GCC-10.4 compiler, and the MPICH-3.2 MPI implementation. We are aware of the current incompatibility between UCX and MPI_Comm_spawn in MPICH.

For this reason, we opted to use OFI as the transport layer protocol instead. At the same time, a new version of DMR (DMRV2) is under development, incorporating support for Open MPI and UCX. Preliminary results of this work have been published in [36].

The RMS is a customized version of Slurm to support malleability included in the dynamic resource management framework DMR-lib. The malleability-enabled Slurm provided in DMRlib is based on Slurm-17.02.0-0pre1 and extends its functionality through a resource selection plugin that supports malleability. The plugin builds upon the select/linear resource selection policy and remains fully compatible with Slurm's native support for moldability. An extensive evaluation of this plugin, considering workloads with varying proportions of moldable and malleable jobs, is presented in [34]. That work concludes that moldability is not enough to reap the benefits of dynamic resource management, and malleability is required to achieve significant improvements in resource utilization and job throughput.

In the current work, we have extended that plugin with performance metrics from TALP to drive malleability decisions as explained in Section 4.4. For this reason, every time a malleable job triggers a reconfiguration, Slurm will determine the action that the runtime must perform, which may be expand, shrink, or none. Notice that it is out of the scope of this work to evaluate scheduling or priority policies, thus, Slurm is configured with the default parameters of the sched/backfill and priority/multifactor policies. We have also enabled the Slurm job epilog mechanism to communicate job completions to the job submitter UBS described in Section 3.1.

The experiments are scoped within 125 nodes for up to 48 hours, which are the limits of the default QoS of the BSC researchers in MN5. Slurm is deployed using a node running its controller daemon, while the remaining 124 nodes act as actual compute nodes.

## 4.2. Traditional users: the baseline workload

The baseline workload is adapted from the most recent available workload on the Parallel Workload Archive KIT-FH2-2016⁶ log, recorded from the ForHLR II system located at the Karlsruhe Institute of Technology in Germany. Notice that this is the most recent workload publicly available and it is still a representative example of a general-purpose computational partition, unaffected by interference from deep

<footnote n="2">² https://www.mpich.org</footnote>

<footnote n="3">³ https://slurm.schedmd.com</footnote>

<footnote n="4">⁴ https://pop-coe.eu/node/69</footnote>

<footnote n="5">⁵ https://www.bsc.es/supportkc/docs/MareNostrum5/overview</footnote>

<footnote n="6">⁶ https://www.cs.huji.ac.il/labs/parallel/workload/1_kit_fh2/index.html</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Distribution of job submissions and platform capacity over the days of July 2017 (x-axis). Each color represents a different user. The horizontal line in the bottom graph is the maximum capacity of the platform (M = 124 × 24 = 2,976 node-hours per day).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Distribution of job execution time in the baseline workload.</figcaption></figure>

learning jobs, better suited for accelerated partitions. To adapt the log to our testbed, we perform the following steps:

1. The original infrastructure has two queues: the default queue with 1152 20-core compute nodes and the visualization queue with 21 nodes comprising CPUs and GPUs. We focus on the first queue and exclude the GPU-enabled.

2. The record is 1.5 years long, and we only have two days of computation available in our testbed. Consequently, we decide to speed up the replay by applying a time-scaling factor of 10, i.e., execution times and inter-arrival times are divided by 10 in the replay.<fnref n="7" /> In this regard, we can replay 2 × 10 = 20 days from the original workload.

3. By studying the distribution of job submissions over time, we notice that the month of July 2017 features a homogeneous volume of submissions over days, well-distributed among users, and the presence of the characteristic day/night and weekday/weekend patterns.

Moreover, the load submitted during this month (611k node-hours) is representative of the other months (min = 199k, mean = 510k, med = 576k, max = 709k).

As a result, we select for the experiments the **19-day period from July 3rd to July 21st** (included), 2017.

4. Finally, we **apply the user sampling method** described in Section 3.2, with a target in average node-hour per day of .84M, where M = 124 × 24 is the maximum number of node-hour available per day in our testbed.

The resulting workload contains 23 users and 1895 jobs. The median number of node-hours submitted per day is 2,272, i.e., 76% of the maximum capacity of the testbed. The average number of node-hours submitted per day is 2,496, i.e., 84% of the maximum capacity of the testbed. Some characteristics of this workload are provided in the form of graphs. Fig. 2 displays the number of daily submissions and node-hours submitted per day. Each color represents a particular user, and the stacked graphs show the submission volume per day. Fig. 3 shows the cumulative proportion of jobs as a function of their duration. It shows that a vast majority lasts less than one hour (slightly more than 60%). Fig. 4 shows the distribution of job execution times and job sizes. Jobs are mainly requesting 1, 3, or 4 nodes (see Fig. 4a). But it also shows

<footnote n="7">⁷ This factor was chosen to balance the tradeoff between replaying more days from the original workload and suffering from overheads of phenomena that we cannot speed up (like the time taken by scheduling decisions).</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Job sizes in the baseline workload. The distribution is shown by the number of jobs (top) and number of node-hours (bottom), and the cumulative distribution is represented in grey.</figcaption></figure>

that large jobs (see 26, 30, and 32 nodes in the x-axis of Fig. 4b), while rare, represent a significant part of the total executed mass. Since the jobs in the log have a fixed duration, the replayed jobs submitted to the cluster will allocate resources for the same period, either by using sleep commands or implementing active waits.

**Warm-up Period.** Looking at Fig. 2b, we note that the first day of the workload, July 3rd, features high activity in node-hours submitted. This day, Monday, comes after a weekend of low activity on the platform. However, it is not until 11:00 AM that the platform runs in nominal conditions. In other words, at that time, resources are virtually fully allocated, and there are pending jobs waiting for resources. We thus define this period as a “warm-up” in the experiments, and malleable jobs start being submitted just after it ($t_0$ input of the *generative user* described in Section 3.1).

### 4.3. A *generative user: the PhD student*

To thoroughly evaluate our methodology, we count on an additional generative user to submit jobs on top of the baseline workload. “The student” is working with a simulation tool, particularly, a positive definite transport equation solver named MPDATA<fnref n="8" />. Its algorithm performs iterative time steps to simulate physical phenomena, requiring five input arrays and producing a single output array essential for subsequent time steps. The malleable version of MPDATA<fnref n="9" /> redistributes the input arrays between source and target processes, respectively, the number of processes before and after a reconfiguration. MPDATA malleable demonstrated to increase the utilization of resources and reduce the power consumption in dynamic workloads [22].

For her study, the student must run ten instances ($N$) of MPDATA sequentially. She will start launching jobs after the warm-up period and wait two scaled hours (720 actual seconds) before submitting the next job. This think time ($\Delta t$) corresponds to a hypothetical analysis of the results and the preparation of the subsequent execution.

The student is requested to complete her workload in 14 days; in other words, she has a deadline of two weeks to present her results. Since she is in a hurry, an alarm is triggered on her phone when a job finishes, and she immediately starts processing data and setting up the next job.

MPDATA is configured with a computational domain of 8,192 × 1,024 × 128 cells, iterating during 1800 steps. This computational domain enables the user to run MPDATA from 1 to 64 nodes. Fig. 5 shows the scalability of MPDATA with the given configuration in MN5. The application scales up to 16 nodes linearly, from where it shows speedups lower than 2x.

### 4.4. *Experiments definition*

Five experiments have been designed to study and analyze the effect of job malleability and dynamic management of resources over an existing workload. Each experiment was launched on an independent 48-hour reservation of the 125-node testbed (see Section 4.1).

With these experiments, we will illustrate scenarios that represent different user behaviors:

1. **Baseline**: this experiment corresponds to executing the baseline workload described in Section 4.2. This experiment sets the original activity of the workload execution in the cluster.

2. **StaticN32**: this experiment executes sequentially over the baseline workload, the 10-job workload of the PhD student launched with 32 nodes each. This is a realistic case, since with 32 nodes the execution time is ≈ 14 scaled hours (10x the actual time in Fig. 5). It means that she could finish her executions in less than six scaled days in an ideal scenario where jobs were not delayed.

3. **StaticN16**: this is another non-malleable experiment similar to Static32 but launching the jobs with 16 nodes, instead. In this case, the student works with the shortest configuration time, the one within the 2x speedup in Fig. 5, which lasts ≈ 18.5 scaled hours. Theoretically, she could complete her executions in less than eight scaled days. We have discarded to configure a hypothetical Static8 experiment, since the theoretical time required to run the 10-job workload with eight nodes under ideal conditions is 47.5 scaled hours, while the maximum wall time granted for the experiment is 48 hours.

<footnote n="8">⁸ The MPDATA algorithm serves as the foundation of the EULAG multiscale fluid solver (Eulerian/semi-Lagrangian) [41], which is responsible for calculating the advection of a non-diffusive quantity in a flow field.</footnote>

<footnote n="9">⁹ Code available at https://gitlab.bsc.es/siserte/mpdata-dmr.</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 5. MPDATA scalability in MN5.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6. MPDATA reconfiguration times in MN5.</figcaption></figure>

4. **AlwaysGrow**: in this experiment, the PhD student instantiates malleable jobs over the baseline workload. Jobs are submitted moldable, requesting a 1–64 range of nodes. Since she is under a tight deadline, she decides to disable shrinkages; jobs may only be expanded up to 64 nodes upon the RMS decision. By default, the reconfiguration policy will plan an expansion when sufficient resources are available to upscale. Furthermore, the student has defined a reconfiguration inhibitor of `#current_nodes` iterations to avoid abuse of reconfiguration operations and reduce overhead generated by them (see Fig. 6).

5. **ParEfficiency**: in this case, jobs are submitted within the 2–64 nodes range. Malleability limits are defined as one node at the minimum and 64 at the maximum. However, the student configures an additional inhibitor for reconfigurations longer than 50 scaled minutes. According to Fig. 6, which depicts the measured reconfiguration time for all the possible node combinations of *from-to* reconfigurations, those greater than five minutes are avoided, specifically the reconfigurations 1 to 2, 2 to 1, and 4 to 1. This experiment leverages the performance-aware support of the dynamic resource manager (see Section 3.3). Algorithm 2 depicts the reconfiguration policy. Every time a malleable job triggers a reconfiguration, Slurm will determine the action that the DMRlib runtime must perform. In this regard, Slurm's plugin will check if a pending job may be initiated with some of the resources that would be available if the malleable job relinquishes them after its shrinkage (line 3). If a target job may be initiated, the policy checks the current parallel efficiency of the malleable job. If the parallel efficiency does not reach a minimum threshold (line 4), the target job priority in the queue will be increased (line 5), and the malleable job will be shrunk (line 6). If no job may be initiated and there are available resources in the clus-

ter (line 10), the policy checks the current parallel efficiency. If the value exceeds a determined threshold (line 11), indicating that the execution may still leverage additional resources, the malleable job will be expanded (line 12).

This policy aims to obtain as many resources as possible if no other job in the queue may use them. However, suppose any job in the queue may be initiated, and the malleable job is not reasonably using the resources (within a parallel efficiency threshold). In that case, the RMS reassigns the resources to a pending job in order to increase global cluster productivity and efficiency.

<caption>**Algorithm 2** Reconfiguration Policy Algorithm in Slurm.</caption>

1: result ← NONE
2: η ← EVALUATEmyparallelEfficiency()
3: **if** JOBCANBEINITIATEDWITHPARTOFMYRESOURCES() **then**
4:   **if** η < 0.85 **then**
5:     SETMAXIMUMPRIORITYTO TARGETPENDINGJOB()
6:     result ← SHRINK
7:   **end if**
8: **else**
9:   **if** R ≠ ∅ **then**
11:  **if** η > 0.10 **then**
12:   result ← EXPAND
13:  **end if**
14: **end if**
15: **end if**
16: **return** result
<caption>**Table 1**</caption>

Workloads makespan and average and standard deviation of allocated resources and waiting time for the five experiments, excluding the warm-up period.

<table><thead><tr><th></th><th></th><th>Baseline</th><th>StaticN32</th><th>StaticN16</th><th>AlwaysGrow</th><th>ParEfficiency</th></tr></thead><tbody><tr><td rowspan="2">Workload Maskespan</td><td>Complete</td><td>44.84 h.</td><td>45.15 h.</td><td>44.84 h.</td><td>44.84 h.</td><td>44.84 h.</td></tr><tr><td>Only PhD</td><td>-</td><td>45.15 h.</td><td>38.50 h.</td><td>36.62 h.</td><td>32.88 h.</td></tr><tr><td rowspan="2">Global Resource Allocation</td><td>Avg</td><td>83.70%</td><td>93.86%</td><td>91.50%</td><td>92.63%</td><td>91.86%</td></tr><tr><td>Std</td><td>24.07%</td><td>11.44%</td><td>17.62%</td><td>17.24%</td><td>17.16%</td></tr><tr><td rowspan="2">Actual Jobs Waiting Time</td><td>Avg</td><td>1,725.04 s.</td><td>3,007.24 s.</td><td>3,290.38 s.</td><td>4,365.89 s.</td><td>4,242.71 s.</td></tr><tr><td>Std</td><td>4,450.38 s.</td><td>6,197.19 s.</td><td>7,677.89 s.</td><td>10,013.08 s.</td><td>11,690.31 s.</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** Resource allocation (Y-axis) in each second of the execution (X-axis) for *Baseline* experiment.</figcaption></figure>

Notice that preliminary tests varying the PE upper threshold by ±(0.05–0.1) and lower threshold by ±0.05 showed no significant deviation in the key outcomes: the student job’s makespan remained consistently reduced, and baseline waiting-time impact was negligible, suggesting that the conclusions are robust within a reasonable range of parameter variation.

Furthermore, we completely rely on Slurm’s default scheduling mechanisms, namely the sched/backfill and priority/multifactor policies. When the priority of a job is increased (as in Algorithm 2, line 5), Slurm recomputes the priorities of all queued jobs, ensuring fairness according to its standard policies. This adjustment may affect the ordering of jobs across the entire queue, not just within feasible backfill windows. However, a detailed design of scheduling and priority policies is beyond the scope of this work, as our focus is specifically on the resource selection policy *select/linear* that enables malleability.

# 5. Results

In this section, the results obtained after running the experiments are analyzed. We use the following metrics for the validation of MPI malleability: makespan (Section 5.1), resource allocation rate (Section 5.2), and waiting time (Section 5.3)<fnref n="10" />.

Note that the metrics have different interpretations depending on the type of workload replay, as well-explained by Feitelson [29]. In the case of replay with feedback (like here with the PhD student workload), the primary performance metric is the makespan, i.e., the total time to execute the workload. On the contrary, the makespan in timestamp-based replay (like here with the baseline workload) is dictated by the original submission timestamps. In this case, the primary performance metric is the waiting times for each job.

## 5.1. Makespan

The completion time of the workload, or workload makespan, is a critical metric for the PhD student, who has a deadline to meet. Particularly, she needs to focus on the time it takes to complete her 10-job workload. Table 1 compiles execution metrics of the different experiments. The first two rows of Table 1 (after the header) report the execution times for the two workloads. The first row corresponds to the complete workload (Baseline + PhD student), which also defines the overall experiment duration, while the second row refers exclusively to the PhD student workload.

As discussed in the introduction of this section, the first row is not meaningful, since makespan is dictated by the rigid submission timestamps replayed from the original log. This is why all the values in this row are equal to the *Baseline* makespan. The only exception is *StaticN32*, whose makespan is ruled by the PhD job, as shown in Fig. 8a. The difficulty of scheduling large jobs in a congested system is manifested in this experiment that reports a completion time increased compared to *Baseline*.

In the remaining experiments (*StaticN16*, *AlwaysGrow*, and *ParEfficiency*), the PhD student jobs are completed before the end of the baseline workflow. Of course, reducing the PhD student jobs size from 32 to 16 nodes helps the backfilling scheduler to find gaps to fit the jobs, thus reducing the PhD student workload completion time from 45.15 to 38.50 actual hours. However, it is not enough to meet the student’s deadline of 14 scaled days. Fig. 8b shows that the last two jobs of the student are still running after the deadline. *AlwaysGrow*, leverages malleability to reduce the student makespan, but it still needs 36.62 actual hours, which is still beyond the deadline (see Fig. 9a). Notably, the “smartest” dynamic configuration, *ParEfficiency*, needs ≈ 73% the time to complete the student jobs compared to *StaticN32* (32.88 h. vs 45.15 h., respectively) without delaying the baseline workload and meeting the deadline, since both full workloads are completed in 44.84 h. To sum up, Fig. 9b shows that the last job of the student finishes before the deadline in *ParEfficiency* scenario.

<footnote n="10">¹⁰ Since the execution time is fixed and determined by the log, and the completion time is the sum of waiting plus execution times, the completion time is entirely dependent on the waiting time.</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 8. Resource allocation (Y-axis) over execution time (X-axis) for the static experiments of the student.</figcaption></figure>

In the following subsections, we will see how malleability affects the resource allocation rate and the waiting time, which are the two factors that explain the makespan reduction of the PhD student workload.

## 5.2. Resource allocation rate

Fig. 7 showcases the total number of allocated nodes for every actual second of the *Baseline* experiment. Correspondingly, Figs. 8a, 8b, 9a, and 9b showcase stacked bar plots where each timestamp depicts the sum of the allocated resources by the baseline plus the PhD student workloads. The figures present different patterns: while the static experiments in Figs. 8a and 8b show how the student jobs have to wait for slots that satisfy the static requests, the malleable experiments in Figs. 9a and 9b leverage fragmentation where student jobs run.

Table 1 contains in the 3^rd^ row the allocation rate for each experiment, which is the average node utilization over the makespan of the experiment, excluding the warm-up period. In turn, the 4^th^ row presents the standard deviation of the allocation rate.

*Baseline* provides the lower average allocation rate with the highest standard deviation, which is expected since the platform presents many idle regions of resources during the execution (see Fig. 7). Evidently, the remaining experiments present higher allocation rates because of the 10 extra jobs of the PhD student. While *StaticN32* presents the highest average allocation rate, it also shows the lowest standard deviation, which means that the platform is uniformly more saturated during the experiment (see Fig. 8). The remaining experiments present similar average allocation rates, with higher standard deviations than *StaticN32*, which means that the platform presents more idle regions during the execution (see Figs. 8a, 9b, and 9b), particularly at the end of the workload execution, when the jobs of the baseline workload are still running, but the PhD student jobs have already finished. In this regard, the rigid jobs of *StaticN16* and the malleable jobs of *AlwaysGrow* and *ParEfficiency*

make no difference in the allocation rate. That is why we need to understand better the makespan reduction of the PhD student workload with studying the waiting time in the next section, where we see how MPI malleability can maintain resource utilization while reducing the completion times.

## 5.3. Waiting time

To analyze how baseline jobs are affected by the submission of jobs from the PhD student, we look at their average waiting times and standard deviation in the different scenarios (see Table 1, 5^th^ and 6^th^ rows).

It is patent that the original workload suffers longer delays on average when the PhD student jobs are submitted: the average waiting time increases from 1,725.04 s. to 3,007.24 s. or 4,365.89 s. in the best and worst cases, respectively. This increment of up to ≈ 2.53x is expected since the system is more saturated with the extra jobs.

Besides, the two dynamic experiments (*AlwaysGrow* and *ParEfficiency*) pose a larger impact on the baseline waiting time, in terms of average and standard deviation. This can be explained by the greedy nature of the malleable configurations, starting with fewer resources and expanding as resources are released in the platform, making the baseline jobs wait longer.

In a finer detailed analysis, we studied the accumulated waiting time throughout the execution (Fig. 10). It shows how the waiting time increases as new jobs arrive in the queue, and since there are not enough available resources until around timestamp 77,000 (July 10^th^), they cannot be quickly started. After this milestone, the waiting time progressively starts growing again. More specifically, Fig. 11 showcases the waiting time difference of the various experiments compared to the *Baseline* for each job. One of the most valuable insights we extract is the behavior detected around job 750 (July 10^th^), where waiting times
<figure><img src="image_10.png" /><figcaption>Fig. 9. Resource allocation (Y-axis) over execution time (X-axis) for the dynamic experiments of the student.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Accumulated waiting time throughout the workloads executions.</figcaption></figure>

drop. This is because there is a period around July 8^th^ and 9^th^, as shown in Fig. 2a, where the submissions drastically decrease, and it takes over a day to drain the queue. That is why jobs submitted after that event present a virtually null waiting time.

The gap in think time observed around timestamp 80,000 in Fig. 9 corresponds to Job 7 (see Table 3), which experienced a delay of nearly four hours before initiation. This behavior is not an artifact of our methodology but rather the result of the scheduling dynamics in Slurm. Specifically, the delay was caused by the higher priority of other pending jobs in the queue. While backfilling can allow jobs to advance when resources are available, Slurm's fairness policy also incorporates accumulated waiting time into its priority calculation. Consequently, student's Job 7 remained pending until its priority increased sufficiently to be scheduled.

5.4. *Focus on the PhD student*

Fig. 12 illustrates the average completion time of the student's jobs, defined as the sum of waiting and execution times. Dynamic experiments exhibit longer execution times than the static ones. This is because malleable jobs are submitted with a range of possible node counts, and the RMS can more easily allocate resources within that range at submission time, typically closer to the lower bound. By leveraging backfilling, these smaller jobs start earlier, so the *Student's* jobs experience almost no waiting time when malleability is enabled.

Once started, malleable jobs are progressively expanded according to their respective reconfiguration policies. Assigning more resources reduces execution time (see Fig. 5), but each reconfiguration incurs an overhead (see Fig. 6), which is accounted for in the execution time.
<figure><img src="image_12.png" /><figcaption>Fig. 11. Difference in job waiting times compared to the baseline experiment.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 12. Average student's job completion (waiting + execution) time (Y-axis) for the four experiments (X-axis).</figcaption></figure>

<caption>**Table 2**</caption>

Per experiment: number of reconfigurations (expansions / shrinkages), average reconfiguration time and accumulated overhead (actual seconds).

<table><thead><tr><th rowspan="2"></th><th colspan="3">Expansions</th><th colspan="3">Shrinkages</th><th colspan="3">Total</th></tr><tr><th>#</th><th>Avg.</th><th>Overhead</th><th>#</th><th>Avg.</th><th>Overhead</th><th>#</th><th>Avg.</th><th>Overhead</th></tr></thead><tbody><tr><td>AlwaysGrow</td><td>27</td><td>160 s.</td><td>4,332 s.</td><td>0</td><td>-</td><td>0 s.</td><td>27</td><td>160 s.</td><td>4,332 s.</td></tr><tr><td>ParEfficiency</td><td>18</td><td>87.8 s.</td><td>1,580 s.</td><td>7</td><td>48.9 s.</td><td>349 s.</td><td>25</td><td>76.9 s.</td><td>1,929 s.</td></tr></tbody></table>

The total cost of reconfigurations is summarized in Table 2.

The AlwaysGrow policy was designed to aggressively exploit every opportunity for expansion, regardless of the immediate overhead, with the goal of reaching the job's maximum resource allocation as quickly as possible. Although some of these reconfigurations are slow and yield limited benefits, they highlight the trade-offs of prioritizing rapid growth over efficiency.

In contrast, the ParEfficiency policy applies a more selective approach, avoiding the most expensive reconfigurations and thereby achieving better overall efficiency. The juxtaposition of these two strategies is deliberate: AlwaysGrow illustrates the risks of unfiltered expansion, whereas ParEfficiency demonstrates how threshold-based inhibition can achieve more balanced outcomes. Together, they underscore the need for adaptive policies and justify the design of tunable malleability mechanisms in HPC environments. For completeness, the detailed reconfiguration events of the ParEfficiency experiment are listed in Table 3.

A key advantage of ParEfficiency is its ability to reduce overhead by preventing the slowest reconfigurations. Moreover, this policy adapts malleable jobs to workload and performance dynamics, triggering seven 32-to-16 shrinkages. These shrinkages improve cluster productivity by reducing the PhD student workload time by 10% compared to AlwaysGrow.

The impact of these policies is also reflected in node-hour consumption. Fig. 13 reports the node-hours consumed by PhD student jobs

across all experiments, with the dashed line showing the mean per configuration. Static jobs display a virtually constant node-hour consumption (standard deviation < 1). In an ideal linear-scaling scenario (i.e., a 2x speedup when doubling the number of nodes), the node-hour consumption would be identical in StaticN32 and StaticN16. However, the application exhibits sublinear scalability (see Fig. 5), achieving only a 1.42x speedup when increasing from 16 to 32 nodes. As a result, the StaticN32 configuration consumes 17 additional node-hours compared to StaticN16.

Malleable jobs, by design, show variable node-hour consumption within the experiments (standard deviation > 4). Interestingly, both AlwaysGrow and ParEfficiency contain the two “cheapest” PhD student jobs, which spend most-or even all-of their execution on 8 nodes. This is significant, as eight nodes provide the best efficiency point for this application, leading to the lowest node-hour cost.

## 6. Discussion

Our methodology adapts historical workload traces to a target infrastructure, accounting for the fact that reproducing logs on the same system is often not feasible. The goal is to provide a realistic path for gradual DRM adoption in HPC centers.

We do not expect full adoption of DRM techniques to happen immediately. Instead, early adopters (technically skilled users motivated to exploit malleability) would be the first to submit DRM-enabled jobs to
<figure><img src="image_14.png" /><figcaption>Fig. 13. Individual student job's Node-hours (Y-axis) grouped by colors for the four experiments (X-axis). Dashed lines represent the average per experiment (Y-axis).</figcaption></figure>

<table><caption>**Table 3**
Reconfigurations in the *ParEfficiency* experiment (time in actual seconds). Gray rows represent shrinkages.</caption><thead><tr><th>PhD Job Id</th><th>Initial Nodes</th><th>Final Nodes</th><th>Rec. Time</th></tr></thead><tbody><tr><td>0</td><td>16</td><td>-</td><td>-</td></tr><tr><td rowspan="3">1</td><td>2</td><td>4</td><td>254 s.</td></tr><tr><td>4</td><td>8</td><td>136 s.</td></tr><tr><td>8</td><td>16</td><td>72 s.</td></tr><tr><td rowspan="4">2</td><td>8</td><td>16</td><td>69 s.</td></tr><tr><td>16</td><td>32</td><td>43 s.</td></tr><tr><td>32</td><td>16</td><td>47 s.</td></tr><tr><td>16</td><td>32</td><td>34 s.</td></tr><tr><td>3</td><td>8</td><td>-</td><td>-</td></tr><tr><td rowspan="4">4</td><td>4</td><td>8</td><td>133 s.</td></tr><tr><td>8</td><td>16</td><td>69 s.</td></tr><tr><td>16</td><td>32</td><td>35 s.</td></tr><tr><td>32</td><td>16</td><td>49 s.</td></tr><tr><td>5</td><td>8</td><td>16</td><td>68 s.</td></tr><tr><td>6</td><td>8</td><td>16</td><td>66 s.</td></tr><tr><td>7</td><td>32</td><td>16</td><td>47 s.</td></tr><tr><td rowspan="4">8</td><td>2</td><td>4</td><td>253 s.</td></tr><tr><td>4</td><td>8</td><td>135 s.</td></tr><tr><td>8</td><td>16</td><td>67 s.</td></tr><tr><td></td><td></td><td></td><td></td></tr><tr><td rowspan="9">9</td><td>16</td><td>32</td><td>38 s.</td></tr><tr><td>32</td><td>16</td><td>48 s.</td></tr><tr><td>16</td><td>32</td><td>36 s.</td></tr><tr><td>32</td><td>16</td><td>51 s.</td></tr><tr><td>16</td><td>32</td><td>38 s.</td></tr><tr><td>32</td><td>16</td><td>51 s.</td></tr><tr><td>16</td><td>32</td><td>34 s.</td></tr><tr><td>32</td><td>16</td><td>50 s.</td></tr></tbody></table>

production environments still dominated by fixed-size workloads. This hybrid workload composition enables us to assess the incremental benefits of malleability without needing full-scale user migration.

This setup also offers valuable insight into how malleable jobs can coexist with standard jobs and make use of fragmented resources that would otherwise go idle. Running our experiments on a real Slurm deployment allowed us to combine malleability with Slurm's native backfilling policy, improving resource utilization.

While the number of test scenarios is limited by the need for live system execution, our methodology produces high-fidelity results. Moreover, it can be reused by organizations with their own workload traces, enabling site-specific evaluation of malleability's practical benefits and deployment challenges.

We argue that our method is more robust than many alternatives commonly used in the literature (see Section 2). We (i) use a real work-

load trace rather than synthetic jobs, and (ii) right-size the workload by randomly sampling users, which preserves authentic submission and temporal patterns.

The interpretation of our results highlights several important trade-offs. At the end of the day, the PhD student successfully meets the deadline in the *ParEfficiency* scenario. Counterintuitively, large fixed resource requests, which users might assume to be beneficial, actually work against minimizing completion time. Additionally, overly aggressive malleable strategies can introduce excessive reconfiguration overhead, limiting their ability to efficiently leverage resource fragmentation.

Fragmented resource availability, a common characteristic of HPC infrastructures, represents the ideal use case where malleability becomes crucial. This fragmentation is what our student leverages. Table 4 focuses on the time period during which the student is actively running jobs (corresponding to the red areas in Figs. 8a, 8b, 9a, and 9b). The table presents the resource consumption in *nodes/day* for the workloads, along with their proportion of the total available area (124 nodes × time) shown in the first row.

The data highlights an interesting trade-off. Although the most cost-effective strategy for the student is *StaticN16* (third row), it does not only allow meeting the deadline but also results in increased costs for the baseline workload (second row) and higher overall cluster resource consumption during that period (fourth row), compared to the optimal *ParEfficiency*.

Given these findings, users who contribute to higher infrastructure efficiency by running *parallel-efficiency-aware malleable jobs*—which increase resource utilization without extending job execution times—could be incentivized. One possible approach would be to award a reduction in their usage quota proportional to their efficiency gains, encouraging broader adoption of malleability in HPC environments.

Despite our best efforts, we recognize several potential threats to the validity and generalizability of our results. First, baseline jobs were replayed using active waits rather than real computations, which could in principle overlook issues of resource interference. This would be problematic in environments where resources are shared among concurrent jobs. However, in HPC systems such as MareNostrum 5—the target of our work—nodes within large partitions are allocated exclusively to individual jobs, ensuring isolation and preventing interference, thereby safeguarding the integrity of our setup.

Second, the original trace was compressed by a factor of 10 to replay more days within our 48-hour reservations. While this reduces realism by amplifying phenomena that cannot be sped up, such as scheduler decision times, both execution and inter-arrival times were consistently scaled. As a result, queue dynamics and workload arrival patterns remain preserved, and all experiments remain directly comparable.

Third, our evaluation considered only one dynamic job at a time, thus not capturing competition between multiple dynamic jobs. This
<table><caption>**Table 4**
Resource consumption (nodes/day) and its total percentage during the student's activity for the different experiments.</caption><thead><tr><th></th><th>StaticN32</th><th>StaticN16</th><th>AlwaysGrow</th><th>ParEfficiency</th></tr></thead><tbody><tr><th>Total</th><td>232.83 n/day (100%)</td><td>200.54 n/day (100%)</td><td>189.09 n/day (100%)</td><td>171.52 n/day (100%)</td></tr><tr><th>Baseline</th><td>193.5 n/day (83.11%)</td><td>173.7 n/day (86.62%)</td><td>164.2 n/day (86.84%)</td><td>145.68 n/day (84.93%)</td></tr><tr><th>Student</th><td>23.95 n/day (10.29%)</td><td>17.4 n/day (8.68%)</td><td>20.01 n/day (10.58%)</td><td>18.25 n/day (10.64%)</td></tr><tr><th>Accumulated</th><td>217.45 n/day (93.39%)</td><td>191.1 n/day (95.29%)</td><td>184.21 n/day (97.42%)</td><td>163.93 n/day (95.57%)</td></tr></tbody></table>

was an intentional choice to isolate and analyze malleability in controlled conditions. Still, our preliminary work [34] has shown that scenarios with multiple dynamic jobs are feasible and have been explored in earlier studies.

Fourth, we evaluated only a single application, MPDATA. This code was selected because of its step-based structure and near-linear scalability, traits that are representative of many HPC workloads. While our empirical results are application-specific, MPDATA serves as a meaningful abstraction for a broad class of malleable applications. Furthermore, previous work [34] studied additional applications with diverse communication/computation balances, supporting the generality of our methodology.

Finally, experiments were performed using a single input dataset—the July 2017 segment of the KIT-FH2-2016 workload—and a fixed random seed for user sampling, without replication. In our experience, and consistent with practices commonly found in the state of the art, execution times at this scale tend to be stable enough that single runs provide representative results, unlike microbenchmarks (e.g., bandwidth or latency) where variability can significantly affect outcomes. Nevertheless, replicating the study with alternative inputs or complementing it with simulations would strengthen confidence in the conclusions while avoiding the prohibitive costs of large-scale experimental campaigns. Nevertheless, we agree that including confidence intervals and statistical analyses would strengthen future work, particularly for smaller-scale or reduced-size workloads where multiple repetitions are feasible.

## 7. Conclusion

In this work, we have introduced a novel methodology to validate dynamic resource management techniques by adapting supercomputer logs to DRM-enabled computing clusters. Unlike previous studies that relied on benchmarks or simulations with synthetic workloads, we employ a novel methodology called User-Based Submitter that combines user sampling and workload replay with feedback. By leveraging real-world data and testing on an MPI malleability-enabled supercomputer's partition, we provide a realistic and actionable framework for evaluating new resource management and job scheduling policies.

We demonstrated the effectiveness of our approach through a novel use case, where an HPC user successfully leveraged MPI malleability to complete her workload on time. Validation in a supercomputing environment showed a 27% reduction in malleable workload time maintaining the resource utilization rate. While individual jobs may experience delays, the baseline workload remains unaffected in terms of execution time. We believe this methodology is broadly applicable and represents a promising step toward integrating dynamic resource management into next-generation HPC infrastructures.

## Declaration of interests

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Sergio Iserte and Antonio J. Peña are involved in the project The European PILOT, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreements No. 101034126 and No. PCI2021-122090-2A under the MCIN/AEI and the EU NextGenerationEU/PRTR. They are also grateful for the support from the Department of Research and Universities of

the Government of Catalonia to the AccMem (Code: 2021 SGR 00807). Antonio J. Peña was partially supported by the Ramón y Cajal fellowship RYC2020-030054-I funded by MCIN/AEI/ 10.13039/501100011033 and by “ESF Investing in your future”. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work the authors used ChatGPT in order to improve language and readability. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

## CRediT authorship contribution statement

Sergio Iserte: Writing – original draft, Visualization, Software, Project administration, Methodology, Investigation, Conceptualization; Maël Madon: Writing – original draft, Visualization, Software, Methodology, Investigation; Georges Da Costa: Writing – review & editing, Supervision, Investigation; Jean-Marc Pierson: Writing – review & editing, Supervision, Investigation; Antonio J. Peña: Writing – review & editing, Supervision, Resources, Funding acquisition.

## Declaration of competing interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

## Funding sources

The researchers from BSC are involved in the project The European PILOT, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreements No. 101034126 and No. PCI2021-122090-2A under the MCIN/AEI and the EU NextGenerationEU/PRTR. Antonio J. Peña was partially supported by the Ramón y Cajal fellowship RYC2020-030054-I funded by MCIN/AEI/ 10.13039/501100011033 and by “ESF Investing in your future”.

## Data availability

No data was used for the research described in the article.

The artifact source-code can be cloned from GitHub: https://github.com/siserte/dmr-poc.

## References

[1] H. Martínez, J. Tárraga, I. Medina, S. Barrachina, M. Castillo, J. Dopazo, E.S. Quintana-Ortíz, A dynamic pipeline for RNA sequencing on multicore processors, in: Proceedings of the 20th European MPI Users' Group Meeting, EuroMPI '13, Association for Computing Machinery, New York, NY, USA, 2013, pp. 235–240. https://doi.org/10.1145/2488551.2488581

[2] H. Zhong, X. Pan, Z. He, H. Wang, D. Huang, Z. Chen, GPU Acceleration for DNA sequence alignment algorithm and its application, CCF Trans. High Perform. Comput. 7 (2) (2025) 169–177. https://doi.org/10.1007/s42514-024-00203-0
[3] B. Hess, C. Kutzner, D. van der Spoel, E. Lindahl, et al., GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable molecular simulation, J. Chem. Theory Comput. 4 (3) (2008) 435–447. https://doi.org/10.1021/ct700301q

[4] M. Vázquez, G. Houzeaux, S. Koric, A. Artigues, J. Aguado-Sierra, R. Arís, D. Mira, H. Calmet, F. Cucchietti, H. Owen, A. Taha, E.D. Burness, J.M. Cela, M. Valero, Alya: multisphysics engineering simulation toward exascale, J. Comput. Sci. 14 (2016) 15–27. https://doi.org/10.1016/j.jocs.2015.12.007

[5] D. Caviedes-Voullième, M. Morales-Hernández, M.R. Norman, I. Özgen Xian, SERGHEI (SERGHEI-SWE) V1.0: a performance-portable high-performance parallel-computing shallow-water solver for hydrology and environmental hydraulics, Geosci. Model Dev. 16 (3) (2023) 977–1008. https://doi.org/10.5194/gmd-16-977-2023

[6] R. Martínez-Cuenca, J. Luis-Gómez, S. Iserte, S. Chiva, On the use of deep learning and computational fluid dynamics for the estimation of uniform momentum source components of propellers, iScience 26 (2023) 1–14. https://doi.org/10.1016/j.isci.2023.108297

[7] P. Rosciszewski, A. Krzywaniak, S. Iserte, K. Rojek, P. Gepner, Optimizing throughput of seq2seq model training on the IPU platform for AI-accelerated CFD simulations, Future Gen. Comput. Syst. 143 (2023) 149–162. https://doi.org/10.1016/j.future.2023.05.004

[8] W.F. Godoy, P. Valero-Lara, K. Teranishi, P. Balaprakash, J.S. Vetter, Large language model evaluation for high-performance computing software development, Concurrency Comput.: Pract. Exp. 36 (26) (2024) e8269. https://doi.org/10.1002/cpe.8269

[9] H.-J. Bungartz, C. Riesinger, M. Schreiber, G. Snelting, A. Zwinkau, Invasive computing in HPC with X10, in: Proceedings of the Third ACM SIGPLAN X10 Workshop, X10 '13, ACM, New York, NY, USA, 2013, pp. 12–19. https://doi.org/10.1145/2481268.2481274

[10] M. Garcia, J. Labarta, J. Corbalan, Hints to improve automatic load balancing with leWI for hybrid applications, J. Parallel Distrib. Comput. 74 (9) (2014) 2781–2794. https://doi.org/10.1016/j.jpdc.2014.05.004

[11] V. Lopez, J. Criado, R. Peñacoba, R. Ferrer, X. Teruel, M. Garcia-Gasulla, An openMP free agent threads implementation, in: OpenMP: Enabling Massive Node-Level Parallelism: 17th International Workshop on OpenMP, IWOMP 2021, Bristol, UK, September 14-16, 2021, Proceedings, Springer-Verlag, Berlin, Heidelberg, 2021, pp. 211–225. https://doi.org/10.1007/978-3-030-85262-7_15

[12] D. Huber, M. Streubel, I. Comprés, M. Schulz, M. Schreiber, H. Pritchard, Towards dynamic resource management with MPI sessions and PMIx, in: Proceedings of the 29th European MPI Users' Group Meeting, EuroMPI/USA '22, Association for Computing Machinery, New York, NY, USA, 2022, p. 57–67. https://doi.org/10.1145/3555819.3555856

[13] S. Iserte Agut, High-throughput Computation through Efficient Resource Management, Ph.D. Thesis, Universitat Jaume I, 2018. Accepted: 2018-12-05T10:00:25Z Publication Title: TDX (Tesis Doctorals en Xarxa). https://doi.org/10.6035/14101.2018.176272

[14] G. Martín, M.-C. Marinescu, D.E. Singh, J. Carretero, FLEX-MPI: An MPI extension for supporting dynamic load balancing on heterogeneous non-dedicated systems, in: Euro-Par Parallel Processing, 2013, pp. 138–149.

[15] R. Bhattarai, H. Pritchard, S. Ghafoor, Dynamic resource management for elastic scientific workflows using PMix, in: 2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2024. https://doi.org/10.1109/IPDPSW63119.2024.00131

[16] S. Iserte, I. Martín-Álvarez, K. Rojek, J.I. Aliaga, M. Castillo, W. Folwarska, A.J. Peña, Resource optimization with MPI process malleability for dynamic workloads in HPC clusters, Future Gen. Comput. Syst. (2025) 107949. https://doi.org/10.1016/j.future.2025.107949

[17] R. Sudarsan, C.J. Ribbens, ReSHAPE: a framework for dynamic resizing and scheduling of homogeneous applications in a parallel environment, in: 2007 International Conference on Parallel Processing (ICPP 2007), 2007, pp. 44–44. https://doi.org/10.1109/ICPP.2007.73

[18] O. Sarood, A. Langer, A. Gupta, L. Kale, Maximizing throughput of overprovisioned HPC data centers under a strict power budget, in: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '14, IEEE Press, New Orleans, Louisiana, 2014, pp. 807–818. https://doi.org/10.1109/SC.2014.71

[19] S. Prabhakaran, M. Neumann, S. Rinke, F. Wolf, A. Gupta, L.V. Kale, A batch system with efficient adaptive scheduling for malleable and evolving applications, in: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium, IPDPS '15, IEEE Computer Society, USA, 2015, pp. 429–438. https://doi.org/10.1109/IPDPS.2015.34

[20] S. Iserte, R. Mayo, E.S. Quintana-Ortí, V. Beltran, A.J. Peña, DMR API: Improving cluster productivity by turning applications into malleable, Parallel Comput. 78 (2018) 54–66. https://doi.org/10.1016/j.parco.2018.07.006

[21] R. Sudarsan, C.J. Ribbens, D. Farkas, et al., Dynamic resizing of parallel scientific simulations: a case study using LAMMPS, in: Proceedings of the 9th International

Conference on Computational Science: Part I, ICCS '09, Springer-Verlag, Berlin, Heidelberg, 2009, pp. 175–184. https://doi.org/10.1007/978-3-642-01970-8_18

[22] S. Iserte, K. Rojek, A study of the effect of process malleability in the energy efficiency on GPU-based clusters, J. Supercomput. 76 (2020) 255–274. https://doi.org/10.1007/s11227-019-03034-x

[23] S. Iserte, H. Martínez, S. Barrachina, M. Castillo, R. Mayo, A.J. Peña, et al., Dynamic reconfiguration of noniterative scientific applications: a case study with HPG aligner, Int. J. High Perform. Comput. Appl. 33 (5) (2019) 804–816. https://doi.org/10.1177/1094342018802347

[24] N. Zakay, D.G. Feitelson, Preserving user behavior characteristics in trace-based simulation of parallel job scheduling, in: 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems, 2014, pp. 51–60. https://doi.org/10.1109/MASCOTS.2014.15

[25] S. Schlagkamp, Influence of dynamic think times on parallel job scheduler performances in generative simulations, in: 3rd Conference on Networked Systems Design & Implementation, 2017. https://doi.org/10.1007/978-3-319-61756-5_7

[26] B. Schroeder, A. Wierman, M. Harchol-Balter, Open versus closed: a cautionary tale, in: Proceedings of the 3rd Conference on Networked Systems Design & Implementation - Volume 3, NSDI'06, USENIX Association, USA, 2006, p. 18.

[27] N. Zakay, D.G. Feitelson, On identifying user session boundaries in parallel workload logs, in: Workshop on Job Scheduling Strategies for Parallel Processing, 2013. https://doi.org/10.1007/978-3-642-35867-8_12

[28] M. Madon, G. Da Costa, J.-M. Pierson, Replay with feedback: how does the performance of HPC system impact user submission behavior?, Future Gen. Comput. Syst. 155 (C) (2024) 66–79. https://doi.org/10.1016/j.future.2024.01.024

[29] D.G. Feitelson, Resampling with feedback: a new paradigm of using workload data for performance evaluation, in: Workshop on Job Scheduling Strategies for Parallel Processing, 2021. https://doi.org/10.1007/978-3-030-88224-2_1

[30] D.G. Feitelson, Packing schemes for gang scheduling, in: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, IPPS '96, Springer-Verlag, Berlin, Heidelberg, 1996, pp. 89–110.

[31] U. Lublin, D.G. Feitelson, The workload on parallel supercomputers: modeling the characteristics of rigid jobs, J. Parallel Distrib. Comput. 63 (11) (2003) 1105–1122. https://doi.org/10.1016/S0743-7315(03)00108-4

[32] J.I. Aliaga, M. Castillo, S. Iserte, I. Martín-Álvarez, R. Mayo, et al., A survey on malleability solutions for high-performance distributed computing, Appl. Sci. 12 (2022) 1–32. https://doi.org/10.3390/app12105231

[33] A. Tarraf, M. Schreiber, A. Cascajo, J.-B. Besnard, M.-A. Vef, D. Huber, S. Happ, A. Brinkmann, D.E. Singh, H.-C. Hoppe, A. Miranda, A.J. Peña, R. Machado, M.G. Gasulla, M. Schulz, P. Carpenter, S. Pickartz, T. Rotaru, S. Iserte, V. Lopez, J. Ejarque, H. Sirwani, F. Wolf, et al., Malleability in modern HPC systems: current experiences, challenges, and future opportunities, IEEE Trans. Parallel Distrib. Syst. (2024) 1–14. https://doi.org/10.1109/TPDS.2024.3406764

[34] S. Iserte, R. Mayo, E.S. Quintana-Ortí, A.J. Peña, et al., DMRlib: Easy-coding and efficient resource management for job malleability, IEEE Trans. Comput. 70 (2020) 1443–1457. https://doi.org/10.1109/TC.2020.3022933

[35] D.G. Feitelson, L. Rudolph, Towards convergence in job schedulers for parallel supercomputers, in: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, IPPS '96, Springer-Verlag, Berlin, Heidelberg, 1996, pp. 1–26.

[36] S. Iserte, G. Houzeaux, P. Sandás, A.J. Peña, M. Garcia-Gasulla, Malleable computational fluid dynamics simulations, in: Proceedings of the 36th Parallel CFD International Conference, Merida, Yucatan, Mexico, 2025.

[37] I. Martín-Álvarez, J.I. Aliaga, M. Castillo, S. Iserte, et al., Proteo: a framework for the generation and evaluation of malleable MPI applications, J. Supercomput. (2024). https://doi.org/10.1007/s11227-024-06277-5

[38] D. Huber, S. Iserte, M. Schreiber, A.J. Peña, M. Schulz, et al., Bridging the gap between genericity and programmability of dynamic resources in HPC, in: ISC High Performance 2025 Research Paper Proceedings (40Th International Conference), 2025, pp. 1–11.

[39] S. Iserte, I. Martín-Álvarez, K. Rojek, J.I. Aliaga, M. Castillo, A.J. Peña, Towards the democratization and standardization of dynamic resources with MPI spawning, in: R. Wyrzykowski, J. Dongarra, E. Deelman, K. Karczewski (Eds.), Parallel Processing and Applied Mathematics, Springer Nature Switzerland, Cham, 2025, pp. 287–300. https://doi.org/10.1007/978-3-031-85697-6_19

[40] V. Lopez, G. Ramírez Miranda, M. Garcia-Gasulla, et al., TALP: A lightweight tool to unveil parallel efficiency of large-scale executions, in: Proceedings of the 2021 on Performance Engineering, Modelling, Analysis, and VisualizationOn Strategy, PERMAVOST '21, Association for Computing Machinery, New York, NY, USA, 2021, pp. 3–10. https://doi.org/10.1145/3452412.3462753

[41] K. Rojek, R. Wyrzykowski, Parallelization of 3D MPDATA algorithm using many graphics processors, in: Proceedings of the 13th International Conference on Parallel Computing Technologies - Volume 9251, Guide Proceedings, 2015, pp. 445–457. https://doi.org/10.1007/978-3-319-21909-7_43