# On the Impact of Interference from Concurrent Jobs on Checkpointing Performance

Méline Trochon, Jean-Thomas Acquaviva, Francieli Boito, Brice Goglin, François Tessier, Luan Teylo

► To cite this version:

Méline Trochon, Jean-Thomas Acquaviva, Francieli Boito, Brice Goglin, François Tessier, et al. On the Impact of Interference from Concurrent Jobs on Checkpointing Performance. 2025. hal-05294610v2

HAL Id: hal-05294610

https://hal.science/hal-05294610v2

Preprint submitted on 29 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On the Impact of Interference from Concurrent Jobs on Checkpointing Performance

Méline Trochon¹,², Jean-Thomas Acquaviva², Francieli Boito¹, Brice Goglin¹, François Tessier³, Luan Teylo¹

¹Univ. Bordeaux, CNRS, Bordeaux INP, Inria, LaBRI, UMR 5800, F-33400, Talence, France
{first.last}@inria.fr

²DataDirect Networks, Paris, France
{mtrochon, jtacquaviva}@ddn.com

³University of Rennes, Inria, CNRS, IRISA - Rennes, France
francois.tessier@inria.fr

**Abstract**—I/O has been identified as one of the main bottlenecks in HPC. Among the most I/O-intensive operations is checkpointing, which is necessary to save the state of an application and allow it to be restarted at an advanced stage of computation. However, near the parallel file system, concurrency prevents checkpoint phases from reaching the best I/O performance. In this paper, we study I/O interference in this specific context: we look at performance of a checkpoint phase when faced with different interference patterns, exploring aspects such as scale, number of processes, operation, number of files, etc. Through an extensive experimentation, in two systems, we show the impact of these aspects on checkpoint. Moreover, we show that some configurations — namely, an application that does random accesses — lead to degraded system I/O performance. This paper provides an important background for any effort into mitigating I/O interference.

**Index Terms**—high-performance computing, parallel file systems, cross-application interference, checkpoint.

## I. INTRODUCTION

High-performance computing (HPC) plays a critical role in advancing scientific discovery and innovation by enabling the rapid processing of complex simulations and large-scale data analyses. As the scale of HPC applications continues to grow, the performance of the underlying storage infrastructure becomes increasingly vital to ensure that data can be read and written at speeds that match the computational demands.

In these systems, persistent data is stored in a parallel file system (PFS), which distributes files across a number of storage targets (OSTs), from where they can be retrieved in parallel. An important thing to notice about this I/O infrastructure is that it is shared by all running jobs. In this context, concurrent accesses may have lower performance compared to what they would have in isolation, in a phenomenon typically called *interference* [23]. It has been reported that interference may depend on application characteristics, with some applications being particularly detrimental to the others [20], [22], [23]. Furthermore, concurrent accesses may degrade the global bandwidth of the I/O infrastructure, wasting expensive and power-hungry resources [4]. Finally, interference causes performance variability, with the same application often observing I/O performances that can vary by orders of magnitude [4], [18], [21]. In this paper, we are interested in investigating this interference phenomenon.

More specifically, we are interested in the impact of interference on checkpoint operations. Checkpointing is a crucial technique in HPC for ensuring fault tolerance and minimizing the loss of computational progress in the event of system failures. Indeed, as HPC systems scale, the probability of hardware or software failures during long-running applications increases significantly. Checkpointing consists in periodically saving application data to persistent storage, from where it can be retrieved to restart it in case of failure. As applications become larger and handle more data, the amount of data to persist in checkpoint files also increases, making the performance of these I/O operations more critical.

For example, large HPC applications, like the GYSELA simulation code [10], write large amounts of data during checkpointing, and that amount is expected to grow when running at larger scale. During a previous work [16], we have studied Gysela and shown the importance of well tuning checkpointing I/O operations as to prepare the code to turn on bigger HPC architectures.

Another relevant example comes from training large language models (LLMs) [19]. Indeed, these models are trained for a long time — the training of LLaMA3-405B on 16,000 GPUs encountered failures every three hours on average [8], and failures during the two-month training of OPT-175B resulted in the loss of 178,000 GPU hours [25]. Moreover, these LLM checkpoints can be very large, e.g. 4.2 TB for GPT-3 [5] and 6.5 TB for the PaLM-540B model [6].

Our goal is to improve checkpointing performance by mitigating interference. In order to do that, our first step, presented in this paper, is to **investigate how checkpoint operations are impacted by concurrent applications' characteristics**. We conduct a large experimental campaign, on two systems, featuring BeeGFS and Lustre, HDDs and SSDs, where we study one application aspect at a time. Among other conclusions, we show that the scale and file strategy of the interfering code are the most important predictors of checkpoint performance, and that sharing the I/O infrastructure with an application that issues random requests may degrade the global I/O performance. Our findings provide important information for anyone working to mitigate I/O interference, specially for checkpoint operations.
The remainder of this paper is organized as follows. The
next section discusses related work. Section III discusses the
experimental methodology of this study. Results are presented
in Section IV, separated by aspect. In Section V we use an ML
model to rank all the studied characteristics by importance in
the prediction of the checkpoint performance and variability.
Finally, Section VI summarizes our findings and discusses
future work.

## II. RELATED WORK

Some solutions, such as the one proposed by Wang et al. [19], leverage application-specific characteristics (e.g., LLM training) to improve checkpointing performance, for example by reducing the amount of data that must be stored. Meanwhile, Herault et al. [11] show that combining optimal checkpoint periods (defined by the Young/Daly formula [24]) with I/O scheduling strategies significantly improves overall application performance. In their approach, they employ cooperative algorithms that prevent applications from checkpointing concurrently. Specifically, they propose different strategies in which either the application is blocked while waiting for the I/O token (i.e., its turn to perform the checkpoint), or it continues processing while waiting for the token. Their results demonstrate that even the blocking strategies outperform uncoordinated checkpointing, highlighting the negative impact of concurrent I/O on checkpointing operations.

Our study is complementary to these efforts, as it aims to
understand how interference affects the write performance of
checkpointing. The insights gained from our analysis can be
leveraged to enhance the effectiveness of such approaches.

I/O interference is a significant challenge in HPC. When multiple jobs or processes access shared storage resources concurrently, they often compete for bandwidth and I/O operations, leading to contention and unpredictable slowdowns [20], [22], [23]. Numerous research efforts have addressed this issue. These include I/O scheduling strategies [3], [15], the use of temporary I/O resources to isolate jobs, such as burst buffers [18] and ephemeral file systems [17], and adaptive strategies that observe the current system load and adjust application I/O behavior accordingly (e.g., reducing the amount of data to be written) [14], among others.

Wyatt et al. [20] train a machine learning model to classify applications as sensitive or not to I/O interference, using features obtained from the batch scheduler (user, job name, etc.). They measure interference by identifying jobs that execute concurrently with an injected I/O benchmark and comparing their performance to similar jobs of the same application that did not share the system with the benchmark. In contrast, our results show that the I/O slowdown suffered by applications does *not* depend solely on their own characteristics, but also on those of the competing workload. Moreover, our work does not aim to detect when I/O degradation occurs, but rather to understand why it happens in the presence of concurrent applications. Instead of relying on a limited set of I/O profiles (they consider five), we conduct a detailed experimental study that isolates and evaluates the impact of different application

characteristics on the I/O performance of checkpointing operations.

Boito et al. [2] identified that executing multiple instances of the same pattern, namely an IOR execution with large, contiguous requests to a shared-file, does *not* degrade global performance, as the instances share the available bandwidth equally. Their findings are limited to up to 4 concurrent applications, all identical and using the same number of compute resources.

In [9], the authors propose a new benchmark called NumIO that mimics not only the I/O phase but also the computation and communication phases of applications. They then use NumIO to study the impact of concurrent applications on runtime performance. Unlike our work, their experiments are limited to concurrency between applications with identical characteristics (i.e., concurrent executions of the same application). Moreover, their main focus is on the overlap between different phases and its impact on shared resources.

As part of a proposal for automatically allocating I/O forwarding nodes to applications, Ji et al. [12] studied interference between pairs of applications sharing these nodes. They found that the following characteristics make an application more disruptive to others: i) using the shared-file strategy, ii) being metadata-intensive, and iii) issuing large contiguous requests (when the concurrent application uses small requests). In this paper, we conduct a broader analysis that includes additional application characteristics and targets parallel file systems. This leads to different conclusions, which we discuss in Section IV.

Yildiz et al. [23] experimentally investigated cross-application interference using the OrangeFS parallel file system and the IOR benchmarking tool. They found that: i) non-contiguous patterns caused more severe interference, particularly when HDDs were used for storage; and ii) using more processes per node increased interference. In our results, we confirm the first finding but not the second. Furthermore, we extend the analysis to additional scenarios, including applications at different scales.

## III. APPROACH

To assess the impact of interference on applications performing checkpoints, we ran a large number of experiments on two systems. Each experiment consists of running a *checkpoint* operation, representing a single checkpointing phase, at the same time as another code, here called *interference*. We change the characteristics of the latter and measure the I/O bandwidth observed by each application.

The code is publicly available.<fnref n="1" />

### A. *Experimental platforms*

Part of the experiments are conducted on PlaFRIM, a 132-node experimental platform located at the Inria Center at the University of Bordeaux, in France. We use the Bora cluster, whose 44 nodes are powered by two 18-core Intel Xeon

<footnote n="1">¹https://gitlab.inria.fr/hpc_io/interference_benchmark</footnote>
processors and 192 GB of RAM. All Bora nodes run CentOS 7.6.1810 with Linux kernel v.3.10.0-957.el7.x86_64.

In terms of storage, PlaFRIM features a BeeGFS v.7.2.6 parallel file system, deployed across two hosts used for both data and metadata. Each host runs one Object Storage Server (OSS) with four OSTs and one Metadata Server (MDS) with one Metadata Target (MDT). The OSTs use 12 Toshiba AL15SEB18EQY HDDs with 10,000 RPM and 1.8 TB of capacity, organized in RAID-6. For metadata, each MDT uses one Samsung SSD of 1.6 TB. The Bora nodes and storage servers are connected via a 100 Gbit/s Omni-Path network.

Another part of the experiments was conducted on DDN euLab, located at CINECA in Bologna, Italy. The tests were performed on the X440 partition, which is composed of 16 nodes powered by an AMD EPYC 7543P 32-Core Processor and 252 GB of RAM. The nodes run Ubuntu 24.04.2 LTS with Linux kernel v.6.8.0-63-generic. In terms of storage, DDN euLab features a PFS with Lustre v.2.14.0_ddn213 deployed on 4 OSS servers, each hosting 2 OSTs (8 in total), and 4 MDTs (one per MDS). Nonetheless, during our experiments, one of the OSS was observed to be much slower than the others, and hence we used only the other 3 OSS (6 OST). These OSTs, which were used in our experiments, store data in SSDs, and additional 4 OSTs form an HDD tier. The storage servers are connected to the compute nodes via four 100 Gbit/s HDR links.

In both platforms, we forced all files to be striped across all used OSTs (8 in PlaFRIM and 6 in DDN euLab), to ensure all PFS resources are shared by the applications.

## B. Benchmarking tool

We use the IOR benchmarking tool [1] (version 4.1.0) to create both the checkpoint and interference operations. Compared to running real applications that checkpoint, using IOR provides us with more generalizable results and allows us to easily explore different access patterns. Moreover, IOR is widely used in the HPC community, which ensures reproducibility and facilitates comparison across studies.

We use the POSIX API for the tests because we want to completely control the applications I/O behaviors. However, it is important to notice that such POSIX behaviors could be the result of using a higher-level I/O library. For example, MPI-IO collective write operations would generate a pattern of sequential, large POSIX write requests to a shared file. In this sense, extending this analysis to use higher-level I/O libraries such as MPI-IO and HDF5 would only make the results harder to interpret, without really including new behaviors.

1) The checkpoint operation: To represent a checkpoint writing phase, we selected I/O parameters such as to replicate the operation in the GYSELA simulation [16], which is designed for gyrokinetic studies of plasma turbulence in fusion devices. Using a semi-Lagrangian approach, it enables detailed analysis of transport phenomena in tokamak plasma to replicate future ITER devices [10]. The operation is performed by multiple processes per node, using a file-per-process strategy, and issuing sequential, large (8 MiB) requests to the PFS. The

total amount of data written per node corresponds to one fourth of its available RAM memory.

Other parameters — number of nodes and of processes per node — were experimentally chosen to ensure the checkpoint can reach close to the peak performance of the system by itself. This was done to facilitate the quantification of interference, because the application is sure to be impacted by sharing the system with another. On PlaFRIM, we used 8 nodes with 8 processes per node, each process writing a total of 6 GiB [2]. On DDN euLab, we used 4 nodes, with 32 processes per node, each process writing a total of 4 GiB.

2) The interference codes: We aimed at covering a large diversity of interfering codes, and thus we varied multiple parameters that are known to impact I/O performance [2], [7], [13], [18]: file strategy (file-per-process vs. shared-file), request size, spatiality (sequential vs. random), amount of data, operation (read vs. write), and number of nodes and processes. Table I summarizes all the tested configurations.

As to avoid cache effects while reading, in our experiments: a write test is performed first, with multiple iterations and creating different file(s) per iteration (-m option in IOR), and then the read test executes, with multiple iterations, reading the files at the same order as they were written. As long as the sum of the file sizes is larger than the cache size, which is the case because it is larger than the total RAM of the nodes, than files should not be found in the clients' cache.

## C. Benchmark execution

The two codes are launched asynchronously from the same job script. Each IOR is configured to repeat at least five times (-i 5), with no waiting time between iterations. The different experiments are repeated 3 times, providing therefore fifteen performance measurements (one per iteration). We discard all iterations that did not overlap with the other concurrent application, because we are interested in the interference impact. For partially-overlapping iterations, we estimate the bandwidth of the overlapping portion, using the bandwidth obtained by the application when running in isolation, and only keep this part. Then we take the median of the measurements for each application. In some experiments we had to adapt the number of iterations to ensure there were enough overlapping iterations (notably when the interference code is many times slower than the checkpoint).

It is important to notice that in our experiments both applications are executed inside the same job (and consequently by the same user and project id). We chose to do that to eliminate the impact of the request schedulers of the file system servers, which may do some variation of a round robin between jobs and/or users (as is the case for the BeeGFS in PlaFRIM). By doing so, we can study what the interference looks like when sharing I/O resources, in a way that is more generic (because different resources could have different strategies) and can even be used to guide the proposal of new scheduling algorithms.
<table><caption>TABLE I: Configuration of interference applications used in this study. Parenthesis refer to the used IOR options. The checkpoint application is always the same: 64 processes on 8 nodes in PlaFRIM or 128 processes on 4 nodes in DDN euLab, writing sequentially to file-per-process in 8 MiB requests for a total of 6 GiB per process in PlaFRIM (4 GiB in DDN euLab).</caption><thead><tr><th>System</th><th>Nodes</th><th>Processes (total)</th><th>Operation (-x, -w)</th><th>File strategy (-F)</th><th>Spatiality (-z)</th><th>Data per process (-b)</th><th>Transfer size (-t)</th></tr></thead><tbody><tr><td>PlaFRIM</td><td>4, 6, 8, 12, 16</td><td>32, 64, 128</td><td>write, read</td><td>single shared file (SSF) or file-per-process (FFP)</td><td>sequential or random</td><td>3, 6, 12 GiB</td><td>4, 16, 64, 256, 512, 2048, 8192 KiB</td></tr><tr><td>DDN euLab</td><td>2, 4, 8</td><td></td><td></td><td></td><td></td><td>1, 2, 4, 8 GiB</td><td>4, 64, 512, 2048, 8192 KiB</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Experiments on PlaFRIM</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Experiments on DDN euLab</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Experiments on PlaFRIM</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Experiments on DDN euLab</figcaption></figure>

## IV. RESULTS

In this section, we present and discuss the obtained results. We organize this discussion by looking at one characteristic of the interfering code at a time, to investigate its impact. It is important to notice the checkpoint application is always the same, as described in Section III-B1.

Throughout this section, due to space constraints and to the large set of experiments that were conducted, we present only a part of them. The full set of results can be found in the Appendix.

### A. Number of compute nodes

Fig. 1 presents the results for a scenario where both applications are identical (including in the number of processes per node), except for the number of nodes used by the interference code. We can see that when using the same number of nodes (8 for PlaFRIM, 4 for DDN euLab), the applications share equally the available bandwidth (as expected, since they are the same). However, in the other cases, the application that uses more nodes reaches a proportionally larger performance.

In both systems, the first bar is slightly lower than the others, because the combined number of nodes, used by the two applications, is not able to saturate the system performance.

The observed behavior is somewhat expected, as using more nodes means having more available resources (the PFS clients, thread pools, network, etc). Moreover, this behavior is arguably desirable, since, from the system point of view, it may make more sense to prioritize the application that is using more compute resources, so it can finish and free these resources faster.

<figure><img src="image_6.png" /><figcaption>Fig. 2: Bandwidth of checkpoint (bottom, blue) and interference (top, orange) running together, varying the number of processes or the amount of data written per process of the interference (x-axes). The two applications are otherwise identical. The lines depict their performance when running in isolation.</figcaption></figure>

### Interference factor #1: number of compute nodes

From the checkpointing point of view, it is important to notice that sharing the PFS with an application that runs at a higher scale may mean reaching a write performance much lower than what would be observed in isolation.

### B. Number of processes and total amount of data

Similarly to what was done in the previous section, we now investigate the impact of the number of processes, which has been shown in the literature to impact performance [2] and interference [23]. To do that, we select an interference code that is identical to the checkpoint one, except for the number of processes. Results can be seen in Fig. 2a. In PlaFRIM (BeeGFS), this parameter does not impact how the two applications share the available bandwidth. Indeed, all the processes in a node share a single PFS client and network interface, so as long as the system performance is
saturated, adding processes does not change the performance of the application relative to the other concurrent ones.

However, in DDN euLab (Lustre), we observe the *interference* performance is more sensitive to the number of processes per nodes. When sharing the system with the *checkpoint* (which uses 128 processes), the *interference* gets higher bandwidth when it has less processes (without lowering *checkpoint* performance, nonetheless). This variation in global I/O performance is due to the system not being saturated with less than 32 processes per node. Still, these results seem to confirm the observation by Yildiz et al. [23], where using less processes per node resulted in suffering less interference.

### Interference factor #2: number of processes

In some systems, and specifically when they are not saturated, using less processes per node may lead to suffering less interference from concurrent applications.

Fig. 2b shows results of experiments where we vary the amount of data per process of the *interference* application. We found that this characteristic has no impact on interference.

### C. File strategy

The first plot (in the left) of Fig. 4a and 4b presents results for a scenario where the *interference* application writes to a shared-file, whereas the *checkpoint*, as always in this paper, uses the file-per-process strategy. Comparing these results to the ones previously seen in Fig. 1, we can see the *checkpoint* is getting a larger portion of the available bandwidth, as the *interference* has to be twice as large to reach half of it in PlaFRIM, and that did not even happen in DDN euLab.

### Interference factor #3: file strategy

If the *checkpoint* is done to a shared-file, it may be heavily impacted by concurrent applications that use the file-per-process strategy. This is the opposite of what was observed in the literature when sharing I/O forwarding nodes [12].

### D. Spatiality

Still looking at Fig. 4a and 4b, we can see the impact on performance when the *interference* code issues requests to random offsets of the file(s). In PlaFRIM, that does *not* impact the proportions of the available bandwidth that the two applications receive. However, it does severely impact the global bandwidth of the system.

When the *interference* does random accesses, the combined system throughput decreases in up to three times. Moreover, its collapse is more pronounced the larger the *interference* application is (regarding the number of compute nodes). Finally, interestingly, the combined throughput when the *interference* code does random access under single-shared-file remains higher than under file-per-process (6.5 GiB/s vs. 5.3 GiB/s at 4 nodes), reinforcing that

<figure><img src="image_7.png" /></figure>

(a) Experiments on PlaFRIM

<figure><img src="image_8.png" /></figure>

(b) Experiments on DDN euLab using SSDs

<figure><img src="image_9.png" /></figure>

(c) Experiments on DDN euLab using hard disks

Fig. 3: Bandwidth of *checkpoint* (bottom, blue) and *interference* (top, orange) running together, varying the number of nodes used by the *interference* (x-axis), which writes using different file strategies and spatialities. Please notice the file-per-process sequential plot was shown in Fig. 1. The two applications are otherwise identical. The lines depict their performance when running in isolation.

file-per-process combined with random access is the most risky scenario for concurrent workloads.

On the other hand, in DDN euLab, we do *not* observe the same degradation of global performance. Indeed, when issuing random requests, the *interference* code gets less of the bandwidth compared to when it issues sequential requests, but the *checkpoint* is able to use the rest of it.

We believe an explanation for this comes from the use of SSDs in DDN euLab. While issuing random requests still means the *interference* has lower performance, in part due to not benefitting as well from cache mechanisms, it does not impose such a heavy slow-down to the shared servers. To test this hypothesis, we conduct more experiments in DDN euLab using the HDD tier of OSTs. Results are presented in Fig. 3c, and show a somewhat similar behavior than what was seen in PlaFRIM, albeit with a more modest degradation of global performance (and only for the file-per-process case).
<figure><img src="image_10.png" /><figcaption>(a) Experiments on PlaFRIM</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Experiments on DDN euLab</figcaption></figure>

Fig. 4: Bandwidth of checkpoint (bottom, blue) and interference (top, orange) running together, varying the transfer size used by the interference (x-axis), which writes using different file strategies. The two applications are otherwise identical. The lines depict the performance of the applications when running in isolation. Results for random accesses can be found in the appendix.

### Interference factor #4: spatiality

If the system stores data in HDDs, sharing the system with an application that does random accesses, even if in smaller scale, may severely degrade not only the checkpoint performance, but the overall efficiency of the system. That confirms but also extends the findings from Yildiz et al. [23].

### E. Transfer size

Fig. 4 shows results for scenarios where we vary the transfer size used by the *interference*. Looking at the lines (the performance of applications in isolation), and most specifically to the *interference* line, we can see the impact of this parameter in performance is only important for single-shared file in PlaFRIM and file-per-process in DDN euLab, and much more in the former than in the latter. Accordingly, when the *interference* application has lower performance due to issuing smaller requests, the *checkpoint* gets more of the available bandwidth.

### Interference factor #5: transfer size

The checkpoint may have better performance when sharing the system with an application that issues small requests (when it, consequently, has lower performance).

### F. Operation

Finally, we investigate the impact of concurrent reads on checkpoint performance in Fig. 5. Here, for PlaFRIM, the variability of checkpoint performance is so high that we decided against summarizing results by their median, as

<figure><img src="image_12.png" /><figcaption>(a) Experiments on PlaFRIM</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) Experiments on DDN euLab using hard disks</figcaption></figure>

Fig. 5: Bandwidth of checkpoint (blue) and interference (orange) running together, when interference reads using different file strategies. Results for random accesses can be found in the appendix.

done so far. We observe that in PlaFRIM the performance of the read operation is rather consistent, while that of the checkpoint varies greatly, sometimes using most of the system performance, but sometimes being many times slower. That variation was *not* observed in DDN euLab, however, even when using the HDD storage tier (Fig. 5c). In that system, when read is done from per-process files and the two applications have the same scale, then the *interference* gets most of the bandwidth. The behavior is the opposite for shared-file *interference*.

### Interference factor #6: read operations

Concurrent read operations may induce high variability to checkpointing performance, sometimes severely degrading its performance and the global I/O bandwidth as well. That was observed using HDDs and BeeGFS. With Lustre, if the reads are done to independent files, then the checkpoint suffers from heavy interference.

## V. WHAT ASPECTS ARE THE MOST IMPORTANT?

We train a Random Forest model using all results generated in this study, and then look at the built-in feature importances to rank the characteristics of the *interference* application that are the best predictors for *checkpoint* performance. Results are shown in Fig. 6.

Aside from the number of processes per node, the key characteristics remain consistent across the machines used. As expected, the most influential factor is the number of nodes (Fig. 1). On both machines, the file strategy and spatiality (Fig. 3) also have a noticeable impact. However, the PlaFRIM machine is additionally affected by the transfer size (Fig. 4) and the type of operation (read or write, Fig. 5), whereas the DDN euLab machine is more sensitive to the number of
<figure><img src="image_14.png" /><figcaption>(a) Experiments on PlaFRIM</figcaption></figure>

<caption>(b) Experiments on DDN euLab</caption>

<caption>Fig. 6: The characteristics of interference that impact checkpoint, sorted by importance.</caption>

processes per node (Fig. 2a) and less influenced by transfer size and operation type.

## VI. CONCLUSION AND FUTURE WORK

In this paper, we presented a study into the interference imposed on checkpointing operations by other concurrent jobs that share the I/O infrastructure. We conducted a comprehensive experimental campaign where we varied many application characteristics that are known to have an impact on I/O performance — such as file strategy and spatiality. Our results confirm some findings from the literature, contradicts others, and extends them with more scenarios. We provide valuable information for any work aiming at mitigating I/O interference. As future work, we plan on working towards this goal for checkpoint operations, based on the conclusions of this study.

## ACKNOWLEDGEMENTS

The authors gratefully acknowledge the support from the ExaDoST project of the NumPEx PEPR program (ANR-22-EXNU-0004), as part of the France 2030 initiative. Experiments were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Université de Bordeaux, Bordeaux INP and Conseil Regional d'Aquitaine (see https://www.plafrim.fr).

## APPENDIX

In the following, we include the results that were removed from the main paper due to space constraints.

### A. Analysis of the impact of transfer size when interference issues random requests

When interference accesses are random, as illustrated in Fig. 7, the overall trend remains consistent. Although random accesses reduce the interference bandwidth, thereby decreasing the proportion of total bandwidth they consume, they do not alter the behavior seen in Fig. 4.

### B. Analysis of the impact of reads when interference issues random requests

Introducing random read accesses for interference, as shown in Fig. 8, does not significantly alter the behavior previously observed in Fig. 5. The significant variation in checkpoint bandwidth on PlaFRIM remains, and the overall pattern on the DDN euLab cluster continues to follow a similar trend.

<figure><img src="image_15.png" /><figcaption>(a) Experiments on PlaFRIM</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(b) Experiments on DDN euLab</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(a) Experiments on PlaFRIM</figcaption></figure>

<caption>(b) Experiments on DDN euLab</caption>

<caption>Fig. 8: Bandwidth of checkpoint (blue) and interference (orange) running together, when interference reads randomly using different file strategies.</caption>

## REFERENCES

[1] IOR benchmark version 4.1.0. https://github.com/hpc/ior, 2025.

[2] F. Boito, G. Pallez, and L. Teylo. The role of storage target allocation in applications' I/O performance with BeeGFS. In *CLUSTER 2022 - IEEE International Conference on Cluster Computing*, Heidelberg, Germany, Sept. 2022.

[3] F. Boito, G. Pallez, L. Teylo, and N. Vidal. IO-SETS: Simple and efficient approaches for I/O bandwidth management. *IEEE Transactions on Parallel and Distributed Systems*, 34(10):2783–2796, Aug. 2023.

[4] F. Boito, L. Teylo, M. Popov, T. Jolivel, F. Tessier, J. Luettgau, J. Monniot, A. Tarraf, A. Carneiro, and C. Osthoff. A Deep Look Into the Temporal I/O Behavior of HPC Applications. In *39th IEEE International Parallel & Distributed Processing Symposium (IPDPS)*, Milan, Italy, June 2025.

[5] T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.

[6] A. Chowdhery, S. Narang, J. Devlin, et al. Palm: Scaling language modeling with pathways. *J. Mach. Learn. Res.*, 24:240:1–240:113, 2023.

[7] F. Chowdhury, Y. Zhu, T. Heer, S. Paredes, A. Moody, R. Goldstone, K. Mohror, and W. Yu. I/O characterization and performance evaluation
of beegfs for deep learning. In *Proceedings of the 48th International Conference on Parallel Processing*, ICPP '19, New York, NY, USA, 2019. Association for Computing Machinery.

[8] A. Dubey, A. Jauhri, A. Pandey, et al. The llama 3 herd of models, 2024. arXiv preprint: https://arxiv.org/abs/2407.21783.

[9] A. Fuchs, J. Squar, and M. Kuhn. Ensemble-based system benchmarking for hpc. In *2024 23rd International Symposium on Parallel and Distributed Computing (ISPDC)*, pages 1–8. IEEE, 2024.

[10] V. Grandgirard, Y. Sarazin, X. Garbet, G. Dif-Pradalier, P. Ghendrih, N. Crouseilles, G. Latu, E. Sonnendrücker, N. Besse, and P. Bertrand. GYSELA, a full-f global gyrokinetic semi-lagrangian code for ITG turbulence simulations. 871.

[11] T. Herault, Y. Robert, A. Bouteiller, D. Arnold, K. Ferreira, G. Bosilca, and J. Dongarra. Optimal cooperative checkpointing for shared high-performance computing platforms. In *2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)*, pages 803–812, 2018.

[12] X. Ji, B. Yang, T. Zhang, X. Ma, X. Zhu, X. Wang, N. El-Sayed, J. Zhai, W. Liu, and W. Xue. Automatic, Application-Aware I/O forwarding resource allocation. In *17th USENIX Conference on File and Storage Technologies (FAST 19)*, pages 265–279, Boston, MA, Feb. 2019. USENIX Association.

[13] L. Pottier, R. F. da Silva, H. Casanova, and E. Deelman. Modeling the performance of scientific workflow executions on hpc platforms with burst buffers. In *2020 IEEE International Conference on Cluster Computing (CLUSTER)*, pages 92–103, 2020.

[14] Z. Qiao, Q. Liu, N. Podhorszki, S. Klasky, and J. Chen. Taming i/o variation on qos-less hpc storage: What can applications do? In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 1–13, 2020.

[15] M. H. Rashid and D. Dai. Adaptbf: Decentralized bandwidth control via adaptive token borrowing for hpc storage. In *Proceedings of the 39th IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, San Francisco, CA, May 2025. IEEE.

[16] M. Trochon, J. Bigot, V. Grandgirard, and D. Midou. Checkpointing Optimisation to Prepare Future Exascale Plasma Turbulence Simulations. In *IPDPSW 2025 - 39th IEEE International Parallel & Distributed Processing Symposium*, Milan, Italy, June 2025. IEEE.

[17] M.-A. Vef, N. Moti, T. Süß, T. Tocci, R. Nou, A. Miranda, T. Cortes, and A. Brinkmann. Gekkofs - a temporary distributed file system for hpc applications. In *2018 IEEE International Conference on Cluster Computing (CLUSTER)*, pages 319–324, 2018.

[18] T. Wang, S. Byna, G. K. Lockwood, S. Snyder, P. Carns, S. Kim, and N. J. Wright. A zoom-in analysis of i/o logs to detect root causes of i/o performance bottlenecks. In *2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)*, pages 102–111, 2019.

[19] Z. Wang, W. Zhu, Z. Zhang, C. Yan, F. Guo, Y. Li, and Y. Xu. Amber: Towards fast and space-efficient incremental checkpointing in large language model training. In *Proceedings of the 54th International Conference on Parallel Processing (ICPP)*, San Diego, CA, Sept. 2025. September 8–11, 2025.

[20] M. R. Wyatt, S. Herbein, K. Shoga, T. Gamblin, and M. Taufer. Canario: Sounding the alarm on io-related performance degradation. In *2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 73–83, 2020.

[21] B. Xie, Z. Tan, P. Carns, J. Chase, K. Harms, J. Lofstead, S. Oral, S. S. Vazhkudai, and F. Wang. Applying machine learning to understand write performance of large-scale parallel filesystems. In *2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW)*, pages 30–39, 2019.

[22] B. Yang, X. Ji, X. Ma, X. Wang, T. Zhang, X. Zhu, N. El-Sayed, H. Lan, Y. Yang, J. Zhai, W. Liu, and W. Xue. End-to-end I/O monitoring on a leading supercomputer. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)*, pages 379–394, Boston, MA, Feb. 2019. USENIX Association.

[23] O. Yildiz, M. Dorier, S. Ibrahim, R. Ross, and G. Antoniu. On the root causes of cross-application i/o interference in hpc storage systems. In *2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 750–759, May 2016.

[24] J. W. Young. A first order approximation to the optimum checkpoint interval. *Communications of the ACM*, 17(9):530–531, 1974.

[25] S. Zhang, S. Roller, N. Goyal, et al. OPT: open pre-trained transformer language models. CoRR, abs/2205.01068, 2022.