# A deep look into the temporal I/O behavior of HPC applications -extended version

Francieli Boito, Luan Teylo, Mihail Popov, Théo Jolivel, François Tessier, Jakob Luettgau, Julien Monniot, Ahmad Tarraf, André Carneiro, Carla Osthoff

► To cite this version:

Francieli Boito, Luan Teylo, Mihail Popov, Théo Jolivel, François Tessier, et al. A deep look into the temporal I/O behavior of HPC applications -extended version. RR-9577, Inria & Labri, Univ. Bordeaux. 2025, pp.1-42. hal-04978752

HAL Id: hal-04978752

https://inria.hal.science/hal-04978752v1

Submitted on 11 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Inria

# A deep look into the temporal I/O behavior of HPC applications — extended version

Francieli Boito, Luan Teylo, Mihail Popov, Theo Jolivel, François Tessier, Jakob Luettgau, Julien Monniot, Ahmad Tarraf, André Carneiro, Carla Osthoff

## RESEARCH REPORT

Nº 9577

March 2025

Project-Teams TADaaM and KerData
<figure><img src="image_2.png" /></figure>
Inria

A deep look into the temporal I/O behavior of
HPC applications — extended version

Francieli Boito*, Luan Teylo*, Mihail Popov*, Theo Jolivel†,
François Tessier†, Jakob Luettgau†, Julien Monniot†, Ahmad

Tarraf‡, André Carneiro§, Carla Osthoff§

Project-Teams TADaaM and KerData

Research Report n° 9577 — March 2025 — 42 pages

**Abstract:** The increasing gap between compute and I/O speeds in high-performance computing (HPC) systems imposes the need for techniques to improve applications’ I/O performance. Such techniques must rely on assumptions about I/O behavior in order to efficiently allocate I/O resources such as burst buffers, to schedule accesses to the shared parallel file system or to delay certain applications at the batch scheduler level to prevent contention, for instance. In this paper, we verify these common assumptions about I/O behavior, specifically about temporal behavior, using over 440,000 traces from real HPC systems. By combining traces from diverse systems, we characterize the behaviors observed in real HPC workloads. Among other findings, we show that I/O activity tends to last for a few seconds, and that periodic jobs are the minority, but responsible for a large portion of the I/O time. Furthermore, we make projections for the expected improvement yielded by popular approaches for I/O performance improvement. Our work provides valuable insights to everyone working to alleviate the I/O bottleneck in HPC.

**Key-words:** high-performance computing, parallel file systems, workload characterization, temporal I/O behavior

RESEARCH CENTRE
Centre Inria de l'Université de Bordeaux

200 avenue de la Vieille Tour
33405 Talence Cedex

<footnote>* Univ. Bordeaux, CNRS, Bordeaux INP, Inria, LaBRI, UMR 5800, F-33400, Talence, France.
{first name.last name}@inria.fr</footnote>

<footnote>† Univ Rennes, Inria, CNRS, IRISA, Rennes, France.</footnote>

<footnote>{first name.last name}@inria.fr</footnote>

<footnote>‡ Department of Computer Science, Technical University of Darmstadt, Darmstadt, Germany.
ahmad.tarraf@tu-darmstadt.de</footnote>

<footnote>§ LNCC, Petrópolis - RJ, 25651-075, Brazil.
{andrerc,osthoff}@lncc.br</footnote>
# Un regard approfondi sur le comportement temporel des E/S des applications HPC — version étendue

**Résumé** : L'écart croissant entre les vitesses de calcul et d'entrée/sortie dans les systèmes de calcul à haute performance (HPC) impose le besoin de techniques pour améliorer les performances des applications en matière d'E/S. Ces techniques doivent s'appuyer sur des informations sur le comportement des E/S afin d'allouer efficacement les ressources d'E/S telles que les *burst buffers*, d'ordonnancer les accès au système de fichiers parallèle partagé ou de retarder certaines applications au *batch scheduler* pour éviter la contention, par exemple. Dans cet article, nous vérifions ces hypothèses courantes sur le comportement des E/S, en particulier sur le comportement temporel, en utilisant plus de 440 000 traces provenant de systèmes HPC réels. En combinant des traces provenant de divers systèmes, nous caractérisons les comportements observés dans les charges de travail HPC réelles. Entre autres résultats, nous montrons que l'activité E/S a tendance à durer quelques secondes, et que les travaux périodiques sont minoritaires, mais responsables d'une grande partie du temps d'entrée/sortie. En outre, nous établissons des projections concernant l'amélioration attendue des approches les plus courantes en matière d'amélioration des performances d'E/S. Notre travail fournit des informations précieuses à tous ceux qui s'efforcent de réduire le goulot d'étranglement des E/S dans le domaine du calcul intensif.

**Mots-clés** : Calcul haute performance, systèmes de fichiers parallèles, caractérisation de la charge de travail, comportement temporel des E/S
# Contents

<table><tr><td>1</td><td>Introduction</td><td>4</td></tr><tr><td>2</td><td>Datasets of job I/O activity</td><td>5</td></tr><tr><td>2.1</td><td>Filters and final dataset sizes</td><td>6</td></tr><tr><td>2.2</td><td>General characteristics of the datasets</td><td>6</td></tr><tr><td>3</td><td>When do applications perform I/O?</td><td>8</td></tr><tr><td>4</td><td>Are applications periodic?</td><td>10</td></tr><tr><td>4.1</td><td>Common periods</td><td>11</td></tr><tr><td>4.2</td><td>Improving performance for periodic applications only</td><td>13</td></tr><tr><td>4.3</td><td>Improving performance after observing some I/O phases</td><td>15</td></tr><tr><td>5</td><td>How long are I/O intervals?</td><td>15</td></tr><tr><td>5.1</td><td>Improving performance after observing recent accesses</td><td>17</td></tr><tr><td>5.2</td><td>Potential for asynchronous I/O</td><td>18</td></tr><tr><td>6</td><td>Do applications always present the same temporal I/O behaviors?</td><td>20</td></tr><tr><td>7</td><td>Are there usage behavior patterns for users and projects?</td><td>23</td></tr><tr><td>8</td><td>How common are concurrent accesses to the shared I/O infrastructure?</td><td>24</td></tr><tr><td>9</td><td>A classification of application temporal I/O behaviors</td><td>29</td></tr><tr><td>9.1</td><td>Methodology for clustering I/O time series</td><td>29</td></tr><tr><td>9.2</td><td>Categories of I/O behavior</td><td>30</td></tr><tr><td>10</td><td>Related Work</td><td>33</td></tr><tr><td>11</td><td>Conclusion</td><td>34</td></tr><tr><td>A</td><td>On the coverage of Blue Waters jobs by Darshan</td><td>41</td></tr></table>
# 1 Introduction

In high-performance computing (HPC) platforms, applications, usually submitted as batch jobs, rely on a parallel file system (PFS) to access persistent data. The PFS is deployed over a set of dedicated resources, separate from compute resources, which are shared by all concurrent jobs. Over the years, the gap between computing performance and the ability of storage systems to store and retrieve data has widened considerably: while supercomputers have become increasingly efficient for computing tasks, storage systems have become relatively less efficient [51]. At the same time, new, non-traditional, HPC applications such as data analytics and machine learning have shifted the workload toward more data intensity [32]. Therefore, I/O has been an important bottleneck for performance and efficient system usage at scale.

Many techniques have been proposed to improve I/O performance, often focusing on mitigating contention in concurrent accesses. Common examples are the allocation of resources such as burst buffers or I/O nodes [4, 33, 34, 48, 63, 67, 69, 73, 75, 78], scheduling concurrent accesses to the shared PFS [6, 13, 26, 40, 44], adapting applications (by changing when and how much data they write, for example) according to current load [62, 64, 67, 72] or I/O-aware batch scheduling [11, 35, 49, 71]. With rare exceptions, these techniques rely on some assumptions about the workload characteristics. As a result, information about applications' temporal I/O behaviors is an important building block to allow for smart decision-making for I/O optimization.

In this paper, we verify such assumptions by extensively studying the temporal I/O behavior of HPC applications over a decade. While many efforts have described other I/O characteristics such as request size and used I/O API [9, 53, 61, 66, 77], the temporal behavior has been the focus of less work. Moreover, assumptions about it are often based on experience (e.g., periodic writes can be due to checkpointing, which is something applications often need [22, 67]) and on knowledge from specific applications or domains [20, 23, 29, 79]. By studying large sets of traces from real HPC systems, we aim at characterizing the workload we can expect in practice and therefore provide valuable tools for anyone working to improve HPC I/O. For this analysis, we gather traces of I/O activity from four different systems, including both large and small scale systems and facilities, covering a diverse group of user communities. Through this in-depth study, we answer the following questions:

a) **When do applications perform I/O?** While work on resource allocation usually considers steady activity [10, 57, 58], which may waste resources [3, 67], efforts to overlap I/O with compute time (dedicated core to perform I/O operations [25], use of an ephemeral file system [73]) rely on operations being more sparse, with sufficient compute time in between. While answering this question, we also verify the common belief that HPC applications read at the beginning and write at the end of their execution.

b) **Are applications periodic?** Another assumption about HPC applications is that they perform I/O periodically. This motivates I/O improvement techniques where the behavior of an I/O phase is used to predict the behavior of the next ones [28, 42, 46, 68]. Information about the frequency of I/O phases has also been used for I/O scheduling [13].

c) **How long are I/O intervals?** We investigate for how long I/O activities usually last. This information is important because, in the absence of prior knowledge about applications' behaviors, many techniques were proposed to characterize them based on recent accesses [37, 44]. Such approaches require some observation time before being able to improve I/O performance, hence the importance of knowing the impact of the observation window length.

d) **Do applications always present the same temporal I/O behaviors?** An alternative to approaches where recent accesses are observed in order to make predictions about an application's behavior is to rely on knowledge obtained from previous executions of the same application [10, 24, 78]. Answering this question is therefore of paramount importance to evaluate
the viability of such proposals.

e) Are there usage behavior patterns for users and projects? They are widely assumed to exist [78,81]. We study the “campaign duration” of users and projects/accounts and categorize typical usage patterns. Answering these questions about I/O interactions helps the optimization of data staging and hierarchical management [22,23].

f) How common are concurrent accesses to the shared I/O infrastructure? Our goal with this is twofold: first, we look for evidence of the global I/O performance being degraded due to contention, which motivated the proposal of many contention-avoidance techniques [19,36]. Secondly, we also quantify how often these concurrent accesses actually happen, and hence how much benefit can be obtained from a technique such as I/O scheduling.

After answering these questions, we propose a *general classification of temporal application I/O behaviors*. We quantify the prevalence of different classes, which allows for the creation of realistic workloads for evaluation of new proposed techniques. Moreover, this classification provides insights for the design of new I/O improvement techniques. Our contributions also include *publishing two large datasets* of I/O traces that allow for further analysis on temporal I/O behaviors. Indeed, the fact that such datasets are extremely rare is one of the reasons why temporal I/O behavior has not been studied so much before.

This paper is organized as follows. After presenting the datasets in Section 2, we answer the questions a) to f), listed above, in Sections 3 to 8. In Section 9 we present our classification of temporal I/O behaviors. Related work is discussed in Section 10, followed by final remarks in Section 11.

## 2 Datasets of job I/O activity

In this paper, we extract insights about temporal I/O behaviors observed in practice by HPC systems. Therefore, we base our study on datasets of job I/O activity obtained from real systems. To focus on realistic workloads, we gathered traces from jobs running over a period of time instead of profiling a set of chosen applications ourselves. Furthermore, we study traces from diverse systems, covering different scales, facilities, user communities and parallel file systems. Our datasets, listed below, cover different periods of time over 11 years. Finally, we have two types of traces: system-side, obtained from the parallel file system, and application-side, from the Darshan profiling tool [18,54]. Each type has its own limitations and benefits, hence, having the two makes our analyses more complete and robust. The file systems traces, collected by us, have been made available [14] together with the code we developed for the analysis. This allows for reproducibility and extensions of our results.

a) PlaFRIM is an experimental platform from Inria Center at the University of Bordeaux. It has 192 nodes and a BeeGFS with two OSS, each with four OST, default stripe count of 4, and a 100 Gbps network. Its peak I/O performance is of ≈12 GiB/s. We collected data over a period of 26 months (from May 2022 to July 2024). The `beegfs-ctl` command was used to obtain bandwidth (grouped by user) every second.

b) SDumont is located at the National Laboratory for Scientific Computing (LNCC), in Brazil, has 36,472 cores on 1,134 nodes, and is capable of 5.1 petaflops. The main research fields of the applications running on SDumont are Chemistry, Physics, Engineering, Biological Sciences, and Computer Science. Its Lustre is deployed on 10 OSS, each with one OST. It has a default stripe count of 1 and a peak performance of 30 GiB/s. `collectl` was used every 15 seconds from each compute node to obtain information from the PFS, from January 2020 to December 2020 (12 months). The workload of this machine has been studied by Carneiro et al. [15] and Bez et al. [8]
<table><caption>Table 1: Number of jobs before and after the different filters for file system traces</caption><thead><tr><th></th><th>Jobs</th><th>Concurrent from the same user</th><th>Concurrent on the same nodes</th><th>Do not use the PFS</th><th>Not completed batch jobs</th></tr></thead><tbody><tr><th>PlaFRIM</th><td>1,434,330</td><td>163,400</td><td>-</td><td>200,273</td><td>470,950</td></tr><tr><th>SDumont</th><td>380,840</td><td>-</td><td>2,191</td><td>141,193</td><td>116,182</td></tr></tbody></table><table><thead><tr><th></th><th>In a blind spot</th><th>Too short</th><th>I/O benchmarks</th><th>Final number of jobs</th></tr></thead><tbody><tr><th>PlaFRIM</th><td>1,832</td><td>486,547</td><td>94,854</td><td>16,474</td></tr><tr><th>SDumont</th><td>-</td><td>55,560</td><td>-</td><td>65,714</td></tr></tbody></table>

c) **Intrepid** was a system from the Argonne National Laboratory, composed of 40,960 nodes. It offered two parallel file systems, PVFS and GPFS, sharing the same 128 file servers [47]. We obtained one year (2013) of Darshan traces, which were downloaded a few years ago from the ALCF I/O Data Repository [16], when they were still publicly available online. This I/O infrastructure has been studied by Lang et al. [47], and two months of traces from 2010 were studied by Carns et al. [17] (however, not regarding temporal behavior).

d) **Blue Waters** was a 13.3 petaflops system in Illinois, USA, decommissioned in 2021. It featured more than 26,000 compute nodes. The scratch Lustre partition was distributed across 360 OSSs and 1440 OSTs. The report by Jones et al. [43] studied general I/O characteristics in the machine. For this work, we used publicly available Darshan traces from 2019 [1], the peak year for machine usage. A preliminary analysis of this data was conducted by Jolivel et al. [42]

## 2.1 Filters and final dataset sizes

To answer most of our questions, we are interested in observing the I/O behavior of HPC applications. For this reason, in the file system traces, we excluded all jobs that did not access the PFS or that did not correspond to a full application execution: jobs that completed unsuccessfully, interactive jobs, and jobs that were impacted by temporary issues with the monitoring tool (the “blind spots”). Moreover, due to the granularity of monitored data, we could not obtain traces from concurrent jobs by the same user (in PlaFRIM) and sharing compute nodes (in SDumont). Additionally, we excluded from PlaFRIM jobs from a large campaign of I/O benchmark executions. Finally, we only considered jobs that executed for at least a certain time (4 seconds in PlaFRIM and 60 seconds in SDumont), because otherwise the time series were not long enough for any meaningful analysis. Table 1 presents the number of jobs filtered out at each step. We are left with 16,474 jobs for PlaFRIM and 65,714 for SDumont.

For the Darshan traces, we only filtered out corrupted Darshan files and jobs that did not access the PFS. 42,066 jobs were left for Intrepid. For Blue Waters, 316,399 Darshan traces remained, which actually correspond to a smaller number of individual jobs. In the rest of this paper, for simplicity, we refer to these traces as “jobs”.

## 2.2 General characteristics of the datasets

In this section, we present a preliminary analysis of the datasets before using them to answer the questions we identified. Table 2 presents general statistics about the studied jobs. All presented metrics are from the datasets obtained after applying all previously described filters.
<table><caption>Table 2: Job statistics for the four studied data sets</caption><thead><tr><th rowspan="2">PlaFRIM</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>run time (minutes)</td><td>0.7</td><td>2.8</td><td>20</td><td>87</td><td>770</td><td>45</td></tr><tr><td># compute nodes</td><td>1</td><td>1</td><td>4</td><td>5</td><td>18</td><td>2.8</td></tr><tr><td>amount of data (MiB)</td><td>1</td><td>785</td><td>7,168</td><td>27,195</td><td>1 TiB</td><td>62,612</td></tr><tr><td>peak bandwidth (MiB/s)</td><td>0.4</td><td>78</td><td>513</td><td>1,462</td><td>5,538</td><td>9,828</td></tr><tr><td>fraction of time spent on I/O</td><td>0.08</td><td>0.25</td><td>0.65</td><td>0.95</td><td>1</td><td>0.38</td></tr></tbody></table><table><thead><tr><th rowspan="2">SDumont</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>run time (minutes)</td><td>2.2</td><td>8</td><td>46</td><td>557</td><td>2,759</td><td>210</td></tr><tr><td># compute nodes</td><td>1</td><td>1</td><td>3</td><td>6</td><td>24</td><td>3</td></tr><tr><td>amount of data (MiB)</td><td>0.2</td><td>2</td><td>101</td><td>1,045</td><td>116 GiB</td><td>15,502</td></tr><tr><td>peak bandwidth (MiB/s)</td><td>0.004</td><td>0.04</td><td>2</td><td>9</td><td>217</td><td>87</td></tr><tr><td>fraction of time spent on I/O</td><td>0.12</td><td>0.37</td><td>0.75</td><td>0.97</td><td>1</td><td>0.44</td></tr></tbody></table><table><thead><tr><th rowspan="2">Intrepid</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>run time (minutes)</td><td>9.3</td><td>27</td><td>81</td><td>332</td><td>701</td><td>99</td></tr><tr><td># compute nodes</td><td>512</td><td>512</td><td>1,024</td><td>4,096</td><td>8,192</td><td>1,171</td></tr><tr><td>amount of data (MiB)</td><td>3.6</td><td>116</td><td>5,516</td><td>39,693</td><td>1.2 TiB</td><td>71,861</td></tr></tbody></table><table><thead><tr><th rowspan="2">Blue Waters</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>run time (minutes)</td><td>2</td><td>8</td><td>46</td><td>145</td><td>1,409</td><td>88</td></tr><tr><td># compute nodes</td><td>4</td><td>16</td><td>80</td><td>123</td><td>512</td><td>57</td></tr><tr><td>amount of data (MiB)</td><td>98</td><td>606</td><td>3,976</td><td>8,652</td><td>315 GiB</td><td>35,635</td></tr></tbody></table>
For Darshan traces (Intrepid and Blue Waters), we have aggregated counters of I/O time per accessed file (some per rank and some for all the ranks), thus the application I/O time cannot be properly estimated. For this reason, we decided not to compute the fraction of time spent on I/O for these jobs.

We can see that SDumont jobs present in general low I/O performance (mean of 87 MiB/s), significantly below the system's theoretical peak. They access less data (mean of 15 GiB) than jobs in all three other systems, and that could both be a reason and a consequence of low I/O bandwidth. Compared to all the others, Intrepid jobs, which executed almost a decade earlier, are the longest (on average 99 minutes), largest (on average 1,171 nodes), and the ones that access the most data.

It is important to notice that, in both datasets of file system traces (PlaFRIM and SDumont), the amount of time spent on I/O is somewhat overestimated because of the resolution of the trace collection: more so on SDumont, with measurements every 15 seconds, than on PlaFRIM, with measurements every 1 second. Nonetheless, the fact that both data sets present somewhat similar values for time spent on I/O indicates that this estimation is not that far from reality, or at least that short I/O tends to happen repeatedly in relatively long periods of time. We further study such I/O intervals in Section 5.

## 3 When do applications perform I/O?

The general perception of HPC applications is that they often read at the beginning of their execution, then write periodically throughout the run and/or at the end [27, 31]. The well-known exception is machine learning codes, which issue read operations during most of their executions [20, 29, 32]. In this section, we focus on the question of when I/O happens at a high level. Later, in Section 9, we further characterize temporal I/O behavior. The aspect regarding periodic I/O behavior is examined in Section 4.

We first break job executions into four equal-size segments, followed by classifying them into write (W), read (R), both (B), or no I/O (O). To mitigate noise from small operations, we discard segments with peak bandwidth below 1 MiB/s and then filter out all 0000 jobs. Additionally, to avoid excessively breaking I/O intervals into segments, we filter jobs shorter than 12 seconds (3 seconds per segment, based on median I/O interval duration as shown in Section 5). For SDumont, we actually filter all jobs shorter than 180 seconds, so they have at least three points per segment. After the filters, we classify 10,916 jobs from PlaFRIM, 18,231 from SDumont, 40,519 from Intrepid, and 172,452 from Blue Waters.

Figure 1 shows the temporal behaviors that were most common in the datasets. We further discuss them below and summarize their prevalence in Table 3. For Intrepid and Blue Waters,

<table><caption>Table 3: When do applications perform I/O?</caption><thead><tr><th rowspan="2"></th><th colspan="3">read</th><th colspan="3">write</th></tr><tr><th>beginning</th><th>throughout</th><th>others</th><th>end</th><th>throughout</th><th>others</th></tr></thead><tbody><tr><td>PlaFRIM</td><td>32%</td><td>39%</td><td>29%</td><td>7%</td><td>34%</td><td>59%</td></tr><tr><td>SDumont</td><td>36%</td><td>10%</td><td>54%</td><td>15%</td><td>20%</td><td>65%</td></tr><tr><td>Intrepid</td><td>79%</td><td>17%</td><td>4%</td><td>15%</td><td>24%</td><td>61%</td></tr><tr><td>Intrepid-large</td><td>75%</td><td>23%</td><td>2%</td><td>3%</td><td>95%</td><td>2%</td></tr><tr><td>Blue Waters</td><td>69%</td><td>2%</td><td>29%</td><td>37%</td><td>21%</td><td>42%</td></tr><tr><td>BW-large</td><td>68%</td><td>5%</td><td>27%</td><td>10%</td><td>76%</td><td>14%</td></tr></tbody></table>
<figure><img src="image_3.png" /><figcaption>Figure 1: Temporal I/O behavior. “Others” combine all the classes that account for less than 3% of the jobs.</figcaption></figure>
we add smaller datasets composed of the 516 and 1095 (respectively) longest and largest jobs. They were created by taking the jobs with execution time and number of compute nodes higher than the 90% quartiles (332 and 145 minutes, 4,096 and 123 nodes).

**Read at the beginning:** This behavior is indeed very common by the prevalence of classes such as `R000`, `R00W`, `B000`, `B00B`, among others. Most jobs read at the beginning of their execution (97% in Intrepid, 86% in PlaFRIM, 78% in Blue Waters and 51% in SDumont). On the other hand, a smaller portion read *only* at this moment, with the exception of the `-large` datasets (see Table 3).

**Read throughout the execution:** If we count all classes where there are at least 3 segments with read operations (R or B), they account for more jobs (39%) in PlaFRIM than in all others (up to 23%).

**Write at the end** Although approximately 40% of the jobs in PlaFRIM, SDumont and Intrepid (96% in Blue Waters) write at the end, only between 7% and 37% write *only* at the end. This behavior is even more rare among the largest jobs. Moreover, the anticipated `R00W` class only accounts between 1% and 20% of the jobs (with `R0WW` accounting for less than 3% on all machines). Interestingly, Blue Waters has many jobs with I/O at the beginning and at the end only, but those include writes in the first quarter (`B00B`, `B00W`, etc).

**Write throughout the execution** Between 20% and 34% of the jobs have writes (W or B) in at least three segments. Again, the “stereotype” for HPC applications represented by the `RWWW`, `BWWW`, `RRWW`, and `RBWW` classes account together for a small portion of jobs: up to 8% (in Intrepid). Nonetheless, when considering only the largest jobs, these classes account for 65% on Intrepid and 46% on Blue Waters.

When do applications perform I/O?

Most applications read at the beginning of their execution. However, that is the *only* moment most read only for the oldest system and among the largest jobs. Indeed, the widely accepted description of HPC applications — reading at the beginning and then writing throughout and/or at the end of their execution — does not seem to be that common among shorter jobs, which can write anytime. Finally, in practice a large number of jobs is expected to not fall into any of the described behaviors. Therefore, system-wide I/O improvement techniques should avoid relying on assumptions about when write operations happen.

# 4 Are applications periodic?

Due to common patterns in HPC, like checkpointing, many applications perform their I/O periodically. In this section, we examine this statement closer. For that, we used the existing tool FTIO [68], which captures periodic I/O phases. It utilizes popular techniques from the signal processing domain to extract the frequency of the I/O phases. In particular, it combines the Discrete Fourier transformation (DFT) with outlier detection methods (e.g., Z-score, DBScan, etc.) to find the dominant frequency candidates in the signal. Depending on the number of candidates found, the tool reveals if the I/O phases are periodic or not. For our purposes, we extended FTIO to accept the file system traces (CSV files) as input. We also used its API to develop a parallel extension that processes an entire folder containing traces from different runs. Besides bringing great speedups and completely automating the analysis, this extension merges the results into a single CSV file that can be later analyzed. For example, the execution for all 16,474 PlaFRIM traces takes 120.96 seconds with 30 processes on a PlaFRIM (Bora) node. For Intrepid the tool consumed 431.65 sec for the 42,066 jobs, 1313.0285 sec for the 65,714 jobs from SDumont, and 11592.4869 sec for the 316,399 from Blue Waters.
Darshan traces contain aggregated counters per accessed file. To estimate a time series for them, we use the timestamps of first and last read/write to each file. Therefore, we filtered out jobs that accessed a single file, as they would be seen as having a single I/O phase, leaving 41,013 and 246,699 jobs from Intrepid and Blue Waters in this section. As in Section 3, we create the Intrepid-large and BW-large datasets of size 519 and 1,077, respectively.

Upon inspecting many jobs, we noticed varying values for the confidence metric. This metric gauges the confidence in the obtained results, but does not allow us to distinguish between periodic and non-periodic behavior. Thus, most jobs for which a dominant frequency was detected had a high confidence value (over 0.5). Initially we consider the I/O phases periodic for all jobs for which FTIO could find a dominant frequency. Hence, we further limit this definition to include jobs for which FTIO detects at least three phases, so it better fits the notion of “periodic I/O”. Table 4 summarizes our findings. We can see the periodic jobs with more than two I/O phases are the minority, between 7 and 26% of the studied jobs. Nonetheless, they are more common among larger jobs (between 31 and 52%).

<table><caption>Table 4: Proportion of jobs performing periodic I/O</caption><thead><tr><th></th><th></th><th>Periodic read</th><th>Periodic write</th><th>Periodic read AND write</th><th>Total periodic</th></tr></thead><tbody><tr><td rowspan="4"></td><td>PlaFRIM</td><td>43.3%</td><td>40.6%</td><td>27.4%</td><td>56.5%</td></tr><tr><td>SDumont</td><td>13.5%</td><td>30.7%</td><td>9.4%</td><td>34.8%</td></tr><tr><td>Intrepid</td><td>34.3%</td><td>24.5%</td><td>17.4%</td><td>41.4%</td></tr><tr><td>Blue Waters</td><td>32.9%</td><td>38.1%</td><td>17.4%</td><td>53.6%</td></tr><tr><td rowspan="6">&gt;2 phases</td><td>PlaFRIM</td><td>16.2%</td><td>16.5%</td><td>7.1%</td><td>25.6%</td></tr><tr><td>SDumont</td><td>5.1%</td><td>13.8%</td><td>1.6%</td><td>17.3%</td></tr><tr><td>Intrepid</td><td>1.8%</td><td>5.8%</td><td>0.3%</td><td>7.3%</td></tr><tr><td>Intrepid-large</td><td>14.6%</td><td>17.1%</td><td>0.8%</td><td>30.9%</td></tr><tr><td>Blue Waters</td><td>6.8%</td><td>3.4%</td><td>0.5%</td><td>9.7%</td></tr><tr><td>BW-large</td><td>3.1%</td><td>50.8%</td><td>2.3%</td><td>51.6%</td></tr></tbody></table>

Table 5 shows the percentage of resources consumed by the periodic jobs. As mentioned in Section 2, because Darshan traces contain only aggregated counters per file, we cannot properly estimate their I/O time. We can see periodic jobs were responsible for over a third of the total read/write time in PlaFRIM and SDumont. Jobs that periodically write account for a quarter to a third of the node-hours in these systems. They also account for over a third of the total amount of written data in all systems.

It is important to notice that the numbers in Table 4 are an underestimation. First, PlaFRIM and SDumont traces are limited by their resolution: by Nyquist, any frequency higher than 0.5 and 0.033 Hz, respectively, will appear as constant I/O activity. Second, in Darshan traces all periodic I/O involving a single file cannot be detected.

## 4.1 Common periods

Figure 2 shows the distribution of the detected periods (i.e. time between the start of consecutive I/O phases). Please note the “Less than 30s” category is not present in SDumont data because of its resolution (15s between points). We can see most periodic applications perform I/O every few minutes or more often. We did not find any single period that is particularly more common than the others (e.g. write every hour, etc.). The longer periods - over 30 minutes, appear more
<figure><img src="image_4.png" /><figcaption>Figure 2: Time between the start of consecutive I/O phases for jobs that perform I/O periodically</figcaption></figure>
<table><caption>Table 5: Resources consumed by jobs that perform more than two periodic read/write phases</caption><thead><tr><th></th><th></th><th>Node-hours</th><th>Read/write time</th><th>Amount of read/written data</th></tr></thead><tbody><tr><td rowspan="2">PlaFRIM</td><td>read</td><td>25.6%</td><td>44%</td><td>35.8%</td></tr><tr><td>write</td><td>26.4%</td><td>49.7%</td><td>42.5%</td></tr><tr><td rowspan="2">SDumont</td><td>read</td><td>17.1%</td><td>29.4%</td><td>13.5%</td></tr><tr><td>write</td><td>32.1%</td><td>34.9%</td><td>40%</td></tr><tr><td rowspan="2">Intrepid</td><td>read</td><td>6.3%</td><td>-</td><td>9.1%</td></tr><tr><td>write</td><td>15.8%</td><td>-</td><td>35.2%</td></tr><tr><td rowspan="2">Blue Waters</td><td>read</td><td>4.3%</td><td>-</td><td>4%</td></tr><tr><td>write</td><td>41.4%</td><td>-</td><td>50.1%</td></tr></tbody></table>

often among the large jobs.

## 4.2 Improving performance for periodic applications only

Many I/O scheduling algorithms model applications as periodic [13, 31]. Such techniques will improve performance by mitigating contention, but bad decisions (due to non-periodic applications) may harm it. In this section, we estimate the potential savings that could be achieved by I/O improvement techniques that are able to improve performance for periodic applications, at the cost of a penalty for non-periodic ones. Our estimate is a simplification, because we consider each job independently. If, for each job j, $t_{io}(j)$ gives the original I/O time of the job and $t'_{io}(j)$ is the hypothetical I/O time after an optimization, we compute total savings in time as:

$$ savings = (\sum t_{io} - \sum t'_{io}) / \sum t_{io} \qquad (1) $$

Table 6 lists all information we use about jobs throughout this paper. For this estimate, we define

$$ t'_{io}(j) = \begin{cases} t_{io}(j) \times (1 - \text{improvement}) & \text{if j is periodic} \\ t_{io}(j) \times (1 + \text{penalty}) & \text{otherwise} \end{cases} \qquad (2) $$

Results are presented in Figure 3. The estimations we present in this paper are optimistic because we ignore the interaction between jobs (improving I/O performance may be impossible in a saturated system), so they provide upper bounds to what can be expected in practice. Moreover, we only compute such estimates for PlaFRIM and SDumont, because Darshan traces do not allow for accurately measuring total I/O time by the jobs.

We can see that, because periodic jobs are responsible for a large portion of the I/O activity, such techniques can still be beneficial as long as the penalty they impose to non-periodic jobs is not too high and/or the benefit to the periodic ones is high. To put these values into the system's perspective, the best savings in PlaFRIM translated into 13,923 node-hours, or 72 hours of the whole system in a period of 26 months, and in SDumont into 64.5 days in a period of 12 months.

### Node-hours savings

Because most jobs are *not* I/O intensive, improvements in I/O performance may *not* be highly impactful from the system's point of view. They should be justified by speeding up I/O-intensive applications.
<figure><img src="image_5.png" /><figcaption>Figure 3: Total read/write time savings (Eq. 1) potentially yielded by optimization techniques that are able to improve performance for periodic applications (x axis), but impose a penalty to non-periodic ones (the colors). See Eq. 2.</figcaption></figure>
<table><caption>Table 6: Information about each job j</caption><tr><td>t(j)</td><td>Total execution time</td></tr><tr><td>t<sub>io</sub>(j)</td><td>Total time spent on I/O operations</td></tr><tr><td>n(j)</td><td>Number of periodic I/O phases (given by FTIO)</td></tr><tr><td>t<sub>p</sub>(j)</td><td>Time of each periodic I/O phase. Computed from t<sub>io</sub>(j)/n(j)</td></tr><tr><td>t'<sub>int</sub>(i)</td><td>Duration of I/O interval i. Note that t<sub>io</sub>(j) = &Sigma; t<sup>j</sup><sub>int</sub>(i)</td></tr><tr><td>t<sup>j</sup><sub>comp</sub>(i)</td><td>For interval i, the time since the previous (for reads) or until the next (for writes) I/O interval</td></tr></table>

## 4.3 Improving performance after observing some I/O phases

We now estimate the impact of techniques that can improve I/O performance after observing a number of I/O phases of the job. Predicting future I/O phases is useful, for example, for managing buffers and adapting I/O behaviors (e.g., amount of data) to avoid contention [62]. To estimate it, we compute the time savings as per Eq. 1, assuming such techniques are only effective for periodic applications and that they need to observe N phases before being able to improve the next ones. If job j is periodic, we consider it has n(j) I/O phases of duration t<sub>p</sub>(j) = t<sub>io</sub>(j)/n(j) (we obtain n(j) from FTIO). Thus, we define t'<sub>io</sub>(j) as follows:

$$t'_{io}(j) = \begin{cases} N \times t_p(j) \times \tau + (n(j) - N) \times t_p(j) \times \gamma & \text{if } j \text{ is periodic and } n(j) > N \\ t_{io}(j) \times \tau & \text{otherwise} \end{cases} \quad (3)$$

With:

$$\begin{aligned} \tau &= 1 + \textit{overhead} \\ \gamma &= 1 - \textit{improvement} \end{aligned} \quad (4)$$

For this we chose an overhead of 10%, a reasonable value based on related work that predicts future phases/accesses using grammars [27], DFT [68], etc., and on I/O profiling tools [17]. Figure 4 shows the total savings that could be obtained. We can see that these techniques can be beneficial, even if they need to observe multiple phases to characterize applications, as long as they are able to improve performance for future phases by at least ≈30%.

### Are applications periodic?

Despite it being widely accepted that HPC applications perform I/O periodically, we found that these are actually a minority of them (less than 26%). However, they account for a larger portion of the I/O time and node-hours. Therefore, it can still be worthy to propose I/O improvement techniques that are only good for periodic applications, even if they need to observe some I/O phases before starting to yield improvements. Still, it is important for such techniques to consider other patterns of I/O are the rule, not the exception.

## 5 How long are I/O intervals?

In this section, we look into I/O intervals (i.e. periods of time when jobs are doing I/O). Specifically, we are interested in their length, because it indicates how fast techniques that try to
<figure><img src="image_6.png" /><figcaption>Figure 4: Total read/write time savings (Eq. 1) potentially yielded by optimization techniques that are able to improve performance for I/O phases of periodic applications (x axis) after observing a number of phases (the colors). See Eq. 3.</figcaption></figure>
dynamically adapt to the current load must be in order to be useful. For each job, we measure periods of uninterrupted I/O activity. We focus on PlaFRIM traces only, because they are the ones with the best resolution. Darshan traces would overestimate I/O intervals length due to the way their temporal behavior is estimated from aggregated counters, as previously discussed, and in SDumont the minimum interval length would be 15 seconds.

<table><caption>Table 7: Distribution of I/O intervals length and number per job on PlaFRIM</caption><thead><tr><th rowspan="2"></th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><th>I/O interval length (s)</th><td>2</td><td>2</td><td>5</td><td>13</td><td>217</td><td>13.8</td></tr><tr><th>Read interval length (s)</th><td>2</td><td>3</td><td>6</td><td>17</td><td>207</td><td>14.7</td></tr><tr><th>Write interval length (s)</th><td>2</td><td>2</td><td>4</td><td>8</td><td>92</td><td>8.1</td></tr><tr><th>I/O intervals per job</th><td>1</td><td>4</td><td>13</td><td>67</td><td>1091</td><td>62.4</td></tr><tr><th>Read intervals per job</th><td>1</td><td>3</td><td>10</td><td>51</td><td>657</td><td>43.8</td></tr><tr><th>Write intervals per job</th><td>2</td><td>5</td><td>27</td><td>100</td><td>1564</td><td>91.9</td></tr></tbody></table>

Table 7 shows the I/O intervals length and number per job found in PlaFRIM traces. We found that most intervals (99.9%) last for more than one second, but only 8.9% last for 15 seconds or more. The median duration is 2 seconds. Furthermore, most jobs (70.5%) present at least two intervals, and 44.3% present at least four. Compared to read intervals, write intervals are in general shorter (average of 8 vs. 15 seconds) but jobs have more of them (on average 92 vs. 44). A similar phenomenon regarding the number of phases was observed in SDumont traces: jobs there had on average 59 write phases of 106 seconds, and 12 read phases of 66 seconds.

## 5.1 Improving performance after observing recent accesses

In Section 4.3, we studied the impact of techniques that observe the application for a few I/O phases to improve performance for future I/O phases if they have similar behavior. Here, we conduct a similar analysis, but our new hypothetical technique does not focus on I/O phases, but rather observes a window of recent accesses in order to improve performance for the future ones. Detecting the current behavior is often used for auto-tuning strategies, for example [5]. We assume this will only be effective inside each I/O interval, since future I/O intervals do not necessarily present the same I/O behaviors. Moreover, we assume again a 10% overhead during the observation period.

For job $j$, we have $t_{int}^j(i)$ the duration of each I/O interval $i$ (and hence $t_{io}(j) = \sum t_{int}^j(i)$). For interval $i$, we compute its new duration $t_{int}^{j'}(i)$ as:

$$t_{int}^{j'}(i) = \begin{cases} s \times \tau + (t_{int}^{j}(i) - s) \times \gamma & \text{if } t_{int}^{j}(i) > s \\ t_{int}^{j}(i) \times \tau & \text{otherwise} \end{cases} \quad (5)$$

With the observation time $s$, the overhead during observation, and the improvement after
observation. These last two are again represented by $\tau$ and $\gamma$ as per Eq. 4. Then we have:

$$t'_{io}(j) = \sum t'_{int}(i) \qquad (6)$$

The total savings are estimated using Eq. 1. Figure 5 shows the results of this analysis. We can see even observation periods of a few seconds can bring savings in read and write time. However, an observation window of 4s, for example, can improve performance for over 40% of the jobs while decreasing it for over 50% of them. The ideal window length seems to be between 1 and 2 seconds.

## 5.2 Potential for asynchronous I/O

We now investigate the potential for I/O operations to be made asynchronously. That can be done, for example, by using node-local devices for temporary storage, and by dedicating resources into either prefetching data from the PFS or flushing to it. For this study, we consider that operations can be done asynchronously if there is enough time without I/O operations until the next (write) or since the previous (read) I/O operation, and we allow operations to be partially asynchronous. This is therefore a pessimistic estimation, since multiple operations could be prefetched/flushed at the same time if capacity (temporary storage and/or available PFS bandwidth) allows.

For each interval $i$ of job $j$, of duration $t'_{int}(i)$, we define $t'_{comp}(i)$ as the time since the previous I/O interval (for reads) or until the next one (for writes). Then, we compute its potentially asynchronous time as:

$$t'_{int,async}(i) = \min(t'_{comp}(i), t'_{int}(i)) \qquad (7)$$

We then estimate the I/O time from each job $j$ that can be made asynchronously as $t_{async}(j) = \sum t'_{int,async}$, and report the total fraction of potentially asynchronous I/O time, obtained from:

$$AsyncRatio = \sum t_{async} / \sum t_{io} \qquad (8)$$

Results are presented in Table 8. This analysis is, of course, a simplification because some applications could be negatively impacted by dedicating resources for asynchronous I/O. Still, it shows there is great potential for operations to be done asynchronously, especially for writes.

<table><caption>Table 8: Study of the potential for asynchronous I/O</caption><thead><tr><th></th><th>AsyncRatio<br>(Eq. 8)</th><th>#Jobs with >90%<br>asynchronous I/O</th><th>#Jobs with >99%<br>asynchronous I/O</th></tr></thead><tbody><tr><td>read</td><td>23.5%</td><td>41%</td><td>36.1%</td></tr><tr><td>write</td><td>74.5%</td><td>22.9%</td><td>12.7%</td></tr></tbody></table>

### How long are I/O intervals?

Most jobs present a few — at least two — separate time intervals with I/O activity, each lasting for a few seconds. As a result, I/O behavior can be observed for a certain time, followed by taking actions to adapt the system and improve performance for future accesses. Nonetheless, such approaches must work on a short window, ideally under ≈1.5 seconds. We also observed that the intervals are often separated enough that I/O operations could be done asynchronously, especially for particularly (74% of the time).
<figure><img src="image_7.png" /><figcaption>Figure 5: Impact of techniques that improve performance after observing the job for a certain time (x axes) depending on the expected improvement on future accesses (the colors). Generated with the PlaFRIM data set. The impact on each job is estimated from Eq. 6, and total savings from Eq. 1</figcaption></figure>
## 6 Do applications always present the same temporal I/O behaviors?

An alternative to gathering information about applications by observing them for a few I/O phases (Section 4.3) or for some time (Section 5.1) is to use information about previous runs of the same application. For example, profiles obtained during previous executions are often used for resource management [10, 49]. In this section, we investigate at what extent runs from the same application present similar temporal I/O behaviors. For that, we use the Darshan traces (Intrepid and Blue Waters), which include the information of the executable. We have access to the binary name with the arguments used at runtime. However, this technique has limitations as it ignores other parameters, like environment variables for instance, and may lead to an overestimation of identical executions.

**Definition 1.** we consider multiple traces to come from the same application *a* if they are from the same user, have the same executable name and arguments. We found 9,932 individual applications for Intrepid and 14,622 for Blue Waters.

<table><caption>Table 9: Statistics for applications that were executed more than once — Intrepid</caption><thead><tr><th rowspan="2">Intrepid</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>#runs for repeating apps</td><td>2</td><td>2</td><td>5</td><td>12</td><td>107</td><td>23.7</td></tr><tr><td>#runs for repeating apps at same scale (80%)</td><td>2</td><td>2</td><td>4</td><td>8</td><td>76</td><td>6.8</td></tr><tr><td>#runs for repeating apps at different scale (20%)</td><td>3</td><td>5</td><td>12</td><td>52</td><td>518</td><td>91.4</td></tr><tr><td>Difference in amount of data for repeating apps</td><td>22 KiB</td><td>1 GiB</td><td>7 GiB</td><td>212 GiB</td><td>3 TiB</td><td>180 GiB</td></tr><tr><td>runtime difference (max/min) for repeating apps</td><td>1.08</td><td>1.38</td><td>3.4</td><td>76</td><td>5,075</td><td>343</td></tr><tr><td>runtime difference (max/min) for repeating apps at same scale (80%)</td><td>1.05</td><td>1.29</td><td>2</td><td>14</td><td>2,964</td><td>141</td></tr><tr><td>runtime difference (max/min) for repeating apps at same amount of data (17%)</td><td>1</td><td>1.05</td><td>1.77</td><td>4</td><td>71</td><td>7</td></tr><tr><td>number of classes of temporal I/O behavior, for repeating apps</td><td>1</td><td>1</td><td>2</td><td>2</td><td>6</td><td>1.4</td></tr></tbody></table>
<table><caption>Table 10: Statistics for applications that were executed more than once — Blue Waters</caption><thead><tr><th rowspan="2">Blue Waters</th><th colspan="5">percentile</th><th rowspan="2">mean</th></tr><tr><th>25%</th><th>50%</th><th>75%</th><th>90%</th><th>99%</th></tr></thead><tbody><tr><td>#runs for repeating apps</td><td>2</td><td>5</td><td>22</td><td>65</td><td>1881</td><td>134</td></tr><tr><td>Difference in amount of data for repeating apps</td><td>1.8 KiB</td><td>52 KiB</td><td>803 MiB</td><td>82 GiB</td><td>12 TiB</td><td>1 TiB</td></tr><tr><td>runtime difference (max/min) for repeating apps</td><td>1.16</td><td>2</td><td>9</td><td>289</td><td>84,657</td><td>2,419</td></tr><tr><td>runtime difference (max/min) for repeating apps at same amount of data (18%)</td><td>1</td><td>1.07</td><td>2.26</td><td>14</td><td>4,104</td><td>457</td></tr><tr><td>number of classes of temporal I/O behavior, for repeating apps</td><td>1</td><td>1</td><td>1</td><td>1</td><td>5</td><td>1.18</td></tr></tbody></table>

Tables 9 and 10 present the obtained data. We found that only 3.3% of Intrepid and 0.7% of Blue Waters individual applications were found to execute more than once. Nonetheless, they are responsible for most of the jobs: 80% and 96%, respectively. Repeating applications were executed on average 24 times on Intrepid (median of 2) and 134 times on Blue Waters (median of 5). Most of these runs (80% and 100%, respectively) were at the same scale.

Despite the large proportion of applications that always run at the same scale, only 17% and 18% were found to always access the same amount of data. For the others, the median difference was of 1 GiB (average of 180 GiB) in Intrepid and 52 KiB (average of 1 TiB) in Blue Waters. Furthermore, most (73% for Intrepid and 91% for Blue Waters) applications that executed more than once always presented the same class of temporal I/O behavior (as defined in Section 3, Table 3).

An interesting thing to notice from Tables 9 and 10 is the variability observed in execution time. Since I/O time is typically not the largest portion of execution time, that indicates an even higher I/O performance variability. The longest run time of each repeating application, divided by the shortest one, results in a median of 1.38 (average of 343) times for Intrepid and 2 (average of 2,419) for Blue Waters. Although less dramatic (average of 7 and 457, respectively), that variability is present even among runs where the same amount of data was accessed. Between those, 20% and 32% have runs that took at least twice the time of others, and 12% and 20% have a maximum difference of at least 3 times.

## I/O performance variability

We found high I/O performance variability across application runs, even when handling the same amount of data. For between 20 and 32% of them, the slowest execution took at least twice the time of the fastest one.

We now estimate the impact of a hypothetical technique that gathers information from applications at their first run (imposing an overhead) and is then able to improve performance of the next runs, as long as they have similar behavior.
<figure><img src="image_8.png" /><figcaption>Figure 6: Results achievable by techniques that use information from a previous run to improve performance in the next ones</figcaption></figure>
**Definition 2.** two jobs of the same application *a* have similar behavior if they have the same scale, same temporal I/O behavior (Table 3), and the amounts of data they access do not differ by more than 20%.

For this analysis, we can not rely on I/O time, as we did in the previous sections, because the Blue Waters and Intrepid traces do not have a good estimate for total I/O time. Therefore, we use the total execution time instead, and lower the values we test for overhead and improvement. For job *j* of duration *t*(*j*), corresponding to an execution of an unique application *a*, we compute its new duration *t*′(*j*), impacted by the I/O improvement technique, as follows. We use *τ* (for the overhead) and *γ* (for the improvement) as previously defined (Eq. 4), and consider all jobs for the analysis (including the applications that were executed only once).

$$t'(j) = \begin{cases} t(j) \times \gamma, & \text{if } j \text{ has similar behavior to the previous job of } a, \\ t(j) \times \tau, & \text{if } j \text{ has different behavior or is the first job of } a. \end{cases} \tag{9}$$

Similarly to what was done in Eq. 1, we compute fraction of time saved as:

$$savings = (\sum t - \sum t') / \sum t \tag{10}$$

Results are shown in Figure 6, where the overhead is represented by the colors, and the improvement by the values in the x axes. We can see improvements for up to 80% of the jobs are possible at a cost for at least 20%.

Do applications always present the same temporal I/O behaviors?

Most jobs are runs of the same applications. Repeating applications tend to present similar temporal I/O behaviors across executions (over 70% of the cases). A consequence is that techniques that use information from previous runs to improve future ones seem to have good potential. However, their success depends on a low overhead.

# 7 Are there usage behavior patterns for users and projects?

Behavior patterns are widely assumed to exist in the workflows of users across resources and in time. In this section, we use the Blue Waters dataset to investigate it. These jobs come from 247 users across 124 groups/projects accounts. Blue Waters reports both groups and accounts which largely coincide. This was unexpected as accounts on production systems often correspond to specific projects allowing users to contribute to multiple projects.

Figure 7 shows the lifetime of users and projects. More than half of the users spent less than 53 days using the system, and 75% less than 189 days. Groups and accounts tend to be active for longer, at 50% of groups being active less than 107 days and 75% for less than 260 days.

We classify users and groups temporal behaviors, following a similar schema as outlined in Section 3: we reduce and normalize the read and write activity of each user or group into a 2x4 matrix with the top row encoding read activity over time and the bottom row write activity. We identified five patterns, which are shown in Figures 8 and 9. Table 11 reports their frequency.

The two most common patterns show many users and projects predominantly read or write data, which aligns well with the idea of data producers and consumers. We further define two mixed patterns and distinguish where read and write activity are balanced and coincide in time, and where activity appears to be more randomly distributed over time. Finally, we notice a
<figure><img src="image_9.png" /><figcaption>Figure 7: Distribution of days between first and last job for...</figcaption></figure>

<table><caption>Table 11: Prevalence of user and group patterns. Multiple patterns can be assigned to the same user/group.</caption><thead><tr><th rowspan="2">Pattern</th><th colspan="2">Total Users</th><th colspan="2">Total Groups</th></tr><tr><th colspan="2">(n=247)</th><th colspan="2">(n=124)</th></tr><tr><th></th><th></th><th>%</th><th></th><th>%</th></tr></thead><tbody><tr><td>Mostly Write</td><td>55</td><td>22.3</td><td>39</td><td>31.5</td></tr><tr><td>Mostly Read</td><td>48</td><td>19.4</td><td>23</td><td>18.5</td></tr><tr><td>Mixed R/W</td><td>21</td><td>08.5</td><td>18</td><td>14.5</td></tr><tr><td>Activity Increases</td><td>10</td><td>04.0</td><td>10</td><td>08.1</td></tr><tr><td>Balance R/W</td><td>9</td><td>03.6</td><td>16</td><td>12.9</td></tr><tr><td>No Activity</td><td>113</td><td>45.7</td><td>33</td><td>26.6</td></tr></tbody></table>

number of users and groups that increase their activity over time. Indeed, many teams tend scale up their experiments as the application matures (e.g. adding parallelization and middleware to take advantage of distributed architectures).

Are there usage behavior patterns for users and projects?

Users and group/patterns activity over time can be classified into a few patterns. It is interesting to notice a large portion predominantly read or write. These interactions give rise to I/O optimization opportunities that go beyond the I/O behavior in a particular run or job.

# 8 How common are concurrent accesses to the shared I/O infrastructure?

Several approaches to mitigate I/O interference are based on the assumption that parallel file systems are often accessed concurrently by multiple applications and that contention from these accesses impairs global I/O performance [12, 56, 59, 74, 80]. In this section, we investigate this assumption. We use the file system traces, since for Darshan traces we cannot say when exactly I/O operations happened. Moreover, we use all data, before any of the filters discussed in Section 2.1. In the case of PlaFRIM, since the monitoring tool reports activity per user, it covers all activity to the file system, including accesses from the frontend and through ssh.

First, we quantify how common parallel access to the shared file system is. In total, for PlaFRIM, we collected 6,124,892 data points (each point corresponds to one second). As shown in Figure 10, for 17% of the time, the system is completely idle. When it is not idle, the system is
<figure><img src="image_10.png" /><figcaption>(a) High-resolution I/O activity patterns of different users in comparison to the 4x2 reductions used for classification.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 8: Examples of user activity over time</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 9: Examples of group I/O activity over time</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Figure 10: Percentage of time concurrent users are accessing the shared file system in PlaFRIM.</figcaption></figure>

shared by only two users for most of the time (24%), followed by non-concurrent I/O operations (one user, 23% of the time). Moreover, less than 1% of the observed time involved more than seven users performing I/O.

In the case of SDumont, we collected 1,681,920 data points and used this number to compute the percentages of concurrent access according to the number of jobs. As shown in Figure 12, similar to PlaFRIM, 17% of the time the storage system of SDumont is idle, followed by 15% of the time with no concurrency (only one job) and 14% with two jobs sharing the file system. This tendency continues, with increasing concurrency representing smaller percentages of time. Since SDumont is a larger platform compared to PlaFRIM, and since we are counting jobs instead of users, we observed cases with up to 44 concurrent jobs. However, cases with more than 15 jobs represent less than 3.5% of the execution time in SDumont, which shows that even in larger machines, the degree of concurrency at the I/O level is not that common over time.

Next, we examine the impact on bandwidth when multiple users (or jobs) perform I/O simultaneously. For this, we look at the average global bandwidth observed in scenarios with different numbers of concurrent accesses. Because some numbers correspond to only a few data points, we made groups of six. Figs 11 and 13 present results for PlaFRIM and SDumont, respectively. As shown, for both platforms, the read and write operations exhibit completely distinct behaviors. For PlaFRIM, the average bandwidth of read operations increases with the number of users, while the write bandwidth remains nearly the same regardless of the degree of concurrency. We also observe in Fig. 11 that when the number of users performing read operations is between 13 and 15, there is a slight degradation, suggesting that for this system, the sweet spot for read is up to twelve users.

In the case of SDumont, we see an almost opposite scenario: the average read bandwidth drops by half (from 10,000 KB/s to less than 5,000 KB/s) when the number of concurrent jobs increases from 7-12 to 25-30, but it remains relatively stable for cases with more than 30 jobs. For the write operation, the average bandwidth increases with concurrency up to 19-24 jobs and then drops sharply. For instance, when the degree of parallelism is 43-48, the write bandwidth is significantly lower than when concurrent access is made by up to 24 jobs. These results show that concurrent access impacts some platforms more than others. It is worth noting that PlaFRIM's I/O infrastructure has a peak performance between a third and half of that of SDumont, for almost six times less nodes. Therefore, it can more comfortably absorb its load. These results evidence that contention from concurrent accesses does harm global I/O performance.
<figure><img src="image_14.png" /><figcaption>Figure 11: Average bandwidth in PlaFRIM for different degrees of concurrency</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 12: Percentage of time with I/O accesses by concurrent jobs in SDumont. Percentages smaller than 0.1% were suppressed from the graph to improve readability.</figcaption></figure>

Finally, we observed that global I/O performance surpassed 80% of the maximum observed performance (9.6 GiB/s and 24 GiB/s for PlaFRIM and SDumont, respectively) less than 1.7% of the time for PlaFRIM and even less for SDumont. Such behavior has been documented for larger systems as well [55, 78]. That means we can have performance degradation even when not reaching full system capacity.

## How common are concurrent accesses to the shared I/O infrastructure?

Although concurrent accesses are not that common in the studied systems, our results evidence that concurrent accesses can indeed harm global I/O performance, even when the imposed load is way below systems' capacity. That means other aspects than bandwidth sharing, such as network interference and access patterns interplays, should be the focus of interference mitigation techniques.
<figure><img src="image_16.png" /><figcaption>Figure 13: Average bandwidth in SDumont for different degrees of concurrency. Y axes do not start at zero.</figcaption></figure>

# 9 A classification of application temporal I/O behaviors

To complement the previous research questions, this section proposes a fine-grained classification of application temporal I/O behaviors through clustering of time series. Our goal is to identify common patterns exhibited by applications in practice. We selected SDumont for this analysis as it is both a large system and contains over 60,000 jobs. We only rely on file system traces as we manipulate time series and use I/O bandwidth as a classification criterion. From each job, we extract two dedicated time series: read and write. To remove noise and focus on key I/O activity, we further filter jobs with low peak I/O performance, below 1MiB/s, leaving 37% and 32% of all the read and write time series, respectively.

## 9.1 Methodology for clustering I/O time series

A challenge when comparing these time series and measuring similarity is that they have vastly different duration (from a few to $10^5$ seconds) and amplitudes (the peak I/O performance, which varies within seven orders of magnitude). Therefore, we decided to divide them into groups of similar duration and peak I/O performance. The “duration”, “peak I/O”, and “ratio” columns from Table 12 present these groups. The boundaries between categories were empirically defined and resulted in different values between reads and writes: we aimed both to gather time series with somewhat similar sizes and to avoid groups that are too large, as the resulting clustering cost might not be reasonable (more details below). Interestingly, job duration and peak I/O performance are not strongly correlated, as the whole range of I/O performance appears across all the possible execution times.

Jobs are represented by time series, ordered sets of values with a fixed time gap between samples. To compute a distance between couples of jobs, we use the state of the art method Dynamic Time Warping (DTW). We couple it with K-means clustering, both from the `tslearn` python package [70]. We sample 1,000 jobs per group and apply K-means using 10 clusters. We found that 1,000 is a large enough sample to characterize each group in reasonable time, while 10 clusters cover a large spectrum of behaviors. To further reduce the computation overhead, we also use the `tslearn TimeSeriesResampler` package [70] to reduce the size of long jobs to 80. While this may prune some information, it is a necessary step to scale our classification.
<figure><img src="image_17.png" /><figcaption>Figure 14: An example job of each identified category. The x axis shows time, the y axis the bandwidth, and the letters identify the categories. All remaining jobs that do not fit in a category are contained in a heterogeneous set, not represented.</figcaption></figure>

## 9.2 Categories of I/O behavior

By manually investigating and labeling each cluster across all the groups, we identified a set of commonly occurring categories. Table 12 details all the patterns we identified along with their occurrence across all the groups <fnref n="1" />. Table 13 describes the categories, while Fig. 14 shows an example of job for each. We observe a strong difference between reads and writes, consistently across the various groups. Read activity is often concentrated at the beginning of the jobs, as we observed in Section 3, but longer jobs often present different behaviors. Write activity is more heterogeneous, however we can say writing at the beginning is more common for short jobs, while longer jobs more often write throughout the execution. Having a single burst of write operations at some point is also not uncommon. C, F, G, and I jobs are often periodic.

We present in Fig. 15 the most populated cluster, for each read and write group of duration 80, by displaying its centroid. We observe that read centroids experience a peak at the beginning of the job. The peak is not exactly at time stamp 0 because the centroid represents an average

<footnote n="1">¹We performed the labeling of the clusters manually. This has two limitations. First, sometimes the DTW is not able to perfectly discriminate the time series, as the distance computation is quite sensitive to absolute values. An optimization consists in fine tuning features or further applying a fine grained group decomposition. Second, the manual labeling introduces some biases. For instance, a job can finish with I/O and be very I/O intensive (see category D and E). Similarly, the identification of bursts is skewed for jobs longer than 80 samples due to the reshaping. While it is hard to consistently classify such jobs, we tried to be as consistent as possible to extract insights.</footnote>
<table><caption>Table 12: **Jobs grouping and categorization.** Each entry presents a group of read/write time series according to both their duration and peak I/O performance. Each read/write group has a ratio indicating the proportion of time series that belong to this group. For instance, 15% of all the read time series last between 1 and 20 seconds, with I/O performance between 20 and 100 MiB/s. Then, for each homogeneous group, we identify the different categories of time series that occur. For instance, 51% of the write time series with duration between 1 and 20 seconds and peak I/O performance between 40 and 150 MiB/s fall in category B.</caption><thead><tr><th colspan="10">read categorization</th></tr><tr><th>duration</th><th>peak</th><th>I/O ratio</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>I</th><th>J</th></tr></thead><tbody><tr><td>1 - 20</td><td>1 - 4</td><td>0.07</td><td>0.73</td><td>0.13</td><td>0.01</td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.07</td></tr><tr><td>1 - 20</td><td>4 - 20</td><td>0.12</td><td>0.48</td><td>0.35</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.17</td></tr><tr><td>1 - 20</td><td>20 - 100</td><td>0.15</td><td>0.87</td><td>0.00</td><td>0.00</td><td>0.03</td><td>0.03</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.07</td></tr><tr><td>1 - 20</td><td>100 - 10<sup>6</sup></td><td>0.05</td><td>0.42</td><td>0.43</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.11</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.04</td></tr><tr><td>20 - 80</td><td>1 - 4</td><td>0.05</td><td>0.87</td><td>0.00</td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.07</td></tr><tr><td>20 - 80</td><td>4 - 20</td><td>0.08</td><td>0.79</td><td>0.10</td><td>0.00</td><td>0.00</td><td>0.04</td><td>0.00</td><td>0.04</td><td>0.00</td><td>0.00</td><td>0.03</td></tr><tr><td>20 - 80</td><td>20 - 100</td><td>0.16</td><td>0.70</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.24</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.06</td></tr><tr><td>20 - 80</td><td>100 - 10<sup>6</sup></td><td>0.06</td><td>0.47</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.47</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.06</td></tr><tr><td>80 - 10<sup>5</sup></td><td>1 - 4</td><td>0.03</td><td>0.37</td><td>0.10</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.53</td></tr><tr><td>80 - 10<sup>5</sup></td><td>4 - 20</td><td>0.1</td><td>0.75</td><td>0.07</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.05</td><td>0.07</td><td>0.00</td><td>0.00</td><td>0.07</td></tr><tr><td>80 - 10<sup>5</sup></td><td>20 - 100</td><td>0.07</td><td>0.28</td><td>0.09</td><td>0.00</td><td>0.00</td><td>0.09</td><td>0.00</td><td>0.10</td><td>0.00</td><td>0.00</td><td>0.44</td></tr><tr><td>80 - 10<sup>5</sup></td><td>100 - 10<sup>6</sup></td><td>0.07</td><td>0.19</td><td>0.19</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.19</td><td>0.19</td><td>0.00</td><td>0.24</td></tr></tbody></table><table><thead><tr><th colspan="10">write categorization</th></tr><tr><th>duration</th><th>peak</th><th>I/O ratio</th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>I</th><th>J</th></tr></thead><tbody><tr><td>1 - 20</td><td>1 - 5</td><td>0.07</td><td>0.00</td><td>0.31</td><td>0.05</td><td>0.00</td><td>0.11</td><td>0.11</td><td>0.30</td><td>0.00</td><td>0.00</td><td>0.12</td></tr><tr><td>1 - 20</td><td>5 - 40</td><td>0.11</td><td>0.00</td><td>0.62</td><td>0.02</td><td>0.00</td><td>0.05</td><td>0.05</td><td>0.19</td><td>0.00</td><td>0.00</td><td>0.08</td></tr><tr><td>1 - 20</td><td>40 - 150</td><td>0.08</td><td>0.00</td><td>0.51</td><td>0.00</td><td>0.00</td><td>0.10</td><td>0.03</td><td>0.32</td><td>0.00</td><td>0.00</td><td>0.03</td></tr><tr><td>1 - 20</td><td>150 - 10<sup>7</sup></td><td>0.03</td><td>0.19</td><td>0.35</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.34</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.12</td></tr><tr><td>20 - 80</td><td>1 - 5</td><td>0.05</td><td>0.00</td><td>0.00</td><td>0.13</td><td>0.00</td><td>0.00</td><td>0.37</td><td>0.12</td><td>0.29</td><td>0.00</td><td>0.09</td></tr><tr><td>20 - 80</td><td>5 - 40</td><td>0.14</td><td>0.00</td><td>0.00</td><td>0.06</td><td>0.00</td><td>0.06</td><td>0.00</td><td>0.31</td><td>0.35</td><td>0.06</td><td>0.16</td></tr><tr><td>20 - 80</td><td>40 - 150</td><td>0.11</td><td>0.00</td><td>0.00</td><td>0.11</td><td>0.00</td><td>0.00</td><td>0.07</td><td>0.77</td><td>0.00</td><td>0.00</td><td>0.06</td></tr><tr><td>20 - 80</td><td>150 - 10<sup>7</sup></td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.23</td><td>0.23</td><td>0.23</td><td>0.00</td><td>0.23</td><td>0.00</td><td>0.06</td></tr><tr><td>80 - 10<sup>5</sup></td><td>1 - 5</td><td>0.09</td><td>0.00</td><td>0.00</td><td>0.05</td><td>0.00</td><td>0.00</td><td>0.71</td><td>0.00</td><td>0.20</td><td>0.00</td><td>0.05</td></tr><tr><td>80 - 10<sup>5</sup></td><td>5 - 40</td><td>0.08</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.38</td><td>0.00</td><td>0.16</td><td>0.28</td><td>0.18</td></tr><tr><td>80 - 10<sup>5</sup></td><td>40 - 150</td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.04</td><td>0.00</td><td>0.19</td><td>0.33</td><td>0.00</td><td>0.04</td><td>0.11</td><td>0.28</td></tr><tr><td>80 - 10<sup>5</sup></td><td>150 - 10<sup>7</sup></td><td>0.10</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.42</td><td>0.42</td><td>0.00</td><td>0.15</td></tr></tbody></table>
<table><caption>Table 13: Categories description</caption><thead><tr><td>category</td><td>description</td></tr></thead><tbody><tr><td>A</td><td>Burst at the start of the job</td></tr><tr><td>B</td><td>Longer burst at the beginning of the job</td></tr><tr><td>C</td><td>Multiple, relatively short, I/O phases</td></tr><tr><td>D</td><td>Short job dominated by I/O</td></tr><tr><td>E</td><td>Ends with I/O activity</td></tr><tr><td>F</td><td>I/O activity dominates the job</td></tr><tr><td>G</td><td>Continuous I/O activity for a big part of the job</td></tr><tr><td>H</td><td>Short I/O burst but not at a specific moment</td></tr><tr><td>I</td><td>Dominating I/O with significant fluctuations</td></tr><tr><td>J</td><td>Hectic I/O activity</td></tr></tbody></table>

of all the jobs within the cluster (and some might have values after the start of the job). On the opposite, write centroids present much more diverse behavior, illustrating the complex IO behaviors.

## Patterns of temporal I/O behavior

The fact that we found a relatively small number of temporal I/O behavior patterns indicates it is possible to target specific behaviors for performance optimization. Moreover, the fact that there are dominating clusters in most groups says temporal I/O behavior is somewhat predictable.

Finally, we conclude clustering can be used to classify job traces. However, it is important to notice that in this paper we used manual labeling, and different methods could be required for an automatic solution. At the same time, a combination of the various approaches we used to answer the questions (for example, the segments analysis from Section 3 combined with FTIO output from Section 4) could provide a similar classification at a lower overhead than tracing and clustering time series.

<figure><img src="image_18.png" /><figcaption>Figure 15: Centroid of each dominant cluster per group across read and write time series of length 80. The x axis shows time, while the y axis the normalized bandwidth values of each centroid with respect to its maximum value, so that we can compare the trends between centroids.</figcaption></figure>
# 10 Related Work

Given the widening gap between computing power and I/O performance, understanding I/O behavior on large-scale systems has been a fairly active field for several years [51]. It generally covers the way storage systems absorb I/O, on the one hand, and application access patterns, on the other.

On the storage system side, several studies have focused on analyzing the deployment of specific storage systems after a few months of production [9, 45, 52, 60]. The aim of these studies is generally to provide both users and administrators with optimization suggestions. These conclusions are aimed at a specific system, and are rarely generalizable to other architectures. In addition, this type of work requires a large volume of logs from several sources within the system (file system logs, application monitoring tools, etc.) [50]. Attempts have nevertheless been made to provide more global optimization paths that may be suitable for several systems, albeit limited to the configuration parameters of these systems [5, 12, 20, 65]. However, those approaches require privileged access to storage systems, which makes the task difficult.

On the application side, Darshan traces are often used to study a limited set of supercomputers. For instance, in [53], the authors analyze a year-long set of I/O traces collected from three platforms. The main objective of their study is to investigate how system state impacts the I/O performance of applications. Similarly, in [74], the authors focus on jobs in the "wild" by studying the Darshan traces gathered from production jobs on a Cray XC40 machine over two months. Based on that, they propose a top-down analysis and apply it to more than 88,000 I/O traces, showing that their approach can identify common performance bottlenecks and uncover significant contributing factors for poor performance. In [59], the authors collect and analyze a year of traces from the storage system of a large-scale production system at NERSC. They are able to show that as of 2019, their data indicates that read operations are dominant (i.e., applications are reading more than writing) which is counter-intuitive. In another study, Yu et al. [80] collect and analyze I/O traces from two production HPC systems: Tianhe-1A and SC19, both located in China. In both systems, the traces are collected over a period of less than 30 days (in 2018 and 2019). Their focus is on the spatiality of bursty I/O operations, showing that traffic peaks often originate from a minority of compute nodes located close to each other. In [38], the authors study several years of Darshan logs (from 2017 to 2020) from the Theta supercomputer (at ALCF), with a main focus on understanding the causes of poor I/O throughput. Another work, by Yang et al. [79], proposes a metric to measure burstiness of I/O traces and apply it to conclude half the jobs in their cluster (which only runs CFD applications) are highly bursty. Finally, in [56], the authors study the Intrepid traces (although not from 2013) and observe high I/O performance variability.

Among other results, those papers demonstrate the effectiveness of I/O traces in providing insights into a large-scale I/O system. However, although they provide a relevant perspective on I/O behavior, the temporal aspect is mostly omitted. Such temporal I/O behavior studies are challenging because datasets are limited either in terms of machines, or in terms of the volume of data collected and analyzed. Moreover, their analysis requires the use of adapted techniques such as clustering. On this subject, Betke and Kunkel [7] design and evaluate multiple approaches for clustering 1 million jobs. Their approach aims at covering multiple aspects of I/O behavior, including the temporal one, at once. That leads to a large number of clusters, which makes extracting information from them difficult. In contrast, we focused on clustering individual time series separately.

A straightforward and computationally intensive approach for clustering time series consists in using DTW distance with a clustering algorithm [2, 41, 70]. Alternative methods [30, 76] extract features that act as proxies to calculate the distances. While potentially cheap to apply,
the key challenge of such approaches is to ensure that features capture the statistical properties
that can discriminate in a meaningful way. Deep neural networks (DNNs) showed promising
results in augmenting such features or by directly classifying series [39,82]. tsfresh [21] is a state
of the art feature extraction and series classification toolkit. This work clustered time series
using both DTW distances and tsfresh features: tsfresh feature extraction has better scalability
potential when selecting cheap to collect features as computing DTW distances quickly resulted
in memory saturation. However, we empirically observed that it was easier for us to extract and
label patterns from the different groups in Section 9 with DTW. We plan on exploring other
methods towards a more automatic clustering of I/O traces as future work. In this paper, we
fill a gap in the literature by proposing an in-depth study of temporal I/O behavior based on
massive heterogeneous datasets.

11 Conclusion

In this paper, we studied the temporal I/O behavior of over 440,000 jobs running on four HPC
systems, all different in terms of infrastructure, scale, and users, covering several time periods
over the last 11 years. The data we analyzed came either from parallel file systems (system-
side traces) or from I/O monitoring tools (application-side traces). The aim of analyzing these
traces is to provide an in-depth study of data accesses by HPC applications in the wild. We
have thus identified and addressed a number of questions dealing with the temporality of I/Os,
their periodicity, the existence and prevalence of certain patterns, I/O concurrency between
applications or user practices. We also proposed a classification of temporal I/O behaviors,
which shows a few patterns are able to represent a vast majority of jobs. Finally, we make two
large datasets of traces available for the community. Overall, the results of this study provide
relevant information for anyone working to improve high-performance I/O. They also serve as
a basis for future research into both behavior detection tools and the use of trace analysis,
particularly for scheduling and application optimization.

Acknowledgements

This work has received funding from multiple sources. The authors gratefully acknowledge the support from the Exa-DoST project of the NumPEx PEPR program (ANR-22-EXNU-0004), as part of the France 2030 initiative, and the NCSA-Inria-ANL-BSC-JSC-Riken-UTK Joint-Laboratory on Extreme Scale Computing (JLESC). Furthremore, the authors acknowledge the German Federal Ministry of Education and Research (BMBF) and the Hessian Ministry of Science and Research, Art and Culture (HMWK) for supporting this work as part of the NHR funding. Finally, the authors acknowledge the support of BMBF and the European Commission under the EuroHPC Programme ADMIRE (GA No. 956748, BMBF funding No. 16HPC006K). This work was also conducted within the DecoHPC joint team between Inria and LNCC.

Computational resources and traces were provided by the Argonne Leadership Computing Facility (DOE contract DE-AC02-06CH11357), the PlaFRIM experimental testbed (Inria, CNRS, Université de Bordeaux, Bordeaux INP, and Région Nouvelle-Aquitaine, PlaFRIM), and the SDumont supercomputer at LNCC/MCTI, Brazil. The authors gratefully thank Brice Goglin, Julien Lelaurain, and the entire PlaFRIM team for their technical support in data acquisition.
References

[1] Blue waters data-sets. https://bluewaters.ncsa.illinois.edu/data-sets.

[2] A. Abanda, U. Mori, and J. A. Lozano. A review on distance based time series classification. *Data Mining and Knowledge Discovery*, 33(2):378–412, 2019.

[3] A. Bandet, F. Boito, and G. Pallez. Scheduling distributed I/O resources in HPC systems. In *30th International European Conference on Parallel and Distributed Computing 26 - 30 August 2024 Madrid, Spain 30th International European Conference on Parallel and Distributed Computing*, Madrid, Spain, Aug. 2024.

[4] BeeGFS. Beeond: BeeGFS On Demand, 2018. https://www.beegfs.io/wiki/BeeOND.

[5] B. Behzad, S. Byna, Prabhat, and M. Snir. Optimizing I/O performance of HPC applications with autotuning. *ACM Trans. on Parallel Computing*, 5(4):15:1–15:27, Mar. 2019.

[6] A. Benoit, T. Herault, L. Perotin, Y. Robert, and F. Vivien. Revisiting I/O bandwidth-sharing strategies for HPC applications. Technical Report RR-9502, INRIA, Mar. 2023.

[7] E. Betke and J. Kunkel. The importance of temporal behavior when classifying job I/O patterns using machine learning techniques. In *High Performance Computing: ISC High Performance 2020 International Workshops, Frankfurt, Germany, June 21–25, 2020, Revised Selected Papers 35*, pages 191–205. Springer, 2020.

[8] J. Bez, A. Ramos Carneiro, P. J. Pavan, V. Soldera Girelli, F. Boito, B. A. Fagundes, C. Osthoff, P. Leite da Silva Dias, J.-F. Méhaut, and P. O. Navaux. I/O Performance of the Santos Dumont Supercomputer. *International Journal of High Performance Computing Applications*, 34(2):1–17, Sept. 2019.

[9] J. L. Bez, A. M. Karimi, A. K. Paul, B. Xie, S. Byna, P. Carns, S. Oral, F. Wang, and J. Hanley. Access patterns and performance behaviors of multi-layer supercomputer I/O subsystems under production load. In *Proc. of the 31st Int. Symp. on High-Performance Parallel and Distributed Computing*, page 43–55, Minneapolis MN USA, June 2022. ACM.

[10] J. L. Bez, A. Miranda, R. Nou, F. Boito, T. Cortes, and P. Navaux. Arbitration policies for on-demand user-level I/O forwarding on HPC platforms. In *2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 577–586, 2021.

[11] R. Bleuse, K. Dogeas, G. Lucarelli, G. Mounié, and D. Trystram. Interference-aware scheduling using geometric constraints. In *Euro-Par’18*, pages 205–217. Springer, 2018.

[12] F. Boito, G. Pallez, and L. Teylo. The role of storage target allocation in applications’ I/O performance with beegfs. In *2022 IEEE International Conference on Cluster Computing (CLUSTER)*, pages 267–277, 2022.

[13] F. Boito, G. Pallez, L. Teylo, and N. Vidal. IO-sets: Simple and efficient approaches for I/O bandwidth management. *IEEE Transactions on Parallel and Distributed Systems*, 34(10):2783–2796, 2023.

[14] F. Z. Boito, L. Teylo, M. Popov, T. Jolivel, F. Tessier, J. Luettgau, J. Monniot, A. Tarraf, A. R. Carneiro, and C. Osthoff. A deep look into the temporal I/O behavior of HPC applications [dataset], Mar. 2025. https://doi.org/10.5281/zenodo.14965920.
[15] A. R. Carneiro, J. L. Bez, C. Osthoff, L. M. Schnorr, and P. O. A. Navaux. Hpc data storage at a glance: The santos dumont experience. In *2021 IEEE 33rd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)*, pages 157–166, 2021.

[16] P. Carns. ALCF I/O Data Repository. https://www.alcf.anl.gov/publications/alcf-io-data-repository.

[17] P. Carns, K. Harms, W. Allcock, C. Bacon, S. Lang, R. Latham, and R. Ross. Understanding and improving computational science storage access through continuous characterization. *ACM Trans. Storage*, 7(3), Oct. 2011.

[18] P. Carns, R. Latham, R. Ross, K. Iskra, S. Lang, and K. Riley. 24/7 characterization of petascale I/O workloads. In *2009 IEEE International Conference on Cluster Computing and Workshops*, pages 1–10. IEEE, 2009.

[19] J. Carretero, E. Jeannot, G. Pallez, D. E. Singh, and N. Vidal. Mapping and scheduling hpc applications for optimizing I/O. In *Proceedings of the 34th ACM International Conference on Supercomputing, ICS '20*, New York, NY, USA, 2020. Association for Computing Machinery.

[20] F. Chowdhury, Y. Zhu, T. Heer, S. Paredes, A. Moody, R. Goldstone, K. Mohror, and W. Yu. I/O characterization and performance evaluation of beegfs for deep learning. In *Proceedings of the 48th International Conference on Parallel Processing*, ICPP '19, New York, NY, USA, 2019. Association for Computing Machinery.

[21] M. Christ, N. Braun, J. Neuffer, and A. W. Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (tsfresh-a python package). *Neurocomputing*, 307:72–77, 2018.

[22] R. F. da Silva, R. M. Badia, V. Bala, and et al. Workflows Community Summit 2022: A Roadmap Revolution. Technical report, Zenodo, Mar. 2023. https://zenodo.org/records/7750670.

[23] E. Deelman, T. Peterka, I. Altintas, C. D. Carothers, K. K. van Dam, K. Moreland, M. Parashar, L. Ramakrishnan, M. Taufer, and J. Vetter. The future of scientific workflows. *The International Journal of High Performance Computing Applications*, 32(1):159–175, Jan. 2018.

[24] H. Devarajan and K. Mohror. Mimir: Extending I/O interfaces to express user intent for complex workloads in HPC. In *2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 178–188, 2023.

[25] M. Dorier, G. Antoniu, F. Cappello, M. Snir, R. Sisneros, O. Yildiz, S. Ibrahim, T. Peterka, and L. Orf. Damaris: Addressing performance variability in data management for post-petascale simulations. *ACM Trans. Parallel Comput.*, 3(3), Oct. 2016.

[26] M. Dorier, G. Antoniu, R. Ross, D. Kimpe, and S. Ibrahim. CALCioM: Mitigating I/O interference in HPC systems through cross-application coordination. In *IPDPS'14*, pages 155–164. IEEE, 2014.

[27] M. Dorier, S. Ibrahim, G. Antoniu, and R. Ross. Omnisc'IO: A grammar-based approach to spatial and temporal I/O patterns prediction. In *SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 623–634, 2014.
[28] M. Dorier, S. Ibrahim, G. Antoniu, and R. Ross. Using formal grammars to predict I/O behaviors in hpc: The omnisc'IO approach. *IEEE Transactions on Parallel and Distributed Systems*, 27(8):2435-2449, 2016.

[29] N. Dryden, R. Böhringer, T. Ben-Nun, and T. Hoefler. Clairvoyant prefetching for distributed machine learning I/O. In *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, SC '21, New York, NY, USA, 2021. Association for Computing Machinery.

[30] B. D. Fulcher and N. S. Jones. Highly comparative feature-based time-series classification. *IEEE Transactions on Knowledge and Data Engineering*, 26(12):3026-3037, 2014.

[31] A. Gainaru, G. Aupy, A. Benoit, F. Cappello, Y. Robert, and M. Snir. Scheduling the I/O of hpc applications under congestion. In *2015 IEEE International Parallel and Distributed Processing Symposium*, pages 1013-1022, 2015.

[32] A. Gainaru, D. Ganyushin, B. Xie, T. Kurc, J. Saltz, S. Oral, N. Podhorszki, F. Poeschel, A. Huebl, and S. Klasky. Understanding and leveraging the I/O patterns of emerging machine learning analytics. In J. Nichols, A. B. Maccabe, J. Nutaro, S. Pophale, P. Devineni, T. Ahearn, and B. Verastegui, editors, *Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation*, pages 119-138, Cham, 2022. Springer International Publishing.

[33] J. Garcia-Blas, D. E. Singh, and J. Carretero. IMSS: In-memory storage system for data intensive applications. In *ISC High Performance 2022 International Workshops*, volume 13387 of *Lecture Notes in Computer Science*, pages 190-205. Springer International Publishing, Cham, 2022.

[34] F. Garcia-Carballeira, A. Calderon, J. Carretero, J. Fernandez, and J. M. Perez. The design of the expand parallel file system. *The International Journal of High Performance Computing Applications*, 17(1):21-37, 2003.

[35] R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-resource packing for cluster schedulers. *SIGCOMM Comput. Commun. Rev.*, 44(4):455-466, Aug. 2014.

[36] S. Herbein, D. H. Ahn, D. Lipari, T. R. Scogland, M. Stearman, M. Grondona, J. Garlick, B. Springmeyer, and M. Taufer. Scalable I/O-aware job scheduling for burst buffer enabled HPC clusters. In *Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing*, HPDC '16, page 69-80, New York, NY, USA, 2016. Association for Computing Machinery.

[37] F. Isaila, J. Carretero, and R. Ross. Clarisse: A middleware for data-staging coordination and control on large-scale HPC platforms. In *2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)*, pages 346-355, 2016.

[38] M. Isakov, E. Del Rosario, S. Madireddy, P. Balaprakash, P. Carns, R. B. Ross, and M. A. Kinsy. HPC I/O throughput bottleneck analysis with explainable local models. In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 1-13. IEEE, 2020.

[39] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller. Deep learning for time series classification: a review. *Data mining and knowledge discovery*, 33(4):917-963, 2019.
[40] E. Jeannot, G. Pallez, and N. Vidal. Scheduling periodic I/O access with bi-colored chains: models and algorithms. *J. of Scheduling*, 24(5):469-481, 2021.

[41] Y.-S. Jeong, M. K. Jeong, and O. A. Omitaomu. Weighted dynamic time warping for time series classification. *Pattern recognition*, 44(9):2231-2240, 2011.

[42] T. Jolivel, F. Tessier, J. Monniot, and G. Pallez. Mosaic: Detection and categorization of I/O patterns in hpc applications. In *SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 1311-1319, 2024.

[43] M. D. Jones, J. P. White, M. Innus, R. L. DeLeon, N. Simakov, J. T. Palmer, S. M. Gallo, T. R. Furlani, M. T. Showerman, R. Brunner, A. Kot, G. H. Bauer, B. M. Bode, J. Enos, and W. T. Kramer. Workload analysis of blue waters. *CoRR*, abs/1703.00924, 2017. http://arxiv.org/abs/1703.00924.

[44] E. Karrels, L. Huang, Y. Kan, I. Arora, Y. Wang, D. S. Katz, W. Gropp, and Z. Zhang. Fine-grained policy-driven I/O sharing for burst buffers. In *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, SC '23, New York, NY, USA, 2023. Association for Computing Machinery.

[45] A. Khan, H. Sim, S. S. Vazhkudai, A. R. Butt, and Y. Kim. An analysis of system balance and architectural trends based on Top500 supercomputers. In *The Int. Conf. on High Performance Computing in Asia-Pacific Region*, page 11-22, Virtual Event Republic of Korea, Jan. 2021. ACM.

[46] S. Kim, A. Sim, K. Wu, S. Byna, Y. Son, and H. Eom. Towards HPC I/O performance prediction through large-scale log analysis. In *Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing*, HPDC '20, page 77-88, New York, NY, USA, 2020. Association for Computing Machinery.

[47] S. Lang, P. Carns, R. Latham, R. Ross, K. Harms, and W. Allcock. I/O performance challenges at leadership scale. In *Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis*, SC '09, New York, NY, USA, 2009. Association for Computing Machinery.

[48] Lawrence Livermore National Laboratory. UnifyFS, 2019. https://github.com/LLNL/UnifyFS.

[49] Y. Liu, R. Gunasekaran, X. Ma, and S. S. Vazhkudai. Server-Side Log Data Analytics for I/O Workload Characterization and Coordination on Large Shared Storage Systems. In *SC'16*, pages 819-829, Nov 2016.

[50] Z. Liu, R. Lewis, R. Kettimuthu, K. Harms, P. Carns, N. Rao, I. Foster, and M. E. Papka. Characterization and identification of HPC applications at leadership computing facility. In *Proc. of the 34th ACM Int. Conf. on Supercomputing*, page 1-12, Barcelona Spain, June 2020. ACM.

[51] G. Lockwood, D. Hazen, Q. Koziol, R. Canon, K. Antypas, and J. Balewski. Storage 2020: A Vision for the Future of HPC Storage. In *Report: LBNL-2001072*. Lawrence Berkeley National Laboratory, 2017.

[52] G. K. Lockwood, S. Snyder, S. Byna, P. Carns, and N. J. Wright. Understanding data motion in the modern HPC data center. In *2019 IEEE/ACM Fourth Int. Parallel Data Systems Workshop (PDSW)*, page 74-83, Nov. 2019.
[53] G. K. Lockwood, S. Snyder, T. Wang, S. Byna, P. Carns, and N. J. Wright. A Year in the Life of a Parallel File System. In *SC18: International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 931–943, Nov. 2018.

[54] J. Luettgau, S. Snyder, T. Reddy, N. Awtrey, K. Harms, J. L. Bez, R. Wang, R. Latham, and P. Carns. Enabling Agile Analysis of I/O Performance Data with PyDarshan. In *Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis*, SC-W '23, pages 1380–1391, New York, NY, USA, Nov. 2023. Association for Computing Machinery.

[55] H. Luu, M. Winslett, W. Gropp, R. Ross, P. Carns, K. Harms, M. Prabhat, S. Byna, and Y. Yao. A multiplatform study of I/O behavior on petascale supercomputers. In *Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing*, pages 33–44, 2015.

[56] S. Madireddy, P. Balaprakash, P. Carns, R. Latham, R. Ross, S. Snyder, and S. M. Wild. Analysis and correlation of application I/O performance and system-wide I/O activity. In *2017 International Conference on Networking, Architecture, and Storage (NAS)*, pages 1–10. IEEE, 2017.

[57] J. Monniot, F. Tessier, H. Casanova, and G. Antoniu. Simulation of Large-Scale HPC Storage Systems: Challenges and Methodologies. In *2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)*, pages 232–242, Los Alamitos, CA, USA, Dec. 2024. IEEE Computer Society.

[58] J. Monniot, F. Tessier, M. Robert, and G. Antoniu. Supporting dynamic allocation of heterogeneous storage resources on HPC systems. *Concurrency and Computation: Practice and Experience*, 35(28):e7890, 2023.

[59] T. Patel, S. Byna, G. K. Lockwood, and D. Tiwari. Revisiting I/O behavior in large-scale storage systems: The expected and the unexpected. In *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 1–13, 2019.

[60] A. K. Paul, O. Faaland, A. Moody, E. Gonsiorowski, K. Mohror, and A. R. Butt. Understanding HPC application I/O behavior using system level statistics. In *2020 IEEE 27th Int. Conf. on High Performance Computing, Data, and Analytics (HiPC)*, pages 202–211, 2020.

[61] L. Pottier, R. F. da Silva, H. Casanova, and E. Deelman. Modeling the performance of scientific workflow executions on HPC platforms with burst buffers. In *2020 IEEE International Conference on Cluster Computing (CLUSTER)*, pages 92–103, 2020.

[62] Z. Qiao, Q. Liu, N. Podhorszki, S. Klasky, and J. Chen. Taming I/O variation on QoS-Less HPC storage: What can applications do? In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pages 1–13, 2020.

[63] F. J. Rodrigo Duro, F. Marozzo, J. García Blas, J. Carretero Pérez, D. Talia, and P. Trunfio. Evaluating data caching techniques in DMCF workflows using hercules. In *Proceedings of the Second International Workshop on Sustainable Ultrascale Computing Systems (NESUS 2015)*, pages 95–106, Spain, 2015. Universidad Carlos III de Madrid.
[64] E. Saeedizade, R. Taheri, and E. Arslan. I/O burst prediction for HPC clusters using darshan logs. In *2023 IEEE 19th International Conference on e-Science (e-Science)*, pages 1–10, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.

[65] M. Seiz, P. Offenhäuser, S. Andersson, J. Hötzer, H. Hierl, B. Nestler, and M. Resch. Lustre I/O performance investigations on hazel hen: experiments and heuristics. *The J. of Supercomputing*, 77(11):12508–12536, Nov. 2021.

[66] K. Sivalingam and H. Richardson. Application IO analysis with lustre monitoring using LASSi for ARCHER. In *High Performance Computing: ISC High Performance 2020 International Workshops, Frankfurt, Germany, June 21–25, 2020, Revised Selected Papers*, page 255–266, Berlin, Heidelberg, 2020. Springer-Verlag.

[67] H. Sung, J. Bang, C. Kim, H.-S. Kim, A. Sim, G. K. Lockwood, and H. Eom. BBOS: Efficient HPC storage management via burst buffer over-subscription. In *2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID)*, pages 142–151, 2020.

[68] A. Tarraf, A. Bandet, F. Boito, G. Pallez, and F. Wolf. Capturing periodic I/O using frequency techniques. In *2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 465–478. IEEE, May 2024.

[69] O. Tatebe, K. Obata, K. Hiraga, and H. Ohtsuji. CHFS: parallel consistent hashing file system for node-local persistent memory. In *HPC Asia 2022: International Conference on High Performance Computing in Asia-Pacific Region*, pages 115–124, USA, 2022. ACM.

[70] R. Tavenard, J. Faouzi, G. Vandewiele, F. Divo, G. Androz, C. Holtz, M. Payne, R. Yurchak, M. Rußwurm, K. Kolar, et al. Tslearn, a machine learning toolkit for time series data. *Journal of machine learning research*, 21(118):1–6, 2020.

[71] T. T. Tran, M. Padmanabhan, P. Y. Zhang, H. Li, D. G. Down, and J. C. Beck. Multi-stage resource-aware scheduling for data centers with heterogeneous servers. *J. of Scheduling*, 21(2):251–267, Apr. 2018.

[72] D. Ucar and E. Arslan. Streaming file transfer optimization for distributed science workflows. In *2020 IEEE International Conference on Cluster Computing (CLUSTER)*, pages 187–197, 2020.

[73] M. Vef, N. Moti, T. Süß, T. Tocci, R. Nou, A. Miranda, T. Cortes, and A. Brinkmann. GekkoFS - A temporary distributed file system for HPC applications. In *IEEE International Conference on Cluster Computing (CLUSTER)*, pages 319–324, USA, Sept. 2018. IEEE.

[74] T. Wang, S. Byna, G. K. Lockwood, S. Snyder, P. Carns, S. Kim, and N. J. Wright. A zoom-in analysis of I/O logs to detect root causes of i/o performance bottlenecks. In *2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)*, pages 102–111. IEEE, 2019.

[75] T. Wang, K. Mohror, A. Moody, K. Sato, and W. Yu. An ephemeral burst-buffer file system for scientific applications. In *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC)*, pages 807–818, USA, Nov. 2016. IEEE.

[76] Z. Xiao, X. Xu, H. Xing, S. Luo, P. Dai, and D. Zhan. Rtfn: A robust temporal feature network for time series classification. *Information sciences*, 571:65–86, 2021.
[77] B. Yang, X. Ji, X. Ma, X. Wang, T. Zhang, X. Zhu, N. El-Sayed, H. Lan, Y. Yang, J. Zhai, W. Liu, and W. Xue. End-to-end I/O monitoring on a leading supercomputer. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)*, pages 379–394, Boston, MA, Feb. 2019. USENIX Association.

[78] B. Yang, Y. Zou, W. Liu, and W. Xue. An end-to-end and adaptive I/O optimization tool for modern HPC storage systems. In *2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 1294–1304, 2022.

[79] W. Yang, X. Liao, D. Dong, and J. Yu. A quantitative study of the spatiotemporal I/O burstiness of HPC application. In *2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*, pages 1349–1359, 2022.

[80] J. Yu, W. Yang, F. Wang, D. Dong, J. Feng, and Y. Li. Spatially bursty I/O on super-computers: Causes, impacts and solutions. *IEEE Transactions on Parallel and Distributed Systems*, 31(12):2908–2922, 2020.

[81] W. Zhang, S. Byna, H. Sim, S. Lee, S. Vazhkudai, and Y. Chen. Exploiting user activeness for data retention in HPC systems. In *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*, SC '21, New York, NY, USA, 2021. Association for Computing Machinery.

[82] B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu. Convolutional neural networks for time series classification. *Journal of systems engineering and electronics*, 28(1):162–169, 2017.

# A On the coverage of Blue Waters jobs by Darshan

Figure 16 summarizes the activity of 316,399 application runs in 83,449 jobs across the full year and their distribution across the more than 26,864 nodes. We report both per node job counts and I/O intensity seperated into read and write activity.

In Figure 16a we plot all jobs submitted to the job scheduler (Torque) and compare it the jobs that are covered by Darshan logs in Figure 16b. The result suggest that about 9.4 % percent of jobs are captured by Darshan with a coverage with an average of 165 jobs and a median of 79 jobs per day and node. Interestingly, some of the extreme events are still well represented also in the Darshan data with a maximum of 14,881 jobs running on a node as reported by Torque and 12,808 covered by Darshan.

Figure 16c and Figure 16d add the I/O intensity perspective in TiB/day read and written per node. As Darshan did not retain a rank to host mapping, we distribute the total amount of bytes across all participating nodes to obtain an average perspective of per node I/O intensity. With on average 1,683 TiB (19 GiB/s) read and 1,909 TiB (22 GiB/s) written, jobs and nodes are writing more then they read. It is noticeable that the busiest time on the system in terms of torque jobs also coincides with the time of highest write I/O activity. For both read and write activity we observe occasional burst of very intense I/O activity. The reported peak I/O bytes read/written can only be explained by repeated reads and writes to memory. Unfortunately, the anonymization removes filename information which prevents us from filtering local I/O.
<figure><img src="image_19.png" /><figcaption>Figure 16: System-wide job and I/O activity as observed by Torque and Darshan</figcaption></figure>
Inria

RESEARCH CENTRE
Centre Inria de l'Université de Bordeaux

200 avenue de la Vieille Tour
33405 Talence Cedex

Publisher
Inria
Domaine de Voluceau - Rocquencourt
BP 105 - 78153 Le Chesnay Cedex
inria.fr

ISSN 0249-6399