# Performance Projection for Design-Space Exploration on future HPC Architectures

Clément GAVOILLE

Inria, Univ. Bordeaux, LaBRI
Talence, France

clement.gavoille@sipearl.com

Hugo TABOADA

Laboratoire en Informatique

Haute Performance pour le Calcul et la simulation
Université Paris-Saclay, CEA

91680, Bruyères le Chatel, France

hugo.taboada@cea.fr

Jens DOMKE

RIKEN Center for Computational
Science (R-CCS)
RIKEN

Japan

jens.domke@riken.jp

Brice GOGLIN

Inria, Univ. Bordeaux, LaBRI
Talence, France

brice.goglin@inria.fr

Emmanuel JEANNOT

Inria, Univ. Bordeaux, LaBRI
Talence, France

emmanuel.jeannot@inria.fr

**Abstract**—To address the growing need for performance from future HPC machines, their processor designs are constantly evolving. Assessing the impact of changes in hardware, software stack, and applications on performance is crucial in every step of a codesign process. Here, we propose a performance projection workflow to facilitate the exploration of design space for multicore nodes and multi-threaded applications. For this purpose, we analyze the architectural efficiency of an accessible source machine and determine the maximum sustainable flop/s performance of a hypothetical target machine based on its software stack on a per-thread basis. Finally, we use these characterizations to project the performance evolution from the source machine to the target machine.

In this work, we assess the strengths and weaknesses of our approach by integrating it into the Fugaku-Next Feasibility Study. We compare the accuracy and overhead of our approach with the gem5 cycle-level simulations and a fast exploration methodology based on Machine Code Analyzer (MCA), using NAS Parallel benchmarks and CCS-QCD, a quantum chromodynamics miniapp. The study demonstrates that, compared to gem5, our approach has a prediction deviation of 5% for most cases and up to 30% for extreme cases. Additionally, it exhibits an execution overhead an order of magnitude bigger than MCA but orders of magnitude smaller than gem5.

Finally, we demonstrate our approach's capability to study larger scale and more representative applications than gem5, such as QWS and Genesis, two applications of RIKEN optimized for Fugaku.

## I. INTRODUCTION

As we approach the end of Moore's Law [1], HPC processors are becoming increasingly complex to meet rising performance demands. While architectural advancements like 3D-stacked caches or heterogeneous processors offer potential solutions, ensuring efficient interplay between hardware, software stack, and applications is also essential. It necessitates a co-design approach to jointly develop these components and adapt them to the emerging use-cases (with the recent example of the emerging of AI-driven workloads in HPC). Therefore, it is crucial to rapidly examine how applications and software stack

interact with architectural innovations to optimize the co-design process. Gaining insight into performance evolution resulting from architecture changes is essential for driving co-design initiatives through design-space exploration, highlighting the critical need for a tailored performance prediction approach for each involved actor.

Nowadays, no prediction methodology combines speed, accuracy, and versatility (i.e., the ability to adapt to different settings). Historically, analytical models have been the most used prediction approach thanks to their speed and versatility. They enable exploration across diverse applications and mini-applications, efficiently navigating large design spaces. However, given the increasing complexity of processor architecture, making assumptions becomes necessary to address the problem with a fast and versatile solution while ensuring accuracy.

<figure><img src="image_1.png" /><figcaption>Fig. 1. Performance projection between a source application + software stack + hardware configuration (in blue) to a target configuration of interest (in yellow)</figcaption></figure>
In this work, we present such an analytical model based on performance projection. As described in Figure 1, it aims to project performance between a source (application + software stack + hardware) configuration (in blue) to one such target configuration of interest (in yellow). It relies on hardware characterization with two benchmarks (STREAM and HPL) and an analysis of the instruction mix and cache behavior of source and target binaries. Then, we use this analysis to extract an architectural efficiency of the source configuration with the Roofline model [2]. Finally, we project this efficiency onto the target configuration according to the evolution of the characterized metrics. With this methodology, we can study multithreaded applications by duplicating it to each thread and memory level, leading to a performance interval representing different possible limiting memory levels. By using our workflow, we demonstrate the speed and accuracy of our model in the study of the LARge Cache processor architecture conducted in the Fugaku-Next Feasibility Study [3]. We add our workflow to this study, showing faster prediction than gem5 cycle-level simulation and better accuracy than the Machine Code Analyzer (MCA) approach. We demonstrate the usefulness of this addition to the study by projecting on more representative workflows that cannot be simulated with gem5.

## II. BACKGROUND & MOTIVATIONS

Domke et al. investigated the LARC (for LARge Cache processor) architecture [3]. This architecture aims to study the performance implications of 3D stacked-cache technology on future HPC processors as a possible candidate in the Fugaku-Next Feasibility Study. The following Figure 2 presents the characteristics of one LARC Core Memory Group (CMG) compared to one CMG of a Fujitsu A64FX processor of Fugaku. One LARC CMG has more cores, a bigger shared L2 size, and higher bandwidth at 1/4th of the area (based on a projected transistor size of 1.5nm in the year 2030).

In this paper, the authors study the possible impact of the evolution of 3D-stacking technology on the performance of a broad range of HPC applications. As the LARC architecture is not available, the researchers relied on the two following approaches for performance prediction:

1) using basic-block extractor combined with Machine Code Analyzers (MCA) estimation of the critical execution path for fast maximum speedup exploration considering unlimited L1 cache size, and

2) using accurate, but slow cycle-level simulation based on the gem5 simulator [4] used in the previous Fugaku codesign initiative.

Their work's strength lies in its extensive study of various applications using MCA and Gem5, involving months of Gem5 simulations. Their findings project an average application speedup of 9.7x on a complete LARC processor compared to a current A64FX CPU.

However, the proposed methodology of this study is not suited for real applications when high precision is required. In this case, MCA is not sufficiently accurate as it only considers a L1 cache of infinite size to estimate a maximum speedup,

<figure><img src="image_2.png" /><figcaption>Fig. 2. Comparison between one CMG of A64FX and one CMG of LARC Source: [3]</figcaption></figure>

and the gem5 approach requires an unacceptable amount of time (on the order of several years). Therefore, in this paper, we propose a novel performance projection workflow to drive co-design initiatives for the future of HPC in conjunction with simulations, which provide a new trade-off between speed and accuracy compared to the MCA/gem5 methodologies. It aims to enable a first exploration of the design space around applications, software stacks, and future processors using a roofline projection approach.

To achieve this goal, we utilize the LARC study as a reference point to showcase the advantages of our approach. We aim to design a performance prediction solution faster than gem5 simulation and more accurate than the MCA approach. As a result, we conduct a performance projection of applications on a scale beyond gem5's capabilities, providing deeper insights into the anticipated speedup compared to the MCA approach. Thus, our approach complements the existing study by offering additional tools to drive co-design initiatives, including those in the Fugaku-Next Feasibility Study.

## III. PERFORMANCE PROJECTION METHODOLOGY

This section presents our approach, methodology, and implementation used for performance projection between processors having different architecture and different numbers of cores. More precisely, our methodology applies a performance projection between one configuration of source components (comprising an application, its software stack, and an accessible hardware) to another target one of interest. This projection is made thanks to metrics characterization of all three aspects of each configuration, performance analysis on the accessible source machine, and projection towards the target one (taking into account the changes in the characterization metrics between
the source and target machine) and their observed impact on the performance on the source machine.

A. Projection methodology overview

In our case, we have chosen to rely on a Roofline representation of performance as a performance analysis model due to its versatility. In performance analysis, the Roofline model [2] is a well-known representation of a comparison of the application's behavior and its limits imposed by the hardware.

We build upon the approach outlined in [5] to deal with the multithreaded application scenarios. The concept involves applying a per-thread projection between a source and a target thread and then processing these projections according to the measured metric. In this case, as we look at the number of flop/s, our per-thread projection may result in a flop/s interval to reflect different peak performances depending on the memory hierarchy. Then, we sum these intervals up to obtain the total performance of the application running concurrently on the target node. This approach allows for a more fine-grained analysis and projection by characterizing the differences between each thread and their impact on each other's available bandwidth. As a consequence, the core of this projection is the per-thread projection workflow which is depicted in Figure 3.

The first step of this workflow is to apply the per-thread analysis to each running thread of the target configuration. This analysis consists of four steps:

1) Choice of the source reference thread

2) Roofline weighting and Operational Intensity (OI) measurement (in flop/Bytes) on source and target configurations

3) Source efficiency extraction

4) Target projection

Subsequently, we apply this analysis by considering the limits of each memory level, possibly resulting in different flop/s estimation for each thread, hence the interval. The final step is to sum all these obtained intervals to obtain the application's performance interval. We present each step of our per-thread performance workflow in more detail in the following sections.

B. Choice of the reference thread

As described above, our projection's core idea is a per-thread projection between a source and a target thread. The first step of our projection approach is to choose the source thread that will serve as a reference for projection. There are two cases of projection: the source and target machines either have the same or a different number of cores.

In the first case, the choice is straightforward as we do a one-to-one mapping of threads between source and target application. However, in the other case, we have to choose a source thread that will be used as a reference for each target thread. We have chosen to identify the one with the most similar behavior to the one we want to predict. This choice aims to extract the most representative efficiency ratio for projection. Therefore, our reference thread is the one with:

• The closest Operational Intensity (OI) to the target thread;

• The most similar workload (in number of floating-point operations and bytes moved), obtained by scaling the source input if possible

As an example, when projecting an embarrassingly parallel application from a 32 cores to a 64 cores CPU, if it is possible, we adapt the source input size to be two times smaller than our target input size to keep a similar workload per core. Then, we choose the source reference thread with the closest OI. Once we have a reference thread, the next step is to run our roofline analysis on the source and target threads.

C. Roofline analysis

The objective of our roofline analysis is to extract an efficiency (linked to the application's flop/s) of the source thread running on an accessible machine by characterizing its roofline. This efficiency is then used with a target inaccessible machine's roofline, linked to a memory level, to obtain a flop/s estimation.

For the roofline modeling, we use HPL, the High-Performance Linpack [6] (Perf<sub>HPL</sub>) as a reference for the maximum attainable flop/s performance of the compute-bound region. And, for the memory aspect, we use the STREAM bandwidth [7] (BW<sub>STREAM</sub>) of each memory level as a reference (in B/s). The roofline for a single core (roofline<sub>ref</sub><sup>core</sup>) is obtained by running these benchmarks on a full node and dividing it by the number of cores (N<sub>core</sub>). This function, depending on the Operational Intensity (OI), is presented in eq. (1).

$$ \text{roofline}_{\text{ref}}^{\text{core}}(\text{OI}) = \min \left\{ \frac{\text{BW}_{\text{STREAM}}^{\text{node}}}{\text{N}_{\text{core}}} \times \text{OI}, \frac{\text{Perf}_{\text{HPL}}^{\text{node}}}{\text{N}_{\text{core}}} \right\} \quad (1) $$

However, we do not only use STREAM and HPL values for roofline performance projection, as it only characterizes the hardware and is application-independent. To account for the differences introduced by the software stacks and applications' behavior of the source and target configurations, we have opted to modify this model. It aims to better represent the actual limitations of the application's performance. To do so, we have chosen to weigh the compute and the memory part of our rooflines.

1) Peak performance weighting: Our idea behind the weighting of flop/s compute performance is to consider the maximum attainable flop/s performance of the instruction mix of our binaries. This approach allows us to characterize the usage of Fused Multiply-Add (FMA) and vector operations on each core, similar to the compute weighting described in [8].

Consequently, we calculate the weighted peak sustainable performance (in flop/s) in eq. (2) by using the mean number of floating-point operations per floating-point instruction, according to the maximum number of floating-point operations of a single instruction of the machine. It corresponds to FMA-type operations on full vectors, given by the formula $2 \times \frac{\text{vector size}}{\text{data size}}$.

$$ \text{Perf}_{\text{HPL weighted}}^{\text{core}} = \frac{\text{Perf}_{\text{HPL}}^{\text{core}}}{2 \times \frac{\text{vector size}}{\text{data size}}} \times \frac{\text{N}_{\text{floating point operations}}}{\text{N}_{\text{floating point instructions}}} \quad (2) $$
<figure><img src="image_3.png" /><figcaption>Fig. 3. Performance projection workflow overview. The choice of a source reference thread is presented in Section III-B. Then, we run a roofline analysis and Operational Intensity measurement (in flop/Bytes), this is presented in Sections III-C and III-D. From this analysis of the source configuration, we extract an efficiency ratio and project it on the target one to obtain a flop/s estimation (Section III-E). Each memory level projection results in a performance estimation value (in flop/s) that are grouped in an interval for each thread. The last step is to sum all these intervals to obtain the application performance interval in flop/s (see Section III-F)</figcaption></figure>

2) Weighting of the memory bandwidth: Furthermore, to assess the impact of the difference in sustainable bandwidth between threads on performance, we also weight the bandwidth according to the data provenance, in terms of memory levels. Moreover, as in the LARM model [9], we consider the different sockets in a NUMA topology as another memory level. It is translated to different bandwidths when a core accesses data in the main memory: when accessing data on its Current Socket (CS) or when accessing data on the Other Sockets (OS).

Hence, when weighting the memory part of the roofline according to a memory level, we consider each data going through that particular level. We thus compute an effective bandwidth using the hit ratio of this level and the higher one(s). eq. (3) shows the weighting of the bandwidth of a core where BW<sup>reference</sup> and α are the reference bandwidth obtained with STREAM and the total hit ratio of that memory level respectively.

$$BW_{L1}^{\text{core}} = \frac{\sum_{i=L1}^{i=OS} \alpha_i}{\frac{\alpha_{L1}}{BW_{L1}^{\text{reference}}} + \dots + \frac{\alpha_{CS}}{BW_{CS}^{\text{reference}}} + \frac{\alpha_{OS}}{BW_{OS}^{\text{reference}}}} \quad (3)$$

With this approach, we obtain a weighted roofline (of the different bandwidths and the peak flop/s for each core) for each memory level. It is interesting to note that, in Section IV-B, the difference in shared L2 cache size is reflected in this α parameter. It represents the limitations arising from the interactions between software and hardware and it provides a more accurate maximum of the performance imposed by both components, enabling a more accurate projection of the considered thread's interaction with the system.

### D. Operational Intensity (OI) analysis

To perform the roofline analysis and projection, we need to compute the Operational Intensity (OI), in flop/Bytes, of the application. However, we do not consider only one OI of

the application as we compute one OI per memory level and associate it with its corresponding roofline.

To assess the potential impact of all memory levels on our application's performance, we consider all the bytes passing through the considered memory level, thus including bytes coming from higher levels, following the Cache-Aware Roofline Model (CARM) [10] approach. This choice sets us apart from the Operational Intensities defined in the Integrated Roofline Model of Intel Advisor [11], as we include bytes moved in higher levels due to their significant impact on performance in our projection.

Hence, we define the OI of a memory level as the total floating-point operations (N<sub>flop</sub>) divided by the total of bytes read and written in this memory level i (Bytes<sub>i</sub>) and the ones that are higher in the memory hierarchy. As an example, for the L1 and last level cache (LLC), this leads to <fnref n="1" />:

$$\text{OI}_{L1} = \frac{\text{N}_{\text{flop}}}{\text{Bytes}_{L1} + \dots + \text{Bytes}_{LLC} + \text{Bytes}_{CS} + \text{Bytes}_{OS}} \quad (4)$$

$$\text{OI}_{LLC} = \frac{\text{N}_{\text{flop}}}{\text{Bytes}_{LLC} + \text{Bytes}_{CS} + \text{Bytes}_{OS}} \quad (5)$$

### E. Efficiency extraction and projection

Now that we have a roofline and an OI for each memory level for both source and target configurations, we need to extract the flop/s efficiency by running the application on the source machine to project it on the target roofline. The projection uses the same idea as in the non-weighted roofline case [12]: it considers the ratio, or efficiency, between the flop/s (Perf<sub>source</sub>) and one roofline on the source machine at the OI<sup>source</sup> (roofline<sub>source</sub> (OI<sup>source</sup>)), and projects this ratio (Ratio<sub>source</sub>) on the target machine using the target OI and its

<footnote n="1">¹As a side note, OI<sub>L1</sub> corresponds to the OI described in the CARM, while OI<sub>DRAM</sub> is the one described in the Original Roofline Model</footnote>
corresponding roofline ($\text{roofline}_{\text{target}}$ ($\text{OI}^{\text{target}}$)) as presented in eq. (6).

$$ \text{Perf}_{\text{target}}^{\text{proj}} = \frac{\text{Perf}_{\text{source}} \times \text{roofline}_{\text{target}}(\text{OI}^{\text{target}})}{\text{roofline}_{\text{source}}(\text{OI}^{\text{source}})} \quad (6) $$

This projection leads to a flop/s estimation of one target roofline and OI couple corresponding to a memory level (see Sections III-C and III-D). Then, we duplicate this projection with each memory level to cover the whole memory hierarchy (from the L1 cache to the memory bank of the other socket (OS) for a NUMA node). Hence, depending on the OI of a memory level, the application is limited by either the memory-level bandwidth or the peak performance. In the former case, this analysis results in multiple flop/s estimation (one for each OI/roofline), forming a projection interval (e.g. by applying (6) on each OI). In the latter case, we have only one flop/s estimation (as the peak performance does not depend on the data access pattern) and, hence, no projection interval. In real settings, the case where we do not see a projection is actually observed as, for instance in Figures 6 and 7 for CCS-QCS, EP-OMP, and IS-OMP applications in Section IV.

### F. Interval summation

As explained above, our model computes a projection interval for each application's thread running on a full node. An illustration of every step of the projection of the $OI_{L1}/\text{roofline}_{L1}$ pair is presented in Figure 4. The graph 1 presents the reference roofline obtained with the eq. (1) for the different memory levels. Then, we weigh it according to eq. (2) and eq. (3) to obtain the dashed roofline on graph 2. Then, with a performance analysis on the source machine, we obtain $\text{Perf}_{\text{source}}$ (in flop/s) and $\text{OI}^{\text{source}}$ (in flop/Bytes), which is used to compute the source ratio ($\frac{\text{Perf}_{\text{source}}}{\text{Roofline}_{\text{source}}(\text{OI}^{\text{source}})}$) corresponding to graph 3. Finally, as illustrated in graph 4, we derive a projected performance point for a single thread considering a specific memory level. This is achieved by applying this ratio to the roofline of the target OI ($\text{Roofline}_{\text{target}}(\text{OI}^{\text{target}})$) using eq. (6).

To estimate the number of flop/s of a thread, accounting for potential limitations of every memory level, we consolidate all the projected points into a single interval. We consider every memory level as possibly limiting the performance of our considered thread to account for all the possibilities on the target configuration's behavior.

Next, we sum all these thread intervals to obtain the application's performance for all threads running concurrently on the same node. It is important to note that the performance (in flop/s) gain from increasing the thread count is non-linear, and a critical aspect of our approach is its consideration of the effects (such as cache behavior, bandwidth, and load balancing) that induce non-linear scaling in the interval. This justification supports the validity of the summation presented in eq. (7) as a reliable approximation.

$$ \text{Perf}_{\text{node}} = \left[ \sum_{\text{threads}} \text{min}(\text{Perf}); \sum_{\text{threads}} \text{max}(\text{Perf}) \right] \quad (7) $$

This summation leads to a single application flop/s interval containing the possible performance evolution of the worst and best-case execution of every thread. Then, the idea for design-space exploration is to compare the evolution of this interval according to the different target configurations without necessarily changing of source configuration.

### G. Implementation

This subsection presents the implementation of our workflow used in the experiments presented in the following Section IV. Its objective is to characterize the following metrics for both our source and target hardware and binaries:

1) Stream of each memory level and HPL for each hardware;

2) Operational intensity and mean number of floating-point operations per floating-point instructions of each thread of both source and target binaries;

3) Cache hit ratio for each memory level for both binaries;

4) Performance of the source binary on the source machine.

Figure 5 presents the different steps of the implementation with an application run with source and target software stack to obtain metrics (2) and (3). In parallel, we measure STREAM, HPL, and the application performance to obtain (1) and (4). We use all these metrics on the source machine to run a roofline analysis to extract the efficiency that we project on the target roofline according to the evolution of (1), (2), and (3).

In more detail, for (1) on the source hardware, we need to run both of these benchmarks by adapting STREAM's array size to the different memory levels. In the case of the target machine, we consider these values to be either given by the constructors or extrapolated (using data from [3] in the case of the LARC). Furthermore, for (2), we rely on the instrumentation clients of DynamoRIO [13] for both binaries. Both of these analyses need a single execution on the source machine if it can execute the target binary. In the case of a target binary not using a native instruction set, we rely on the ArmIE [14] emulation tools. Then, (3) also rely on DynamoRIO and its cache simulator tool drcachesim [15]. Similarly to (2), we need a single execution of both binaries to gather the memory traces that we then process with different cache configurations. Finally, (4) requires executions on the source machine to get the execution time and compute the performance in FLOPS with the number of floating-point operations counted by the DynamoRIO client. Once all the metrics values are obtained, analysis and projection can be done on any machine (e.g. a laptop). Hence, the step that takes the longest execution time for each application is obtaining (2) and (3) as (1) only requires one execution per target hardware configuration.

## IV. EXPERIMENTS

As presented in Section II, our experiments capitalize on the performance study efforts of Domke et al. in [3]. They aim to validate the accuracy of our projection approach by comparing it to the MCA prediction and gem5 simulation results to assess the accuracy of our approach. As such, we have chosen the same experimental environment. With an average accuracy of 1.3% on various benchmarks compared to the A64FX chip
<figure><img src="image_4.png" /><figcaption>Fig. 4. Illustration of the performance projection of a memory level of a thread. Graph 1 uses eq 1 whereas Graph 2 corresponds to the eqs. 2 and 3 presented in Section III-C. Graph 3 to the OI analysis and performance extraction of Sections III-D and III-E and Graph 4 describe the efficiency projection of eq. 6</figcaption></figure>

[16], we consider the gem5 results to be the target results for performance prediction on LARC.

## A. Experimental environment

Our experiments aim to project performance from one A64FX CMG toward one LARC CMG presented in Figure 2. However, because of the technical limitation of gem5 cache simulation and *drcachesim*, we cannot simulate an L2 cache of 384 MiB as the possible sizes are limited to a power of 2. It directed the choice to duplicate the LARC target machines in two configurations: LARC<sub>C</sub> (for conservative) and LARC<sub>A</sub> (for aggressive). Another aspect to study of the LARC architecture is the increase in the number of cores per CMG. Hence, one of the target configurations is the A64FX<sub>S</sub> or an A64FX CMG with 32 cores. The objective of this configuration is to study the impact of only increasing the number of cores on the performance of applications. In addition to the different target machines defined in the previous study efforts on LARC, we have defined an A64FX<sub>L</sub> target machine with only a shared cache size increase compared to a CMG of A64FX. The following Table I presents the characteristics of all the source and target configurations.

To obtain the HPL performance on target configurations, we extrapolated the measured HPL on one A64FX CMG and

multiplied per the increase in the number of cores (in this case ×2.66). We use the same extrapolation for the L1 STREAM value whereas we keep the HBM STREAM value for all machines as it does not change between all configurations. Finally, for L2 STREAM, we used the gem5 simulated value from [3].

<table><caption>TABLE I
CHARACTERISTICS AND BENCHMARK RESULTS OF ALL THE CONFIGURATIONS</caption><thead><tr><th>Machines</th><th>A64FX</th><th>A64FX<sub>L</sub></th><th>A64FX<sub>S</sub></th><th>LARC<sub>C</sub></th><th>LARC<sub>A</sub></th></tr></thead><tbody><tr><td>Cores</td><td>12</td><td>12</td><td>32</td><td>32</td><td>32</td></tr><tr><td>HPL (GFLOPS)</td><td>615.11</td><td>615.11</td><td>1640.29</td><td>1640.29</td><td>1640.29</td></tr><tr><td>HBM (GB/s)</td><td>206</td><td>206</td><td>206</td><td>206</td><td>206</td></tr><tr><td>L1 size (KiB)</td><td>768</td><td>768</td><td>2048</td><td>2048</td><td>2048</td></tr><tr><td>L1 (GB/s)</td><td>1900</td><td>1900</td><td>4800</td><td>4800</td><td>4800</td></tr><tr><td>L2 size (MiB)</td><td>8</td><td>256</td><td>8</td><td>256</td><td>512</td></tr><tr><td>L2 size/core (MiB)</td><td>0.67</td><td>21.33</td><td>0.25</td><td>8</td><td>16</td></tr><tr><td>L2 (GB/s)</td><td>791</td><td>791</td><td>791</td><td>791</td><td>1450</td></tr></tbody></table>

The compiler chosen for each application is the Fujitsu Compiler suite version 4.6.1 using the same flags as the gem5
<figure><img src="image_5.png" /><figcaption>Fig. 5. Implementation workflow</figcaption></figure>

simulation. The validation experiments are conducted on 5
NAS class-B benchmarks (IS, CG, MG, FT, and EP) [17]
and CCS-QCD<fnref n="2" />, a RIKEN miniapplication derived from a
quantum chromodynamic code similar to QWS <fnref n="3" />, studied
in Section IV-D. We also study genesis<fnref n="4" /> [18], a molecular
dynamics simulation framework developed by RIKEN. Every
performance and instrumented runs are conducted on A64FX
nodes of Fugaku.

B. *Comparison with gem5 simulation and MCA*

Figure 6 presents our projection results from A64FX to the three target configurations on five NAS CLASS-B kernels and ccs-qcd. The projected interval is represented as a black bar, with the median in blue, the gem5 simulation speedup results in green, and the MCA speedup results are represented as a black triangle (described in Section II). For each application, the projection targets A64FXS, LARC<sub>C</sub>, and LARC<sub>A</sub> from left to right. One important note is that we scaled the source reference thread input size (as described in Section III-B) for CG-OMP, FT-OMP, and ccs-qcd. Furthermore, as A64FXS, LARC<sub>C</sub>, and LARC<sub>A</sub> are hypothetical machines, we consider the gem5 simulation results as the reference of the expected speedup on a CMG.

When comparing with the MCA speedup, it is important
to note that this approach represents the upper bound with
infinite L1 size and no increase in number of cores. Moreover,
the speedup is measured using a 48-core Intel E5-2650v4
(Broadwell) as a reference. Hence, the objective is to describe
the ability of the application to scale with the best possible
memory behavior. Our projection approach leads to predicted
speedup closer to the gem5 simulation for every application and
target machine than MCA. More precisely, one interesting case
is for EP-OMP where MCA predicts a maximum upper bound
of 1.37x while our projection and gem5 simulation predicts
a speedup of 2.67x, related to the increase in the number of
cores per CMG between LARC and A64FX.

On the other hand, our projection predicts a similar speedup
behavior and a good correlation to gem5 for every application,

even in the case of FT-OMP on A64FXS where it exhibits
a slowdown. Our average prediction accuracy is 30% for
A64FXS, 21% for LARC, and 15% for LARC (computed
with L1, L2 and DRAM projections compared with gem5
predicted speedup). The higher accuracy of LARC can be
explained by the higher throughput of LARC that leads to
a more compute-bound behavior of applications and a more
straightforward speedup following the increase in the number
of cores.

Quantitatively, the simulation results are either in the interval
or in a 5% error range for 4 out of the six applications: CG-
OMP, FT-OMP, EP-OMP, and ccs-qcd. However, for IS-OMP,
we miss the speedup evolution behavior between the three
machines as we expect a similar speedup of 2.67× on all three
configurations. This speedup corresponds to the increase in the
number of cores as we expect a compute-bound behavior on
A64FXS. This is due to IS operating on integers that are not
caught in our Operational Intensity as we only consider floating-
point operations. Furthermore, for MG-OMP, we predict a
speedup gain but overpredict it for A64FXS and LARC and
underpredict it on LARC. We correlate it to the performance
of MG-OMP being impacted by the latency of L2 memory
accesses. As we use a roofline approach, our assessment of
performance is latency-blind. In this case, we do not accurately
predict performance because we do not account for the impact
of L2 latency accesses.

C. *Study of the impact of shared cache size on performance*

Figure 7 presents the projection of our model on the same
pool of applications as Section IV-B. The target of these
projections is the A64FX<sub>L</sub> machine, which is a CMG of
A64FX with a bigger L2 size (from 8 MiB to 256 MiB). This
machine is not described in [3], hence, we do not have gem5
results for comparison. With 12 cores, the L2 size increase
has minimal or no impact on the performance of CCS-QCD,
EP-OMP, and IS-OMP. With a maximum performance gain
of 7%, 72%, and 13%, CG-OMP, FT-OMP, and MG-OMP
are the applications that would benefit the most from this
change. In comparison, both gem5 and our approach predict
a similar impact on performance when increasing the cache
size between A64FX<sub>S</sub> and LARC<sub>C</sub>. However, quantitatively,
the performance gains predicted by gem5 are more significant

<footnote n="2">²https://github.com/fiber-miniapp/ccs-qcd</footnote>

<footnote n="3">³https://github.com/RIKEN-LQCD/qws</footnote>

<footnote n="4">⁴https://www.r-ccs.riken.jp/labs/cbrt/</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 6. Comparison between model (median in blue, interval in black), gem5 simulation (in green), and MCA results (black triangle) on 6 applications: ccs-qcd and 5 CLASS-B NAS kernels. Target machines are A64FX<sub>S</sub>, LARC<sub>C</sub>, and LARC<sub>A</sub> from left to right for each application. Speedup is presented using a log scale to more accurately represent the possible slowdown. CCS-QCS, EP-OMP, and IS-OMP's projections lead to a single value and not an interval.</figcaption></figure>

between both of these machines on FT-OMP (+137%), CG-
OMP (+17%), and MG-OMP (+53%). We can verify that cache
contention is increased with 32 cores instead of 12 cores (2.8%
of local miss rate instead of 0.45%), explaining the greater
impact of shared cache size on LARC<sub>C</sub> performance instead
of A64FX<sub>L</sub>. Both of these experiments highlight the sensitivity
of these three applications' performance to the shared L2 size,
even with 12 cores.

This experiment emphasizes the versatility of our approach
in its capacity to study the impact of a single parameter as well
as combinations of these parameters. It allows for isolating the
main factor contributing to performance gain, thereby directing
exploration and co-design efforts toward this identified factor.

D. *Projection on QWS and genesis*

Following the comparison of our projection accuracy with
gem5 simulation on many small kernels, we want to assess
the capacity of our workflow to study performance on bigger,
and more representative, workloads. Hence, we have chosen to
study *QWS*, a Lattice Quantum Chromodynamics and *genesis*,
a molecular dynamics simulation framework developed by
RIKEN. Both of these applications are optimized for Fugaku
and use MPI and OpenMP for parallelization. However, LARC
gem5 simulation results are not available due to the exceedingly
long simulation time it would require and the inability of gem5
to simulate MPI applications. Additionally, the accuracy of
the QWS projections are supported by the previous results on

<figure><img src="image_7.png" /><figcaption>Fig. 7. Model projection on the A64FX<sub>L</sub> machine. Speedup is presented using a log scale to more accurately represent the possible slowdown. CCS-QCS, EP-OMP, and IS-OMP's projections lead to a single value and not an interval.</figcaption></figure>

ccs-qcd as this miniapplication is derived from a similar code
to QWS.

Figure 8 presents the projected speedup obtained of QWS
with two different problem sizes: 32 × 6 × 4 × 3 (small) and
32 × 32 × 32 × 32 (large) and genesis using AMBER Force
field and NPT ensemble (corresponding to DHFR from [19]). It
also presents the MCA result for QWS on the small problem<fnref n="5" />
For QWS, between target machines, the expected speedup is
similar, between 2.5× and 2.8× for the small problem and

<footnote n="5">⁵QWS large and genesis are currently running at the time of the writing</footnote>
2.9× to 3.4× for the large input. It is opposite to the MCA results, showing a slowdown of 0.71× with infinite L1 size but no increase in the number of cores. As a result, we expect the performance gain to be linked with the increase in number of cores per CMG and not with the increase in L2 size and bandwidth. This compute-bound behavior is confirmed by the around 99% L1 hit ratio observed on the source machine. However, the large problem benefits more from this gain in the number of cores.

Furthermore, Genesis' behavior is different from QWS as we observe a compute-bound behavior on the source machine and no interval in the projection. This is confirmed by the predicted speedup at 2.65× for every target configuration. However, contrarily to QWS, the projected speedup is not impacted by the L2 changes between target configurations.

<figure><img src="image_8.png" /><figcaption>Fig. 8. Projection results on QWS for 2 input sizes (32 × 6 × 4 × 3 (small) and 32 × 32 × 32 × 32 (large)) and genesis. Target machines are A64FXS, LARC<sub>C</sub>, and LARC<sub>A</sub> from left to right for each application. Speedup is presented using a log scale. Genesis' projections leads to a single value and not an interval.</figcaption></figure>

We have been able to study two applications of a larger scale than the one simulated with gem5. This study shows the ability of our approach to make projections for applications in a reasonable time. As a point of comparison, the gem5 simulation of QWS would take around 5 years if we take a similar overhead as ccs-qcd.

## E. Study of the workflow overhead

In the studies presented above, we only compared the accuracy of the approach. Another important aspect of a prediction workflow is the overhead according to the source kernel, or the expected time needed to obtain a result. The objective is to compare the average overhead ($T_{\text{ workflow}}/T_{\text{ native}}$) to obtain a prediction result on a target configuration.

Hence, in the following Table II showing the overhead of the different approaches for all studied applications, we use the average simulation time of A64FXS, LARC<sub>C</sub>, and LARC<sub>A</sub> as

the reference time for gem5 for each application. Similarly, we sum the profiling time (with SDE) and the MCA simulation time to obtain the MCA approach overhead computed with the source time of the source Intel CPU. Finally, for our approach, as we have to obtain metrics on source and target configurations, the reference time is this eq. (8):

$$T_{\text{ workflow}}^{\text{projection}} = T_{\text{analysis}}^{\text{source}} + T_{\text{cachesim}}^{\text{source}} + T_{\text{analysis}}^{\text{target}} + T_{\text{cachesim}}^{\text{target}} \quad (8)$$

<table><caption>TABLE II
EXECUTION TIME OVERHEAD OF GEM5, MCA AND OUR WORKFLOW</caption><thead><tr><th>Application</th><th>Source time (s)</th><th>gem5 overhead</th><th>Projection overhead</th><th>MCA overhead</th></tr></thead><tbody><tr><td>IS-OMP</td><td>0.41</td><td>510000x</td><td>18000x</td><td>320x</td></tr><tr><td>CG-OMP</td><td>12.57</td><td>140000x</td><td>1000x</td><td>23x</td></tr><tr><td>MG-OMP</td><td>0.49</td><td>380000x</td><td>26000x</td><td>300x</td></tr><tr><td>FT-OMP</td><td>1.60</td><td>250000x</td><td>4000x</td><td>63x</td></tr><tr><td>EP-OMP</td><td>11.13</td><td>80000x</td><td>1000x</td><td>120x</td></tr><tr><td>ccs-qcd</td><td>40.74</td><td>200000x</td><td>430x</td><td>67x</td></tr><tr><td>QWS</td><td>811.7</td><td>Too long to run</td><td>530x</td><td>1800x for the small input</td></tr><tr><td>genesis</td><td>260.2</td><td>Too long to run</td><td>380x</td><td>Unknown</td></tr></tbody></table>

We see that the overhead of our approach is orders of magnitude smaller compared to gem5, especially for longer applications such as ccs-qcd or CG-OMP. This overhead difference between gem5 and our approach is expected as an important overhead is a characteristic of cycle-level simulations. The smaller overhead on larger applications is due to the size limitation of the traces used to simulate the cache behavior leading to a constant simulation time. In this case, the constant cache simulation time from the traces is alleviated by the longer analysis time. Another strength of our approach is that we can amortize the source instrumentation and cache simulation analysis when computing projection on different target configurations. This allows us to explore the design-space faster around different configurations of interest, such as presented in the previous studies. However, we think that cycle-level simulation is still required at a later stage of design-space to ensure the prediction accuracy and it leads to a better insight of the performance thanks to the numerous metrics obtainable with gem5.

On the other hand, when compared with the MCA approach, we show an order of magnitude bigger overhead for every application. This is also expected as the instrumentation and cache simulation is typically heavier than Machine Code Analyzer simulations. Hence, the overhead of the MCA approach is smaller because it is directed by the length of the simulation. Furthermore, it is important to note that the MCA approach does not need to be run with the three target configurations contrary to our approach which needs to run half of the analysis workflow for every target configurations. However, in this paper, we have shown that we can describe a speedup closer to the one obtained with gem5 simulation
and also to characterize the impact on the speedup of different
target configurations.

These studies show the capability of our approach as a
middle-ground between the accuracy of gem5 and the speed of
the MCA approach used in the Fugaku-Next Feasibility Study.

V. DISCUSSIONS

Our experiments highlighted the strengths of our approach: the ability to make accurate predictions for future HPC architectures of interest with an execution overhead that is several orders of magnitude smaller than that of cycle-level simulations. It also emphasized the possibility of exploring a large design-space on applications closer to actual workloads than miniapplications. However, it has also shown the limitations of our approach, especially in the study of the micro-architecture, instruction latency, and the differences in stalls on the performance evolution between source and target configuration.

In our projection approach, we consider different memory
levels as limiting to cover different possibilities. As described
in a roofline representation, the impact of memory hierarchy on
performance is focused on their difference in bandwidth. The
latency impact on performance is contained in the efficiency
that we project. Hence, when we project this efficiency from
the source roofline to the target roofline, we consider a similar
impact of latency on performance between source and target
machine. As such, we think the current iteration of our
projection workflow is not tailored for precise analysis of the
instruction latency, stalls, and Out-Of-Order resources impact
on performance.

These limitations further complement the need to extend
our workflow's projections with more accurate tools such as
cycle-level simulation, or tools that allow for instruction latency
analysis such as MCA. We think the use of different approaches
are required to cover all the key aspects of performance with
various accuracies at different steps of the design of future
HPC processors.

VI. RELATED WORK

Nowadays, analytical modeling of performance needs to
make assumptions to ensure speed and versatility while
dealing with the increasing complexity of processors and the
interactions of applications and software stack with them.

To speed up their hardware-independent mechanistic ap-
proaches, Jongerius et al. or Van Der Steen et al. [20]–[23]
choose to neglect the impact of runtime and compiler targeting
an architecture. Instead, they rely on LLVM’s Intermediate
Representation (IR) to characterize the application. However,
in the context of recent developments such as ARM in HPC,
and drawing from experience with A64FX [24], we believe
it is crucial to consider the impact of the software stack to
accurately predict the performance of HPC applications.

The CPU Interval Model, developed by Eyerman et al. [25] and implemented in SniperSim [26], is able to study differences brought by software stack in the performance of applications with its accurate predictions. However, it

demonstrates a 4-5x speedup compared to Gem5 when running
a full application without fast-forwarding. Hence, simulating
full-scale application such as QWS or genesis would still
require years of computation.

Learning-based approaches [27]–[29] have been proposed for performing performance projection of applications across different architectures. However, these approaches require training their models on many benchmarks, necessitating a simulator or a prototype of the target machine. This approach proves to be costly in terms of learning time and becomes ineffective when the target machine is unavailable.

Another performance projection approach [30] relies on a set of mini-benchmarks, called SSSP (for Simplified Sustained System Performance) to characterize the behavior of the target configuration and can extrapolate applications’ performance thanks to these benchmark results. Hence, even if it is not as important as learning-based approaches, the main drawback of this approach is the need for results on non-generic benchmarks on a hypothetical or inaccessible machine.

Our approach differs from these methodologies by only
relying on generic benchmarks (such as Stream [7] and High-
Performance Linpack [6]), often provided by the manufacturer,
to characterize the target machine. In this case, even in the
absence of the machine, such characterization can still be
employed using vendor data or extrapolated.

To the best of our knowledge, the concept of using the
roofline model for CPU performance projection was initially
introduced by Kwack et al. [12]. However, their work is
constrained to a single OI, overlooking the potential variability
of OI across different machines. Additionally, they do not
incorporate weights in their roofline to consider the impact
of instruction mix and cache behavior on performance. In a
related context, [5], [31] employs a roofline-based projection,
but their workflow is restricted to single-core projections for the
first, and focuses on GPU for the latter. Moreover, they do not
apply their performance projection method in an exploratory
study similar to the work presented in this paper.

VII. CONCLUSION AND FUTURE WORK

This paper introduces a performance projection methodology
tailored for the first steps of design-space exploration for
future processors of HPC. This workflow relies on a per-thread
projection between one configuration of source components
(comprising an application, its software stack, and an accessible
hardware) to another target configuration of interest (with
hypothetical hardware). The main steps of this workflow is the
choice of the source reference thread followed by a weighted
roofline analysis on this source thread, describing the maximum
attainable flop/s of the instruction mix with the observed cache
behavior. Then, we extract efficiency from this roofline analysis
and project it toward the target roofline, relying only on binary
instrumentation, cache simulation, and STREAM and HPL
results of the target configuration. By duplicating the projection
for each memory level and summing the obtained performance
interval of each thread, we obtain a performance interval of
the entire application.
Subsequently, we applied our model in the Fugaku-Next Feasibility Study [3], comparing the accuracies and overhead of our projection with the extended simulation efforts conducted in that study. Qualitatively, our predictions, when compared to the gem5 cycle-level simulator results, demonstrate high accuracy, consistently capturing performance gains or losses. Quantitatively, the average difference is close to 5% for 4 out of 6 benchmarks. For the other two benchmarks, we have identified the reasons for an average difference of up to 30% due to our latency-blind approach and our focus on floating-point operations.

Furthermore, we have demonstrated the versatility of our approach with its ability to focus on a single parameter change to isolate the most important performance factor in projection.

Additionally, our predictions are obtained much more rapidly than cycle-level simulations, with an overhead difference several magnitudes lower, and an order of magnitude bigger than MCA simulation. This enables us to gain performance insights into larger-scale workloads than achievable with gem5.

Overall, these experiments showed the capability of our approach to be used in the first step of design-space exploration, showing good accuracy with a small overhead thanks to a roofline-based projection and only relying on STREAM and HPL benchmarks to describe the target hardware. However, we think it is important to complement our projection with more simulations (such as gem5 and MCA) in the later stage of codesign. It would allow us to obtain numerous metrics and identify other performance bottlenecks. It would also allow us to compensate for the limitations of our projection, notably on microarchitecture-focused studies.

Encouraged by these findings, we plan to address some weaknesses of the projection workflow by introducing the instruction and memory latency in the projected efficiency. Furthermore, as the nodes become increasingly heterogeneous, we aim to characterize and project performance with kernels operating on accelerators such as GPUs and FPGAs.

## REFERENCES

[1] T. N. Theis and H.-S. P. Wong, "The end of moore's law: A new beginning for information technology," *Computing in Science & Engineering*, vol. 19, no. 2, pp. 41–50, 2017.

[2] S. Williams, A. Waterman, and D. Patterson, "Roofline: An insightful visual performance model for multicore architectures," *Commun. ACM*, 2009.

[3] J. Domke, E. Vatai, B. Gerofi, Y. Kodama, M. Wahib, A. Podobas, S. Mittal, M. Pericàs, L. Zhang, P. Chen, A. Drozd, and S. Matsuoka, "At the locus of performance: Quantifying the effects of copious 3d-stacked cache on hpc workloads," *ACM Trans. Archit. Code Optim.*, vol. 20, no. 4, dec 2023. [Online]. Available: https://doi.org/10.1145/3629520

[4] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood, "The gem5 simulator," *SIGARCH Comput. Archit. News*, vol. 39, no. 2, p. 1–7, aug 2011. [Online]. Available: https://doi.org/10.1145/2024716.2024718

[5] C. Gavoille, H. Taboada, P. Carribault, F. Dupros, B. Goglin, and E. Jeannot, "Relative performance projection on arm architectures," pp. 85–99, 2022.

[6] A. Petitet, R. Whaley, J. Dongarra, and A. Cleary, "Hpl – a portable implementation of the high-performance linpack benchmark for distributed-memory computers," 2008.

[7] J. McCalpin, "Memory bandwidth and machine balance in high performance computers," *IEEE Technical Committee on Computer Architecture Newsletter*, 1995.

[8] D. Marques, A. Ilic, Z. A. Matveev, and L. Sousa, "Application-driven cache-aware roofline model," *Future Generation Computer Systems*, 2020.

[9] N. Denoyelle, B. Goglin, A. Ilic, L. Sousa, and E. Jeannot, "Modeling large compute nodes with heterogeneous memories with cache-aware roofline model," 01 2017.

[10] A. Ilic, F. Pratas, and L. Sousa, "Cache-aware roofline model: Upgrading the loft," *IEEE Computer Architecture Letters*, 2014.

[11] Intel. Integrated roofline model with intel advisor. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/integrated-roofline-model-with-intel-advisor.html

[12] J. H. Kwack, G. Arnold, C. Mendes, and G. Bauer, "Roofline analysis with cray performance analysis tools (craypat) and roofline-based performance projections for a future architecture," *Concurrency and Computation: Practice and Experience*, 2018.

[13] D. L. Bruening, "Efficient, transparent, and comprehensive runtime code manipulation," 01 2004.

[14] Arm, "Arm, emulating sve on existing armv8-a hardware using dynamorio and armie," https://community.arm.com/arm-community-blogs/b/high-performance-computing-blog/postsemulating-sve-on-armv8-using-dynamorio-and-armie, Nov. 2018.

[15] D. Bruening, "drcachesim, tracing and analysis framework," https://dynamorio.org/page_drcachesim.html, Mar. 2024.

[16] Y. Kodama, T. Odajima, A. Asato, and M. Sato, "Evaluation of the riken post-k processor simulator," 04 2019.

[17] D. Bailey, E. Barszcz, J. Barton, D. Browning, R. Carter, L. Dagum, R. Fatoohi, P. Frederickson, T. Lasinski, R. Schreiber, H. Simon, V. Venkatakrishnan, and S. Weeratunga, "The nas parallel benchmarks," *Int. J. High Perform. Comput. Appl.*, vol. 5, no. 3, p. 63–73, sep 1991. [Online]. Available: https://doi.org/10.1177/109434209100500306

[18] J. Jung, T. Mori, C. Kobayashi, Y. Matsunaga, T. Yoda, M. Feig, and Y. Sugita, "Genesis: a hybrid-parallel and multi-scale molecular dynamics simulator with enhanced sampling algorithms for biomolecular and cellular simulations," *WIREs Computational Molecular Science*, vol. 5, no. 4, pp. 310–323, 2015. [Online]. Available: https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/wcms.1220

[19] RIKEN, "Genesis benchmark," https://www.r-ccs.riken.jp/labs/cbrt/benchmark-2020/, 2020.

[20] R. Jongerius, G. Mariani, A. Anghel, G. Dittmann, E. Vermij, and H. Corporaal, "Analytic processor model for fast design-space exploration," 2015.

[21] R. Jongerius, A. Anghel, G. Dittmann, G. Mariani, E. Vermij, and H. Corporaal, "Analytic multi-core processor model for fast design-space exploration," *IEEE Transactions on Computers*, 2018.

[22] S. De Pestel, S. Van den Steen, S. Akram, and L. Eeckhout, "Rppm: Rapid performance prediction of multithreaded workloads on multicore processors," in *2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)*, 2019, pp. 257–267.

[23] S. Van den Steen, S. De Pestel, M. Mechri, S. Eyerman, T. Carlson, D. Black-Schaffer, E. Hagersten, and L. Eeckhout, "Micro-architecture independent analytical processor performance and power modeling," in *2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)*, 2015, pp. 32–41.

[24] J. Domke, "A64FX - your compiler you must decide!" *CoRR*, vol. abs/2107.07157, 2021. [Online]. Available: https://arxiv.org/abs/2107.07157

[25] S. Eyerman, L. Eeckhout, T. Karkhanis, and J. E. Smith, "A mechanistic performance model for superscalar out-of-order processors," *ACM Trans. Comput. Syst.*, vol. 27, no. 2, May 2009. [Online]. Available: https://doi.org/10.1145/1534909.1534910

[26] T. E. Carlson, W. Heirman, S. Eyerman, I. Hur, and L. Eeckhout, "An evaluation of high-level mechanistic core models," *ACM Transactions on Architecture and Code Optimization (TACO)*, 2014.

[27] X. Zheng, H. Vikalo, S. Song, L. K. John, and A. Gerstlauer, "Sampling-based binary-level cross-platform performance estimation," in *Design, Automation & Test in Europe Conference & Exhibition (DATE)*, 2017, 2017.

[28] X. Zheng, L. John, and A. Gerstlauer, "Lacross: Learning-based analytical cross-platform performance and power prediction," *International Journal of Parallel Programming*, vol. 45, 12 2017.

[29] X. Qi, J. Chen, and L. Deng, "Cp3: Hierarchical cross-platform power/performance prediction using a transfer learning approach," in
*Algorithms and Architectures for Parallel Processing*, W. Meng, R. Lu, G. Min, and J. Vaidya, Eds. Cham: Springer Nature Switzerland, 2023, pp. 117–138.

[30] M. Tsuji, W. T. C. Kramer, J.-C. Weill, J.-P. Nominé, and M. Sato, "A new sustained system performance metric for scientific performance evaluation," 2021.

[31] L. Van Lanker, H. Taboada, E. Brunet, and F. Trahay, "Predicting gpu kernel's performance on upcoming architectures," in *Euro-Par 2024: Parallel Processing*, J. Carretero, S. Shende, J. Garcia-Blas, I. Brandic, K. Olcoz, and M. Schreiber, Eds. Cham: Springer Nature Switzerland, 2024, pp. 77–90.