# Compiler, Runtime, and Hardware Parameters Design Space Exploration

Lana Scravaglieri, Ani Anciaux-Sedrakian, Olivier Aumage, Thomas Guignon, Mihail Popov

► To cite this version:

Lana Scravaglieri, Ani Anciaux-Sedrakian, Olivier Aumage, Thomas Guignon, Mihail Popov. Compiler, Runtime, and Hardware Parameters Design Space Exploration. IPDPS 2025 - 39th IEEE International Parallel and Distributed Processing Symposium, Jun 2025, Milan, Italy. hal-04969854

HAL Id: hal-04969854

https://inria.hal.science/hal-04969854v1

Submitted on 27 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Compiler, Runtime, and Hardware Parameters
Design Space Exploration

Lana Scravaglieri*†, Ani Anciaux-Sedrakian*, Olivier Aumage†, Thomas Guignon*, Mihail Popov†

*IFPEN, Rueil-Malmaison, France

†Inria, université de Bordeaux, Bordeaux INP, CNRS, France

{ani.anciaux-sedrakian, thomas.guignon}@ifpen.fr

{lana.scravaglieri, olivier.aumage, mihail.popov}@inria.fr

**Abstract**—HPC systems are increasingly complex with many tunable parameters impacting applications’ metrics—e.g., performance, energy consumption. The main challenges of these systems are finding the appropriate configuration per application on any given system and understanding how the configurations affect applications’ metrics on a system. Both can be addressed with design space exploration (DSE). However, exploring all the configurations available is costly due to the long execution and setup times of these executions. Indeed, it requires instrumenting the applications to collect data, compiling them with different options and setting the parameters for each execution. DSE algorithms can greatly reduce the exploration time by guiding which configuration to execute next to reach the objective without evaluating all the configurations. A DSE study thus requires implementing an exploration algorithm and automating parameters setting, application instrumentation and compilation, and metrics collection. This represents a huge overhead to the actual study, yet most DSE studies still do it from scratch.

To alleviate the setup cost, we propose a unified methodology to perform the exploration and implement it in the CORHPEX framework to setup configurations with compiler, runtime, and hardware parameters, efficiently and flexibly. The framework enables choosing the exploration strategy, the design space to study, the applications to execute and the metrics to collect independently while involving little coding overhead. It is extensible with custom exploration algorithms and data readers.

We demonstrate the versatility and robustness of our framework on parallel codes, including NAS, Rodinia, LULESH benchmarks, and real-world applications, on two systems exposing different parameters with various DSE techniques and goals. We show that working with CORHPEX enables getting insights on code optimization strategies by using exploration algorithms that can speedup the execution by a factor of 10X while preserving 95% the possible gains. Finally, we demonstrate the framework’s potential for more advanced studies by training surrogate models of complex HPC applications achieving over 93% accuracy.

# I. INTRODUCTION

The complexity of hardware and software is ever-increasing. Indeed, processors keep packaging more cores while caches are layered in a complex memory hierarchy, and each new generation of architecture comes with extended instruction sets. This results in multiple intertwined levels of parallelism. Speculative execution—including branch prediction and data prefetching with access pattern recognition—has also become a standard feature in modern processors. To take advantage of the computing power offered by these architectures, and to satisfy the constraints of application fields—e.g., precision, accuracy, robustness—software code bases grow bigger and

need to be tailored to the hardware they are executed on. Compilers play a major role in transforming code into binaries optimized for a target system. However, these numerous optimizations are hard to combine efficiently, and sometimes not portable as they depend on system specificities. Thus, compiler optimizations, software code, and hardware interact in complex ways that are hard to predict, which makes approaching peak performance difficult.

All these parameters combine into a **large design space for application execution** that is **impractical to explore**. Optimization-oriented explorations have proven efficient to find the configuration that maximizes performance or minimizes energy consumption. They are often implemented in autotuners, programs meant for optimizing other programs on a given design space.

**Design space characterization** is another complementary exploration strategy. It aims to understand the design space and can help programmers make better choices upfront when writing code, compiling, and executing High-Performance Computing (HPC) applications.

Like optimization, it relies on design space exploration (DSE) but does not have dedicated tooling. Characterization is mostly done with homemade scripts where the design space metric collection and algorithm are hard-coded and tightly coupled. Thus, these studies often lack portability and reusability. **Updating the different parameters is challenging** as they have **different setting methods** that may vary across different hardware architectures. **Data collection** is also difficult because it is **architecture-dependent**.

To enable a variety of exploration techniques over diverse type of parameters for both optimization and characterization, we propose a new software architecture that overcomes the shortcomings highlighted previously: the **Compilation, Runtime, and Hardware Parameters EXplorer (CORHPEX) framework**. It focuses on offline optimization and characterization of parallel applications on single-node HPC systems.

The main contributions of this paper are:

* **a method** to design project-specific but portable explorations of design spaces (Section III);

* **the CORHPEX framework** to support the method for design spaces related to hardware, runtime, and compilers (Section IV);
* the validation of the proposed approach by demonstrating its efficiency in handling even complex design spaces (Sections V-B and V-C), DSE algorithms, and data collection schemes (Section V-C), for single node applications;

* the insights to optimize applications such as static fluids simulations on Skylake and Naples systems (Sections V-B);

* the full stack optimization potential through a unified interface to set parameters from the entire compute stack that enables DSE methods to operate over compute layers never connected before;

* the training of surrogate models that accurately predict (between 86% to 99%) complex NUMA/prefetch/SMT optimization interactions, showcasing their potential for fast design exploration of HPC infrastructures., using the framework (Section V-D).

The remainder of this paper is organized as follows: Section 2 reviews related works. Section 3 presents our method. Section 4 introduces the CORHPEX framework. Section 5 details our experiments with the framework and their results. Section 6 presents perspectives and future works.

## II. RELATED WORKS

Complex HPC systems with many tunable parameters have the potential to be very fast and energy efficient. Studying all parameters together enables higher gains and a better understanding of these systems. However, few studies address more than four parameters simultaneously. Sanchez et al. [33] and TehraniJamsaz et al. [38] study thread count, thread-binding policies, page-mapping and prefetchers on Intel systems. Scravaglieri et al. [34] add the input size. Menon et al. [24] look at thread count, OpenMP parameters, algorithm choice, and application-specific parameters.

Table I presents the different parameters commonly studied in a DSE problem. Prefetchers are studied on different systems with different strategies: Jimenez et al. [18] tune IBM Power7 prefetcher parameters online. Wu et al. [41] switch all 4 Intel prefetchers on or off online. Khan et al. [21] tune the same 4 prefetchers at a finer grain online. Sanchez et al. [33] and Scravaglieri et al. [34] statically tune the 4 Intel prefetchers with profiling. Thread bindings have also been extensively optimized: Radojkov et al. [30, 29] generate manual bindings while the others [34, 33, 27, 10] identify the best policy. Page mapping is one of the most studied parameters. Dashti et al. [9] propose the carrefour mapping policy that balances memory accesses across NUMA nodes online. Piccoli et al. [26] automatically instrument loops to favor locality in page mapping with migration decisions based on page reuse. Others [34, 33, 27, 10] find the best page mapping policy per application with profiling. Compiler flags selection and order is another open DSE problem [40]. Ansel et al. [2] present it as a use case of OpenTuner auto-tuner and Menon et al. [24] find the best compiler flags for LULESH applications with a Bayesian optimization-based framework. Hall et al. [14] describe a transition from iterative compilation

to domain-specific languages and scheduling languages: the space switches from passes to code transformations. Other frequently studied parameters include Dynamic Voltage Frequency Scaling (DVFS), Uncore Frequency Scaling [23], OpenMP parameters [13, 24], algorithm choice, application-specific parameters [24, 12] and input size [34].

The main challenge to tackle larger design spaces is the execution time. Indeed, exhaustive exploration is often used to find the best configuration [13, 23, 34, 27, 12, 33], and collect data to train a model, as do Maas et al. [23]. Exhaustive search is often deemed a necessary evil and downplayed as a one-time cost. However, in practice, this cost is prohibitive in production environments as acknowledged by Durillo et al. [12]. By contrast, in the work of Maas et al. [23], the sampling techniques to speed up the model training process are outside the scope of the study. To speed up the evaluation, Popov et al. [27], Sanchez et al. [33] and Scravaglieri et al. [34] extract the critical regions to optimize and replay them a small number of times. Nevertheless, in the third study, it still takes over 3 weeks to complete the execution on multiple machines. Another solution is to use exploration algorithms such as genetic algorithms [15] or Bayesian optimization [25], that reduce the exploration time by guiding which configuration to execute next without evaluating all the configurations. Our work enables working with and elaborating on these execution time reduction strategies to study larger design spaces and break down parameter separation for greater overall gains.

Another challenge stems from the technical difficulty of collecting metrics, changing parameters or changing the algorithm. Collecting performance counters such as cache misses, percentage of remote NUMA accesses, IPC, or energy consumption, for instance, to serve as model features is common, [23, 33, 34] but it requires instrumenting applications. Multiple libraries exist for this task like the LIKWID [39] toolsuite, the PAPI [11] library, the Intel RAPL library, or the ROCm SMI library depending on the system, but the process is long and requires good knowledge of the hardware and libraries. The OpenTuner framework [2] offers a flexible approach to define the parameters to explore, but it remains verbose and its exploration algorithm and metric collection are fixed. DAKOTA [1] optimizes scientific and industrial problems by enabling flexible DSE with characteristics similar to our work. It finds appropriate physical designs by launching simulations—e.g., computational fluids dynamics, computational mechanics—and therefore is not particularly adapted to application optimization where parameters have various setting methods and timing—e.g., runtime, compile time. Our work provides flexible parameter definition and choice of exploration algorithm and metric collection scheme specifically for HPC application optimization and characterization.

DSE is also applied to hardware design. RISC-V processor designs based on simulations are proposed by Deffe [22]— which relies on DAKOTA—and A-DECA [42] while the work of Zarouar et al. [43] focuses on ARM designs. By contrast, our work studies HPC applications across the full software and hardware stack of **existing systems** on real hardware.
<table><caption>TABLE I: Related works on design spaces. Each study focuses on tuning a small set of parameters often closely related to one another. For example, compiler flags and runtime parameters such as the number of threads and DVFS are never studied together. Our work enables the study of any combination of these parameters, thus it is more flexible than any existing approach.</caption><thead><tr><th>Papers</th><th>Prefetch</th><th>Threads count</th><th>Thread bindings</th><th>Page mapping</th><th>Compiler flags</th><th>Input size</th><th>DVFS</th><th>UFS</th><th>OpenMP</th><th>Algo</th><th>Appli</th></tr></thead><tbody><tr><td>Wu et al.[41]</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Radojkov et al.[29, 30]</td><td></td><td>X</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Dashti et al.[9]</td><td></td><td></td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Jimenez et al.[18]</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Ancel et al.[2]</td><td></td><td></td><td></td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Piccoli et al.[26]</td><td></td><td></td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Diener et al.[10]</td><td></td><td>X</td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Khan et al.[21]</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Durillo et al.[12]</td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Wang et al.[40]</td><td></td><td></td><td></td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Popov et al.[27]</td><td></td><td>X</td><td>X</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Sanchez et al.[33]</td><td>X</td><td>X</td><td>X</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Menon et al.[24]</td><td></td><td>X</td><td></td><td></td><td>X</td><td></td><td></td><td></td><td></td><td>X</td><td>X</td></tr><tr><td>TehraniJamsaz et al.[38]</td><td>X</td><td>X</td><td>X</td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Scravaglieri et al.[34]</td><td>X</td><td>X</td><td>X</td><td>X</td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Dutta et al.[13]</td><td></td><td>X</td><td></td><td></td><td></td><td></td><td></td><td></td><td>X</td><td></td><td></td></tr><tr><td>Maas et al.[23]</td><td></td><td>X</td><td></td><td></td><td></td><td></td><td>X</td><td>X</td><td></td><td></td><td></td></tr></tbody></table>

Our work reduces the cost of implementing a DSE by abstracting the algorithm, the design space definition—such as the tunable parameters and how to update them—and the data collection mechanism. It also enables selecting among distinct algorithms, distinct data collection methods, or distinct design space layouts by keeping them separated. It enables to extract insights, described in the experiment section, by focusing on the actual characterization, rather than on technical details of gathering the necessary data, by acting as a turnkey data collector with hot-swappable DSE algorithms.

### III. METHOD

In this section, we propose a method to perform design space exploration. It is composed of 3 steps: (1) the definition of the exploration goals, (2) the definition of the design space, and (3) the choice of the exploration technique. These principles apply to general DSE regardless of the field.

#### A. Exploration goals

We identified two purposes of DSE: optimization and characterization. The former intends to find the point in the design space that maximizes (or minimizes) a score with the minimum number of samples. The latter intends to sample the fewest well-chosen points to capture the behavior of the score in the space. Whether we want to optimize or characterize the application will drive the choice of the algorithm.

For an application, possible scores include execution time, energy consumption, memory footprint, any performance counters from cache-miss to branch miss-prediction, or a combination, like the Energy Delay Product (EDP) that multiplies time and energy. Dedicated explorations per score are necessary, as different scores may not be correlated across the design spaces. Different contexts such as different systems also demand dedicated explorations.

#### B. Design space definition

While the problem at hand defines the design space, some parts of the space need to be redefined or dropped when performing multiple explorations in different contexts.

For example, a part of the space may exist in one context but not in another, like the AVX512 instruction set exists on an Intel Skylake CPU but not on an AMD EPYC Naples CPU. We can either choose system-specific design spaces or a common design space on the two systems.

A part of the space may also exist in different ways in two contexts: two systems can have the same number of cores but different memory hierarchies. If thread mapping related to memory hierarchy matters, we may redefine thread placement policies to take that into account.

For applications, some parts of the design space that should receive special attention in this regard are:

* application-specific parameters (e.g. block size, algorithm, precision, data layout AoS/SoA) as they vary across different applications

* inputs (size and nature) as exploring multiple one forces multiple executions

* hardware parameters and specifications—e.g. prefetchers, memory hierarchy, instruction sets—as they may vary significantly across different hardware

#### C. Design space exploration technique

There are many techniques to explore a design space in the literature and presenting a complete list is beyond the scope of this paper. Instead, we provide a quick overview of the characteristics to consider in exploration algorithms to replace exhaustive search when it becomes infeasible.

Exploration techniques are usually iterative: they evaluate one or more points of the space and use the results to choose the next point to evaluate. This decision process relies on two components: exploration and exploitation. Exploration influences the decision toward unknown areas of the space, which makes mainly explorative algorithms better suited to
characterization. *Exploitation* influences the decision toward areas of the space known to yield good scores, which makes mainly exploitative algorithms better suited to optimization.

With a fixed number of evaluations more *exploitation* means less *exploitation* and vice-versa. This trade-off can often be tuned through algorithm parameters. For optimization, some degree of exploration is necessary to avoid local extrema and, for characterization, some degree of exploitation is necessary to properly explore parts of the space with high variations.

Some algorithms are inherently exploitative—i.e. optimization algorithms—with parameters to include more exploration like the probability of mutation in genetic algorithms or the learning rate in gradient descent. Some meta-heuristics like particle swarm optimization or simulated annealing can also add more *exploration* to an optimization algorithm.

Bayesian Optimization (BO) is another more versatile algorithm, which uses a single parameter to determine the tradeoff between exploration and exploitation.

For characterization, sampling algorithms can also be used. Unlike iterative algorithms, all the points to evaluate are chosen at once to cover the whole space as uniformly as possible. Latin Hyper-square Sampling (LHS), maximin, and orthogonal arrays [17] are examples of samplings with different properties. Note that these samplings can also be used to initialize the previous algorithms to improve exploration—e.g. genetic algorithm, particle swarm.

The nature of the design space influences the choice of an exploration technique. Some algorithms are better suited to continuous and differentiable spaces, like gradient descent and orthogonal arrays, while others are better suited to less regular spaces, like genetic algorithms (GA).

# IV. THE CORHPEX FRAMEWORK

In this section, we present the principles that should underpin any implementation of a DSE, then we introduce CORHPEX, a framework that implements these principles to achieve efficient application characterization and optimization.

## A. Principles for DSE implementation

We use abstraction, modularity, and extensibility to implement a DSE as previously described.

A common **abstraction** for modules that serve the same purpose enables the creation of other modules, like new exploration techniques or other measurement methods.

**Modularity** implies a complete separation of the design space, the measurements, the exploration strategy, and the object to optimize (e.g. applications to run). It complements abstraction to reduce the cost of making changes to the DSE. For example, adding a dimension to the design space, changing the exploration strategy or even just its parameters should come at a minimal cost.

Finally, it should be **extensible** with new modules for custom measurement methods or exploration strategies. This enables users to evaluate and compare their work with others with little changes to the context of the study.

Additionally, such a framework can greatly benefit from the following features.

1) **Robustness** to interruption and **incrementality**: restarting an exploration is crucial for long executions that may span over multiple sessions or the system may crash.

2) **Reproducibility**: while randomness is rife in optimization and exploration strategies, choosing deterministic exploration algorithms (i.e. give the same results with the same input data) helps validate results.

3) **Parallel execution**: executing multiple configurations on different identical machines is a common way to speed up a DSE. Special care should still be given to the choice of the machines to make sure they have the same behavior which can be impacted by heat—e.g. position in the rack—, power consumption—e.g. power controller, power source—, age, and other physical properties [8].

4) **Configurability**: defining which modules to use and with what parameters for each part of the DSE should require little code and be readable.

5) **Automatic instrumentation**: instrumenting applications to collect performance counters and execution times is a long and repetitive process, automatic instrumentation drastically reduces the overhead of code preparation and enables the use of more diverse and complex codes.

## B. Framework design

With these principles in mind, we designed the CORHPEX framework to explore and optimize applications' behavior with various compiler, runtime, and hardware parameters. CORHPEX is an open-source software available at https://gitlab.inria.fr/corhpex/corhpex.

The core of CORHPEX is a Python3 library that exposes exploration techniques as *Explorer* objects and metric retrieval methods as *Aggregator* objects.

The exploration is configured in a TOML (Tom's Obvious Minimal Language) file which maps to a dictionary. It is composed of 4 sections: the algorithm parameters, the metrics, the benchmarks and applications, and the design space.

Figure 1 shows the software architecture of the framework with non-exhaustive examples of parameters, types of applications, metric collection schemes and exploration algorithms.

1) *Exploration algorithms:* The library provides 3 exploration algorithms: exhaustive search, genetic algorithm, and Bayesian optimization but others can easily be added. An algorithm determines the next configuration to evaluate from the design space and executes the application.

Each algorithm has its own parameters which can be tuned in the configuration file. Listing 1 shows an example of parameter definition for the genetic algorithm (GA).

2) *Exploration parameters and metrics:* The metrics monitored during the execution of applications are described in the *metrics* section. Global metric parameters include the number of meta-repetitions (*meta-rep*) that will be performed and the statistics to compute (*stat_fn*). Indeed, random noise and perturbations can happen during the execution. Thus, statistics extracted from multiple evaluations with the
<figure><img src="image_2.png" /><figcaption>Fig. 1: CORHPEX framework architecture.</figcaption></figure>

```
1 algo = "ga"
2
3 [algo_params]
4 pop = 10 # population size of a generation
5 gen = 10 # number of generations
6 cross = 0.8 # probability of crossover
7 mut = 0.2 # probability of mutation
8 target = "ener" # score metric described in [measure]
9 target_stat = "med" # use the median of the score
```

## Listing 1: Algorithm parameters for GA

same configuration are more representative of the application behavior than a single measure.

Metrics are grouped by their collection method. Listing 2 shows an example with LIKWID (`metrics.likwid`) and computed metrics (`metrics.computed`). LIKWID allows to monitor groups of counters, in the example the `ENERGY` group will be monitored. By default, LIKWID monitors the whole application. It is also possible to use the LIKWID API to monitor target code sections. Computed metrics are simple expressions built from previously defined metrics.

In addition, CORHPEX offers an automatic instrumenter that inserts probes at runtime around OpenMP parallel regions through OMPT hooks. CORHPEX uses the OMPT callback interface to call LIKWID marker probes to start and stop whenever a parallel region starts and finishes. We differentiate calls from different parallel regions by recording each parallel region using a dedicated LIKWID region name. We can thus use it for fine profiling and tuning on complex OpenMP applications without modifying any source code manually.

3) **Benchmark and application description:** We group applications by benchmarks. Listing 3 shows how to describe a benchmark. During the exploration, the placeholder `<compile_flags>` will be replaced by the parameters corresponding to compiler flags in the design space.

A benchmark contains a list of applications. Either the benchmark or the application need a compilation command. The execution command can have 2 placeholders `<exec_flags>` for the execution flags of the design space and `<variants>` which enables the execution of the same application with different parameters unrelated to the optimiza-

```
1 [measure]
2 meta_rep = 3
3 stat_fn = ["mean", "med"] # compute mean and median
4
5 [metrics.likwid]
6 groups = ["ENERGY"]
7 use_api = true
8
9 [[metrics.likwid.metrics]]
10 id = "energy"
11 section = "TABLE,Region Compute,Group 1 Metric"
12 groups = ["ENERGY"]
13 fields = ["Energy PKG [J]"]
14 cols = [1]
15
16 [[metrics.likwid.metrics]]
17 id = "time"
18 # ...
19
20 [[metrics.computed]]
21 id = "edp"
22 expr = "energy*time"
```

## Listing 2: Exploration metrics.

```
1 [[benchmarks]]
2 name = "Vecto friendly benchmark"
3 id = "vf-bench"
4 root_dir = "physic-bench"
5 compile_cmd = "mkdir -p build-vf;cd build-vf;rm -rf
  -> ./;cmake -DWITH_VECTO:BOOL=ON -DWITH_OMP=ON
  -> -DWITH_LIKWID=ON -DCMAKE_BUILD_TYPE=NoFlags
  -> -DCMAKE_CXX_FLAGS='-march=native -funroll-all-loops
  -> <compile_flags>' ..;make;cd -'
```

## Listing 3: Benchmark declaration.

tion space—e.g., with different datasets.

Listing 4 shows the description of an application with one variation called "rand" that specifies the input file `../datasets/rand1_input_0-2`.

```
1 [[benchmarks.apps]]
2 name = "Capillary Pressure Law"
3 id = "cp"
4 root_dir = "build-vf"
5 exec_cmd = "bench --kernel=cp --iter=100000 --size=10032
  -> <exec_flags> <variants>"
6 variants = ["../datasets/rand_1_input_0-2"]
7 variant_names = ["rand"]
```

## Listing 4: Application declaration.

Applications are identified by a string that contains the benchmark, the application, and the variant names.

4) **Design space description:** From characterization and optimization studies presented in Table I, we identified that parameters fall into 4 categories: (1) execution flags, (2) compiler flags, (3) environment variables and (4) environment commands. Each category of parameters must be set at a different moment and with a different method. Grouping parameters in such a way makes it easier to set up the environment and build the command to run an application with the parameters. Listing 5 shows the declaration of an execution flag with multiple values. Parameters can have an identifier `id` used in configuration names.

For more flexibility, `Explorer` objects accept a pruning function, enabling users to force any DSE algorithm to ignore
<table><caption>TABLE II: Characteristics of experiments.</caption><thead><tr><th></th><th>Design Space</th><th>Measurement</th><th>DSE algorithms</th><th>Task</th><th>System</th></tr></thead><tbody><tr><td>CP</td><td>runtime, compiler</td><td>execution (LIKWID)</td><td>exhaustive exploration</td><td>characterization</td><td>Naples and Skylake</td></tr><tr><td>OPT</td><td>runtime and hardware</td><td>read data from previous execution</td><td>genetic algorithm and Bayesian optimization</td><td>optimization</td><td>Skylake</td></tr><tr><td>MOD</td><td>runtime and hardware</td><td>read data from previous execution</td><td>Bayesian optimization</td><td>modeling</td><td>Skylake</td></tr></tbody></table>

1 [exploration-space]
2
3 [[exploration-space.execflags]]
4 name = "--fpstore"
5 values = ["float", "double"]
6
7 [[exploration-space.compileflags]]
8 name = "-ffast-math"
9
10 [[exploration-space.envvars]]
11 name = "OMP_NUM_THREADS"
12 id = "th"
13 values = [1, 4, 8, 16, 32, 64]
14
15 [[exploration-space.envcmd]]
16 name = "sudo /usr/sbin/wrmsr -a 0x1A4"
17 id = "prefetch"
18 values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
19 reset = 0

Listing 5: Exploration flags declaration.

some parts of the space. Pruning relies on testing parameter values and combining them with Boolean logic.

An Explorer can also train a surrogate model during the DSE and enable users to export it through options in the [algo-params] section of the configuration.

5) Library usage: Listing 6 shows a simple use of the framework with a custom aggregator.

1 import corhpex
2 from CustomAggregator import CustomAggregator
3
4 # Register custom aggregator
5 corhpex.register_aggregator("cust", CustomAggregator)
6
7 # Create the exploration configuration from the config file
8 # The config file contains a [metrics.cust] section to use
  -> the custom aggregator
9 explo_config =
  -> corhpex Configuration("path/to/my/config.toml")
10 # Create the explorer from the configuration
11 explorer = corhpex.explorer_from(explo_config)
12 # Run the exploration
13 explorer.run()

Listing 6: Basic script to use the framework. It registers a custom aggregator—i.e., written by the user—reads the configuration file and runs the exploration.

HPC parameters are complex, and vastly different in nature (compiler flags, thread mapping, prefetching) and thus require separate dedicated infrastructure to optimize. This results in limited decisions when selecting parameters at each stage, leading to missed optimization opportunities. On the contrary, CORHPEX proposes a unified interface to set parameters from the entire compute stack. It enables DSE methods to operate over compute layers never connected before—e.g. SIMD, NUMA and prefetch—in a straightforward manner.

V. EVALUATION

In this section, we detail three distinct and complementary experiments to showcase the method and the possibilities CORHPEX offers. The first one, CP, is a focused study of the Capillary Pressure kernel that demonstrates the ability to change very different parameters and collect metrics for analysis. The second one, named OPT, compares two optimization algorithms on the same design space using previously collected data. The last one, named MOD, evaluates surrogate models trained with few samples selected with Bayesian Optimization. Table II summarizes these experiments.

A. Experimental Setup

<table><caption>TABLE III: Characteristics of Skylake and Naples systems.</caption><thead><tr><th></th><th>Skylake</th><th>Naples</th></tr></thead><tbody><tr><td>Packages</td><td>2</td><td>2</td></tr><tr><td>NUMA nodes/package</td><td>1</td><td>4</td></tr><tr><td>Physical cores/node (logical)</td><td>16 (32)</td><td>16 (32)</td></tr><tr><td>Memory Size (GB)</td><td>192</td><td>128</td></tr><tr><td>Memory type</td><td>DDR4</td><td>DDR4</td></tr><tr><td>Memory frequency</td><td>2666TR/s</td><td>2666TR/s</td></tr><tr><td>L1d size (KB)</td><td>32</td><td>32</td></tr><tr><td>L2 size (KB)</td><td>1024</td><td>512</td></tr><tr><td>L3/node</td><td>1</td><td>2</td></tr><tr><td>L3 size (MB)</td><td>22</td><td>8</td></tr><tr><td>Prefetchers</td><td>4</td><td>1</td></tr></tbody></table>

1) Hardware: The following experiments were performed on an Intel Xeon Gold 6130 and an AMD EPYC 7301. We refer to these two systems as Skylake and Naples respectively. Their characteristics are presented in Table III. Aside from the cores' microarchitectures, the main differences between the two systems are the memory hierarchy and the control of prefetchers. Indeed, Skylake has a flat hierarchy, packages are NUMA nodes with a single L3 cache, while Naples has many levels with four NUMA nodes per package and two L3 caches per node. The Skylake also has bigger caches than the Naples but fewer L3 caches. Skylake has four prefetchers that can be activated independently, while Naples only has one. As a result, Naples is a good system to study thread binding with its multilevel hierarchy, while Skylake is more suited to study prefetching with its large L3 caches shared among 16 cores and 16 prefetcher configurations.

Both systems support simultaneous multithreading (SMT), with two logical cores per physical core. The number of threads and the thread-binding policy both influence SMT.

2) Memory-aware binding policies: Traditional thread-binding policies are *compact* and *scatter*. Popov et al. [27] also introduce the *contiguous* policy that behaves like *compact* with threads balancing on available NUMA nodes. These bindings only take into account NUMA nodes and overlook the rest of
<table><caption>TABLE IV: Examples of thread placements with memory aware thread binding policies.</caption><thead><tr><th rowspan="3">Policy</th><th rowspan="3">Th.</th><th colspan="15">Package 0</th><th colspan="15">Package 1</th></tr><tr><th colspan="5">Die 0</th><th colspan="5">Die 1</th><th colspan="5">Die 2</th><th colspan="5">Die 3</th></tr><tr><th>C 0</th><th>C 1</th><th>C 2</th><th>C 3</th><th>C 4</th><th>C 5</th><th>C 6</th><th>C 7</th><th>C 8</th><th>C 9</th><th>C 10</th><th>C 11</th><th>C 12</th><th>C 13</th><th>C 14</th><th>C 15</th></tr></thead><tbody><tr><td>1111</td><td>32</td><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>10</td><td>11</td><td>12</td><td>13</td><td>14</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>1111</td><td>8</td><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>10</td><td>11</td><td>12</td><td>13</td><td>14</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>1110</td><td>16</td><td>0</td><td>2</td><td>1</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>10</td><td>9</td><td>11</td><td>12</td><td>14</td><td>13</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>0110</td><td>16</td><td>0</td><td>2</td><td>2</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>10</td><td>9</td><td>11</td><td>12</td><td>14</td><td>13</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>1011</td><td>16</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>11</td><td>12</td><td>1</td><td>3</td><td>5</td><td>7</td><td>9</td><td>11</td><td>13</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>0000</td><td>16</td><td>0</td><td>8</td><td>4</td><td>12</td><td>10</td><td>11</td><td>12</td><td>1</td><td>3</td><td>5</td><td>7</td><td>9</td><td>11</td><td>13</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>0000</td><td>8</td><td>0</td><td>8</td><td>4</td><td>12</td><td>10</td><td>11</td><td>12</td><td>1</td><td>3</td><td>5</td><td>7</td><td>9</td><td>11</td><td>13</td><td>15</td><td>16</td><td>17</td><td>18</td><td>19</td><td>20</td><td>21</td><td>22</td><td>23</td><td>24</td><td>25</td><td>26</td><td>27</td><td>28</td><td>29</td><td>30</td><td>31</td></tr><tr><td>scatter</td><td>8</td><td>0</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>1</td><td>5</td><td>5</td><td>5</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td></tr></tbody></table>

the memory hierarchy. Depending on the processor, a NUMA node can have one or more L3 cache, and having 2 cores on different NUMA nodes is a fuzzy indication of distance because the package also matters. From these observations, we conclude that compacting or scattering on NUMA nodes is not enough to fully take advantage of the hardware topology.

To account for all levels of the memory hierarchy, we suggest compacting and scattering—i.e., filling first and last— at the level of (1) packages, (2) dies, (3) L3 caches, and (4) SMT. This family of 16 policies enables finer control of thread placement while still keeping it systematic, focusing on threads sharing memory resources rather than sharing memory controllers only. This relates to the work of Swartvager et al. [37] on MPI process placement. For a given processor, a policy is a list of core contexts where the threads should be assigned in order, but depending on the actual number of threads executed, all the contexts of the list may not be used.

Each policy has four parameters, one per memory level, that can take 2 values First (1) or Last (0). At each level, if the parameter is First, one object must be filled before assigning threads to the next object at the same level, if the parameter is Last, threads are distributed in a round-robin fashion on the objects of this level. When building the list of contexts, the policy indicates the next package on which the next context is located, then the next die—i.e., NUMA node—, then the next L3, and finally the next core.

Table IV shows a few examples of thread placement with this group of policies on a hypothetical architecture. A policy is identified by the 4 values of the parameters, e.g. Package-first Die-first L3-first SMT-first is 1111—i.e., 15—, or Package-first Die-first L3-first SMT-last is 1110—i.e., 14.

## B. CP: Characterization of the Capillary Pressure with exhaustive search

We manually extracted several kernels from a carbon storage simulation using static fluids physics. The insights gathered over these proxy codes with CORHPEX serve to optimize real-world problems in geoscience applications. Here, we present the results obtained with the Capillary Pressure kernel (CP). Its control flow includes branches, which can interfere with vectorization (the use of SIMD instructions) and it uses division and exponentiation instructions which introduce long latencies. This characterization study is performed on the isolated kernel with an exhaustive exploration of small

subspaces to demonstrate CORHPEX capabilities to update and measure widely different optimization spaces.

1) Thread count and thread-binding policies: To explore the thread-binding policies, we executed this kernel on the Naples system with the compiler flags `-O3` and `-march=native`—i.e., AVX2 ISA is used. The speedup and energy savings are computed against a reference configuration with 32 threads and a policy that binds threads to maximize cache and memory sharing for consecutive threads. We use a scatter factor as an indication of the mapping policy without the SMT bit. Increasing the scatter factor minimizes memory sharing between consecutive threads and vice versa. The design space is described in the following table:

<table><thead><tr><th>Parameters</th><th>Values</th></tr></thead><tbody><tr><td>Number of threads</td><td>8, 16, 32, 64</td></tr><tr><td>Thread binding policies (scatter factor)</td><td>0-15</td></tr></tbody></table>

In Figure 2, most configurations with 64 threads and 8 threads have a speedup and energy savings value below 1, meaning they are neither fast nor energy efficient. The most energy-efficient and fastest configurations is in the top right corner. When using 16 threads it is better to compact threads on a single package (scatter factor below 4), while when using 32 threads it is better to scatter them on the whole system (scatter factor greater than 4). It shows that the number of threads and the binding policy should be optimized together.

In practice, for this kernel on Naples, peak performance requires the use of both packages (32 threads) while peak energy optimization needs only one (16 threads). The speedup from the second package does not compensate for the extra power consumption. We also note that SMT is not beneficial.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Speedup and energy savings of capillary pressure kernel on Naples with 8, 16, 32, and 64 threads and 16 thread placements indicated by scatter factor.</figcaption></figure>
2) Instruction set architecture, number of threads, and thread binding policy: To understand how different instruction sets interact with thread parameters, we measured the execution time of the Capillary Pressure kernel on the Skylake system on the design space presented in the following table.

<table><thead><tr><td>Parameters</td><td>Values</td></tr></thead><tbody><tr><td>Number of threads</td><td>8, 16, 32, 64</td></tr><tr><td>Thread binding policies</td><td>0-15</td></tr><tr><td>ISA</td><td>SSE4, AVX2</td></tr></tbody></table>

Figure 3 shows that using 64 threads yields slightly better performance than 32 threads, but we can anticipate lower energy gains. While prioritizing SMT yields the best performance, some configurations where SMT is not prioritized with 8, 32, or 64 are among the top-performing ones. For this kernel on Skylake, best configurations with SSE4 and AVX2 at each thread count yield similar performance. Although they were not measured, energy gains are likely to be higher for the SSE4 ISA than for the AVX2 ISA. This suggests that the choice of ISA, the number of threads, and the thread mapping policy should be optimized together for higher gains.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Speedup of capillary pressure kernel using AVX2 and SSE4 ISA on Skylake architecture with 8, 16, 32, and 64 threads and 16 thread placements.</figcaption></figure>

3) Precision: Finally, we studied how the precision interacts with thread parameters, by executing the Capillary Pressure kernel on the Skylake system. Precision can be changed on this benchmark using execution flags. In mixed precision, values are stored as `float` but computation is performed with `double`. The code was compiled with the `-O3` and `-march=native` flags to use the AVX512 ISA. The following table presents the design space studied:

<table><thead><tr><td>Parameters</td><td>Values</td></tr></thead><tbody><tr><td>Number of threads</td><td>8, 16, 32, 64</td></tr><tr><td>Thread binding policies</td><td>0-15</td></tr><tr><td>Precision</td><td>float, double, mixed</td></tr></tbody></table>

In Figure 4, the reference configuration for the speedup is 32 threads with maximum cache and memory sharing for consecutive threads, and storage and computing in double precision. With 32 threads, a subset of configurations is faster than the reference using only float types for storage and computation. For this kernel, using mixed precision—i.e., double for computing and float for storage—is not beneficial: it performs worse than storage and computing in double precision. The overhead of conversion is not compensated by a faster computation in float probably because the kernel is

very short. The fastest and the slowest configurations both do not prioritize SMT, which confirms the necessity to optimize multiple parameters together for higher gains.

Simple, mixed, and double precision increasingly pressure the memory, thus affecting the effectiveness of SIMD. Various kernels exhibit different levels of sensitivity to this stress. Contrary to expectations, mixed precision did not provide gains for this kernel.

<figure><img src="image_5.png" /><figcaption>Fig. 4: Speedup of capillary pressure kernel using float and double storage and computing precision on Skylake architecture with 8, 16, 32, and 64 threads and 16 thread placements.</figcaption></figure>

In many cases exploration algorithms with exhaustive search becomes infeasible. In the next experiment, we show the proposed efficient alternative to overcome this problem.

### C. OPT: Optimization of OpenMP parallel regions and data reuse

This experiment compares two optimization algorithms, genetic algorithms (GA) and Bayesian optimization (BO), for two optimization goals, execution time and energy consumption, on a large design space described in Table V.

We investigate the following research questions:

* **RQ1:** How does, optimizing time or energy, affect the search algorithm?

* **RQ2:** How suited are these algorithms to find the optimal configurations?

* **RQ3:** How quickly do these algorithms converge on good or optimal configurations?

We answer these questions by leveraging a public dataset [35] that has evaluated the entire optimization space described in Table V on a Skylake system. In particular, it includes the exhaustive performance and energy measurement of 48 OpenMP parallel regions from representative HPC benchmarks including the Rodinia Benchmark Suite [7] (v.3.1), the NAS Parallel Benchmarks [3] (NPB v.3.0 in C), the PARSEC [5] benchmarks, along with LULESH [20, 16, 19] (v.2) and CLOMP [6]. They include linear algebra—e.g. linear equations solving—, machine learning—i.g. backpropagation, kmeans clustering—, physics—e.g. heat propagation—and algorithms—e.g. pathfinding.

Figure 5 describes all the regions and quantifies their importance with respect to the parallel application runtime. By studying less than a third of the regions in dataset (the 48 regions), we cover 87% of the parallel execution time. They
<figure><img src="image_6.png" /><figcaption>Fig. 5: Left: Parallel region coverage—i.e., region’s duration divided by parallel execution time. Regions are named with their file, function, and line if necessary. Right: ratio of the parallel execution time covered by studying the 48 parallel regions. To collect the coverage of the open source dataset, we profiled its applications over all 143 parallel regions on the default configuration. We measured the total execution time (sequential plus parallel) as well as the ratio of execution time spent in OpenMP parallel regions. We want to minimize the number of region we study while maximizing the coverage.</figcaption></figure>

were pruned based on their execution time (representing less than 3% according to the dataset) and energy consumption stability. The parallel execution represents 69% of the total execution. This is due to a few benchmarks—e.g., pathfinder, bfs—performing many serial data operations around the kernel: while significant, these benchmarks do not profile them, they specifically benchmark the parallel execution. Clomp is the only application for which we capture less than 65% of the parallel execution: we only study 5 of its 25 regions.

Note that each region was optimized in isolation: we can thus estimate how quickly GA or BO converge on its best configuration. We used the Pyeasyga [28] and the bayesian-optimization [4] libraries to implement GA and BO respectively with a custom data aggregator that collects the execution time and energy consumption from the open-source data files.

<table><caption>TABLE V: Large design space to study HPC applications on a NUMA system.</caption><thead><tr><td>Parameters</td><td>Values</td></tr></thead><tbody><tr><td>Number of threads</td><td>16, 32, 64</td></tr><tr><td>Thread binding policies</td><td>scatter, compact, contiguous</td></tr><tr><td>Page mapping</td><td>first touch, interleave, balanced</td></tr><tr><td>Prefetchers</td><td>0-15</td></tr><tr><td>Multithreading</td><td>yes, no</td></tr></tbody></table>

Fine-tuning these regions provide substantial performance and energy gains over the default configuration (32 threads, SMT last, 2 NUMA nodes, scatter thread mapping policy, first-touch data mapping policy and all prefetchers on): 1.9x and 7x respectively at the expense of a very resource consuming exploration. Figure 6 demonstrates how CORHPEX mostly preserve these gains and shows GA and BO’s behaviours. For lack of space, we don’t include the detailed results.

The first swarm plot presents the best scores—i.e., the best configuration discovered through exhaustive search divided by the best configuration returned by the algorithm—achieved by GA and BO for performance or energy optimization. BO yields better results than GA to optimize energy consumption: the mean score across all regions is 0.94 against 0.90 for GA, it reaches the absolute best configuration—i.e., score is

<figure><img src="image_7.png" /><figcaption>Fig. 6: Best scores obtained with GA and BO over 48 parallel regions when optimizing performance and energy and number of executions to obtain the best score, 97.5% and 95% of the best score. A score of 1 means that the optimal was found.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7: Best scores obtained with GA and BO for MG psinv region (NPB).</figcaption></figure>

1.0—for 34 regions against 31 for GA, and it reaches 0.975 or above for 41 regions against 36 for GA. Both algorithms fail to find a decent configuration for one region. GA yields better results to optimize performance: the mean score is 0.998 against 0.993 for BO, it finds the absolute best configuration
<figure><img src="image_9.png" /><figcaption>Fig. 8: The ratio of configurations that satisfy 97.5% and 95% of the optimal energy consumption and execution time.</figcaption></figure>

for 35 regions against 23 for BO, and it reaches 0.975 or above for all regions while BO does it for all regions except one. This difference can be explained by Figure 8 which shows the ratio of configurations that reach 97.5% or 95% of the possible gains-i.e., scores of 0.975 resp. 0.95 or higher. The lower the ratio, the more difficult the target score is to achieve. The ratio for 95% of possible gains in energy is less than 0.05 for 41 regions. For 95% of possible gains in performance, it is less than 0.05 for 25 regions and less than 0.1 for 42 region. Thus, both performance and energy are difficult to optimize but energy is the hardest. The most difficult region to optimize is one from LULESH, with a ratio of 0.0013 for 95% of maximum energy gains and 0.0052 for 95% of maximum performance gains. BO and GA failed to optimize it for energy and BO only reached 0.93 for performance.

The second swarm plot shows the number of iterations each algorithm took to reach its best score. Iterations can resample points. Note that BO uses 10 random points to initialize-i.e., first 10 iterations-and GA has generations of 100 individuals, the first 100 iterations are therefore random. We capped BO at 778 iterations which is the number of possible configurations. For energy optimization, GA finds its best configuration in the first 100 iterations-i.e., randomly-for 10 regions while BO does only for 9 regions. With 200 iterations, GA has its best for 22 regions while BO does for 25 regions. Therefore, BO finds better energy scores and faster than GA. For performance, GA finds its best configuration in the first 100 iterations for 18 regions while BO does for only 10 regions. GA needs over 300 iterations to find its best score for 15 regions while for BO it is 25 regions. This shows that the difficulty is only partly quantified by the ratio we introduced in Figure 8. The lower the ratio the less likely it is that a random configuration is exceptionally good. However, this metric does not quantify how difficult the space is to navigate. This is reflected by the higher number of iterations to reach the best performance score compared to the best energy score.

The last two swarm plots show the number of iterations each algorithm took to reach 95% and 97.5% of the best possible score. They show the same tendencies as the previous one: GA finds many good configurations randomly-i.e., 34 regions for 95% of best energy gains and 42 regions for 95% of best performance gains-, both BO and GA are faster at finding good configurations for energy than for performance.

Both GA and BO tend to find better scores for performance than energy. This answers **RQ1**: energy is more difficult to

optimize than time for both GA and BO. Then, we see that GA finds better scores than BO for performance but not for energy optimization, which answers **RQ2**. As for **RQ3**, GA requires fewer iterations than BO to produce a good score thanks to a large initial random sample.

Figure 7 shows how the 2 algorithms progress toward the best energy score they can find on the mg region. GA finds a good random configuration early in the first generation then gains slow down and stall at 92% until the best configuration is hit after 465 executions. BO quickly finds a configuration that yields 88% of the best score at iteration 52, then improves to 96% at iteration 120 but stalls at 99% after iteration 286.

**CORHPEX enabled this comparison at a low cost.**

In fact, only the [algo-param] part of the configuration changed between the two evaluations.

We conclude that, for DSE in HPC systems, BO achieves a good tradeoff between overhead (number of samples) and total gains for energy but GA's randomness give faster and better results for performance.

In the next experiment, we propose to tune BO to build surrogate models and characterize the design space.

<figure><img src="image_10.png" /><figcaption>Fig. 9: Streamcluster 451 region performance surrogate model accuracy: configurations 0 to 291 have 16 threads, 292 to 578 have 32 threads, and 578 to the end have 64 threads. The spikes at 400, 500, 640, and 740 are due to the use of two NUMA nodes instead of one.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 10: FFT step 3 region performance surrogate model accuracy: configurations 0 to 291 have 16 threads, 292 to 578 have 32 threads, and 578 to the end have 64 threads. The spikes at 400, 500, 640, and 740 are due to the use of two NUMA nodes instead of one.</figcaption></figure>

**D. MOD: Modeling OpenMP parallel regions with Bayesian optimization**

In this experiment, we train surrogate models for the same 48 parallel regions to predict speedup and energy savings from
<figure><img src="image_12.png" /><figcaption>Fig. 11: Absolute error of models predictions of speedup and energy savings for all regions.</figcaption></figure>

a reference configuration. We investigate the surrogate model that BO trains during the exploration. We study the same design space as the previous experiment described in Table V and using the same open-source data.

In Figure 11, we can see that models that predict performance are generally more accurate than models that predict energy, which confirms that energy optimization, characterization, and prediction are more difficult than performance. The mean accuracy across all regions is 93% for speedup and 86% for energy while only profiling 18% of the space (150 iterations). With 9% (50 iterations), the speedup prediction accuracy was already acceptable at 89%. To the best of our knowledge, it is the first time that surrogate models have been trained to predict speedups and energy gains on a space combining NUMA and prefetchers parameters.

We evaluated different surrogate models from the Surrogate Modeling Toolbox [36], including Kriging, radial basis function, inverse distance weighting, and Least-squares. In the rest of the paper, we use only Kriging methods as they provided the highest accuracy without tuning.

Models' accuracy range between 86% and 99% depending on the application. A model of a region performing FFT with 98% accuracy is shown in Figure 10. The left plot shows that predictions are close to the real measures. On the right, the model displays difficulty in predicting high speedups, probably because high-speedup configurations were less sampled during training. A model of a region from Streamcluster with a 96% accuracy is presented in Figure 9. On the left, model predictions are further away from measures. On the right, predicted speedups have an offset compared to the real measure and the same increased difficulty in predicting high speedups. However, the Streamcluster model retains the order of speedups despite the offset while the FFT model underestimates top-performing configurations predicting lower speedups than slightly worse-performing configurations.

CORHPEX enabled us to focus on the implementation and tuning of these surrogate models. The training time with executions is significantly reduced, from 500h (3 weeks) to little more than 90h (4 days). It demonstrated that BO-based surrogate models can model the complex interaction of NUMA and prefetch optimizations, thus paving the way for modeling even more complex optimization spaces.

## VI. DISCUSSION AND FUTURE WORKS

The CORHPEX framework was designed to handle any combination of execution parameters, compiler parameters, en-

vironment variables, and commands. It is agnostic of libraries, hardware, language, compile chain, and runtime. While this paper focuses on single node applications, CORHPEX supports more diverse explorations such as distributed computing applications—e.g., MPI—or/and heterogeneous systems—e.g., GPU—. Parallel execution on different configurations of identical systems is also supported to speedup explorations.

CORHPEX collects performance counters using probes which in turn are processed into meaningful insights for the developers. Our current implementation uses LIKWID. We can switch the probe calls for another monitoring tool—e.g., PAPI—to adapt to other specific system constraints.

This probe based mechanism enables us to study the impact of optimization on the whole application or per region. While locally beneficial, loop [32, 31] and region based optimizations can slowdown other regions. In details, Popov et al. [27] (in Figure 11) quantify the loss of optimizing NUMA effects on a parallel region basis (as we did) against optimizing global applications. When considering the whole application, the average gains are reduced by 12% and to up to 70% for some applications. Different parameters have different transition costs (prefetch or compiler changes are less costly than page mapping). Modeling the interplay between fine-tuned regions is a future work that we can consider with surrogate models to co-optimize multiple regions of an application.

We demonstrated the benefits of genetic algorithms and Bayesian Optimization over exhaustive search to optimize and model different parameters. CORHPEX further enables users to implement search methods—e.g., reinforcement learning— to better fit their needs and may integrate them.

Finally, parallel evaluation of configurations on multiple nodes is not implemented. However, it is still achievable with a launch script. Note that some exploration algorithms are not appropriate candidates for such evaluations—e.g., BO—and some applications neither—e.g., MPI applications.

CORHPEX was instrumental in characterizing and modeling the impact of different parameters on a given application with surrogate models. As long-term future work, we plan to improve and augment surrogate models by leveraging the transfer learning capabilities of code embedding.

## VII. CONCLUSION

In this work, we have presented a *design space exploration* method, and the CORHPEX framework to support this method for exploring kernels and applications design spaces through hardware, compiler, and runtime parameters. We illustrated the possibilities offered by the framework in three experiments with diverse design spaces on two different systems, three exploration algorithms, and two data aggregation methods. Switching between these experiments was made easy by the abstraction and modularity of the framework. In one of our experiments, we presented surrogate models to predict speedup and energy savings on a large design space of NUMA and prefetcher parameters, with a mean accuracy of 93% for speedup and 86% for energy, while sampling only 18% of the design space.
REFERENCES

[1] B. M. Adams, W. J. Bohnhoff, K. R. Dalbey, M. S. Ebeida, J. P. Eddy, M. S. Eldred, R. W. Hooper, P. D. Hough, K. T. Hu, J. D. Jakeman, M. Khalil, K. A. Maupin, J. A. Monschke, E. E. Prudencio, E. M. Ridgway, A. A. Rushdi, D. T. Seidl, J. A. Stephens, L. P. Swiler, and J. G. Winokur. *Dakota 6.18.0 documentation*. Albuquerque, NM, USA, 2023. URL: https://snl-dakota.github.io/docs/latest_release/users/index.html.

[2] Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May O'Reilly, and Saman Amarasinghe. "OpenTuner: an extensible framework for program autotuning". In: *Proceedings of the 23rd International Conference on Parallel Architectures and Compilation*. PACT '14. Edmonston, AB, Canada: Association for Computing Machinery, 2014, pp. 303–316. ISBN: 9781450328098. DOI: 10.1145/2628071.2628092.

[3] D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, H. D. Simon, V. Venkatakrishnan, and S. K. Weeratunga. "The NAS Parallel Benchmarks — Summary and Preliminary Results". In: *Proceedings of the 1991 ACM/IEEE Conference on Supercomputing. Supercomputing '91*. New York, NY, USA: ACM, 1991, pp. 158–165. ISBN: 978-0-89791-459-8.

[4] bayesian-optimization. https://bayesian-optimization.github.io/BayesianOptimization/index.html. Accessed: 2024-07-29.

[5] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. "The PARSEC benchmark suite: Characterization and architectural implications". In: *Proceedings of the 17th international conference on Parallel architectures and compilation techniques*. 2008, pp. 72–81.

[6] Greg Bronevetsky, John Gyllenhaal, and Bronis R De Supinski. "CLOMP: accurately characterizing OpenMP application overheads". In: *International Workshop on OpenMP*. Springer. 2008, pp. 13–25.

[7] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and Kevin Skadron. "Rodinia: A Benchmark Suite for Heterogeneous Computing". In: *IEEE International Symposium on Workload Characterization*. IISWC. Oct. 2009, pp. 44–54.

[8] Tom Cornebize. "Calcul haute performance : vers de meilleures prédictions de performance et expériences". 2021GRALM016. PhD thesis. 2021. URL: http://www.theses.fr/2021GRALM016/document.

[9] Mohammad Dashti, Alexandra Fedorova, Justin Fuston, Fabien Gaud, Renaud Lachaize, Baptiste Lepers, Vivien Quema, and Mark Roth. "Traffic Management: A Holistic Approach to Memory Placement on NUMA Systems". In: *18th International Conference on Architectural Support for Programming Languages and Operating Systems*. ASPLOS '13. New York, NY, USA: ACM, Mar. 2013, pp. 381–394. ISBN: 978-1-4503-1870-9. DOI: 10.1145/2451116.2451157.

[10] Matthias Diener, Eduardo H.M. Cruz, Laércio L. Pilla, Fabrice Dupros, and Philippe O.A. Navaux. "Characterizing communication and page usage of parallel applications for thread and data mapping". In: *Performance Evaluation* 88-89 (2015), pp. 18–36. ISSN: 0166-5316. DOI: 10.1016/j.peva.2015.03.001.

[11] Jack Dongarra, Kevin London, Shirley Moore, Phil Mucci, and Dan Terpstra. "Using PAPI for hardware performance monitoring on Linux systems". In: *Conference on Linux Clusters: The HPC Revolution*. Vol. 5. Linux Clusters Institute. 2001.

[12] Juan J. Durillo, Philipp Gschwandtner, Klaus Kofler, and Thomas Fahringer. "Multi-Objective region-Aware optimization of parallel programs". In: *Parallel Computing* 83 (2019), pp. 3–21. ISSN: 0167-8191. DOI: 10.1016/j.parco.2018.03.010.

[13] Akash Dutta, Jordi Alcaraz, Ali TehraniJamsaz, Eduardo Cesar, Anna Sikora, and Ali Jannesari. "Performance Optimization using Multimodal Modeling and Heterogeneous GNN". In: *Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing*. HPDC '23. Orlando, FL, USA: Association for Computing Machinery, 2023, pp. 45–57. ISBN: 9798400701559. DOI: 10.1145/3588195.3592984.

[14] Mary Hall, Cosmin Oancea, Anne C Elster, Ari Rasch, Sameeran Joshi, Amir Mohammad Tavakkoli, and Richard Schulze. "Scheduling languages: A past, present, and future taxonomy". In: *arXiv preprint arXiv:2410.19927* (2024).

[15] John H Holland. "Genetic algorithms". In: *Scientific american* 267.1 (1992), pp. 66–73.

[16] Rich Hornung, Jeff Keasler, and Maya Gokhale. *Hydrodynamics Challenge Problem, Lawrence Livermore National Laboratory*. Tech. rep. LLNL-TR-490254. Livermore, CA, pp. 1–17.

[17] Wojciech Jarosz, Afnan Enayet, Andrew Kensler, Charlie Kilpatrick, and Per Christensen. "Orthogonal Array Sampling for Monte Carlo Rendering". In: *Computer Graphics Forum* 38.4 (2019), pp. 135–147. DOI: 10.1111/cgf.13777. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13777.

[18] Víctor Jiménez, Francisco J. Cazorla, Roberto Gioiosa, Alper Buyuktosunoglu, Pradip Bose, Francis P. O'Connell, and Bruce G. Mealey. "Adaptive Prefetching on POWER7: Improving Performance and Power Consumption". In: *ACM Trans. Parallel Comput.* 1.1 (May 2014). ISSN: 2329-4949. DOI: 10.1145/2588889.

[19] Ian Karlin, Abhinav Bhatele, Bradford L. Chamberlain, Jonathan. Cohen, Zachary Devito, Maya Gokhale, Riyaz Haque, Rich Hornung, Jeff Keasler, Dan Laney, Edward Luke, Scott Lloyd, Jim McGraw, Rob Neely,
David Richards, Martin Schulz, Charle H. Still, Felix Wang, and Daniel Wong. *LULESH Programming Model and Performance Ports Overview*. Tech. rep. LLNL-TR-608824. Livermore, CA, Dec. 2012, pp. 1–17.

[20] Ian Karlin, Jeff Keasler, and Rob Neely. *LULESH 2.0 Updates and Changes*. Tech. rep. LLNL-TR-641973. Livermore, CA, Aug. 2013, pp. 1–9.

[21] Muneeb Khan, Michael A. Laurenzanoy, Jason Marsy, Erik Hagersten, and David Black-Schaffer. "AREP: Adaptive Resource Efficient Prefetching for Maximizing Multicore Performance". In: *Proceedings of the 2015 International Conference on Parallel Architecture and Compilation (PACT)*. PACT '15. USA: IEEE Computer Society, 2015, pp. 367–378. ISBN: 9781467395243. DOI: 10.1109/PACT.2015.35.

[22] Frank Liu, Narasinga Rao Miniskar, Dwaipayan Chakraborty, and Jeffrey S. Vetter. "Deffe: a data-efficient framework for performance characterization in domain-specific computing". In: *Proceedings of the 17th ACM International Conference on Computing Frontiers*. CF '20. Catania, Sicily, Italy: Association for Computing Machinery, 2020, pp. 182–191. ISBN: 9781450379564. DOI: 10.1145/3387902.3392633.

[23] William Maas, Paulo S. S. de Souza, Marcelo C. Luizelli, Fábio D. Rossi, Philippe O. A. Navaux, and Arthur F. Lorenzon. "An ANN-Guided Multi-Objective Framework for Power-Performance Balancing in HPC Systems". In: *Proceedings of the 21st ACM International Conference on Computing Frontiers*. CF '24. Ischia, Italy: Association for Computing Machinery, 2024, pp. 138–146. ISBN: 9798400705977. DOI: 10.1145/3649153.3649185.

[24] Harshitha Menon, Abhinav Bhatele, and Todd Gamblin. "Auto-tuning Parameter Choices in HPC Applications using Bayesian Optimization". In: *2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*. May 2020, pp. 831–840. DOI: 10.1109/IPDPS47924.2020.00090.

[25] Jonas Mockus. *Bayesian Approach to Global Optimization*. Mathematics and its Applications. Springer Dordrecht, 2012. ISBN: 978-94-010-6898-7. DOI: 10.1007/978-94-009-0909-0.

[26] Guilherme Piccoli, Henrique N. Santos, Raphael E. Rodrigues, Christiane Pousa, Edson Borin, and Fernando M. Quintão Pereira. "Compiler support for selective page migration in NUMA architectures". In: *Proceedings of the 23rd International Conference on Parallel Architectures and Compilation*. PACT '14. Edmonton, AB, Canada: Association for Computing Machinery, 2014, pp. 369–380. ISBN: 9781450328098. DOI: 10.1145/2628071.2628077.

[27] Mihail Popov, Alexandra Jimborean, and David Black-Schaffer. "Efficient Thread/Page/Parallelism Autotuning for NUMA Systems". In: *Proceedings of the ACM International Conference on Supercomputing*. ICS '19.

Phoenix, Arizona: ACM, 2019, pp. 342–353. ISBN: 978-1-4503-6079-1. DOI: 10.1145/3330345.3330376.

[28] *Pyeasyga*. https://pyeasyga.readthedocs.io/en/latest/readme.html. Accessed: 2022-07-29.

[29] Petar Radojković, Vladimir Čakarević, Miquel Moretó, Javier Verdú, Alex Pajuelo, Francisco J. Cazorla, Mario Nemirovsky, and Mateo Valero. "Optimal task assignment in multithreaded processors: a statistical approach". In: *SIGPLAN Not*. 47.4 (Mar. 2012), pp. 235–248. ISSN: 0362-1340. DOI: 10.1145/2248487.2151002.

[30] Petar Radojković, Paul M. Carpenter, Miquel Moretó, Vladimir Čakarević, Javier Verdú, Alex Pajuelo, Francisco J. Cazorla, Mario Nemirovsky, and Mateo Valero. "Thread Assignment in Multicore/Multithreaded Processors: A Statistical Approach". In: *IEEE Transactions on Computers* 65.1 (Jan. 2016), pp. 256–269. ISSN: 1557-9956. DOI: 10.1109/TC.2015.2417533.

[31] Mahesh Ravishankar, John Eisenlohr, Louis-Noël Pouchet, J Ramanujam, Atanas Rountev, and P Saturdayappan. "Code generation for parallel execution of a class of irregular loops on distributed memory systems". In: *SC'12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis*. IEEE. 2012, pp. 1–11.

[32] Chandan Reddy and Uday Bondhugula. "Effective automatic computation placement and data allocation for parallelization of regular programs". In: *Proceedings of the 28th ACM International Conference on Supercomputing*. ICS '14. Munich, Germany: Association for Computing Machinery, 2014, pp. 13–22. ISBN: 9781450326421. DOI: 10.1145/2597652.2597673.

[33] Isaac Sánchez Barrera, David Black-Schaffer, Marc Casas, Miquel Moretó, Anastasiia Stupnikova, and Mihail Popov. "Modeling and optimizing NUMA effects and prefetching with machine learning". In: *Proceedings of the 34th ACM International Conference on Supercomputing*. ICS '20. Barcelona, Spain: Association for Computing Machinery, 2020. ISBN: 9781450379830. DOI: 10.1145/3392717.3392765.

[34] Lana Scravaglieri, Mihail Popov, Laércio Lima Pilla, Amina Guermouche, Olivier Aumage, and Emmanuelle Saillard. "Optimizing performance and energy across problem sizes through a search space exploration and machine learning". In: *J. Parallel Distrib. Comput.* 180.C (Oct. 2023). ISSN: 0743-7315. DOI: 10.1016/j.jpdc.2023.104720.

[35] Lana Scravaglieri, Mihail Popov, Laércio Lima Pilla, Amina Guermouche, Olivier Aumage, and Emmanuelle Saillard. *Optimizing Performance and Energy Across Problem Sizes Through a Search Space Exploration and Machine Learning - dataset*. Zenodo, Jan. 2023. DOI: 10.5281/zenodo.7590756.

[36] *Surrogate Modeling Toolbox*. https://smt.readthedocs.io/en/latest/. Accessed: 2024-07-29.

[37] Philippe Swartvagher, Sascha Hunold, Jesper Larsson Träff, and Ioannis Vardas. "Using Mixed-Radix De-
composition to Enumerate Computational Resources
of Deeply Hierarchical Architectures". In: *Proceed-
ings of the SC '23 Workshops of The International
Conference on High Performance Computing, Network,
Storage, and Analysis*. SC-W '23. Denver, CO, USA:
Association for Computing Machinery, 2023, pp. 405-
415. ISBN: 9798400707858. DOI: 10.1145/3624062.
3624109.

[38] Ali TehraniJamsaz, Mihail Popov, Akash Dutta, Emmanuelle Saillard, and Ali Jannesari. "Learning Intermediate Representations using Graph Neural Networks for NUMA and Prefetchers Optimization". In: *2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*. May 2022, pp. 1206-1216. DOI: 10.1109/IPDPS53621.2022.00120.

[39] J. Treibig, G. Hager, and G. Wellein. "LIKWID: A lightweight performance-oriented tool suite for x86 multicore environments". In: *Proceedings of PSTI2010, the First International Workshop on Parallel Software Tools and Tool Infrastructures*. San Diego CA, 2010.

[40] Zheng Wang and Michael O'Boyle. *Machine Learning in Compiler Optimisation*. 2018. arXiv: 1805.03441 [cs.PL]. URL: https://arxiv.org/abs/1805.03441.

[41] Carole-Jean Wu and Margaret Martonosi. "Characterization and dynamic mitigation of intra-application cache interference". In: *Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software*. ISPASS '11. USA: IEEE Computer Society, 2011, pp. 2-11. ISBN: 9781612843674. DOI: 10.1109/ISPASS.2011.5762710.

[42] Lilia Zaourar, Mohamed Benazouz, Ayoub Mouhagir, Carlos Falquez, Antoni Portero, Nam Ho, Estela Suarez, Polydoros Petrakis, Manolis Marazakis, Francesco Sgherzi, Ivan Fernandez, Romain Dolbeau, and Dirk Pleiter. "Case Studies on the Impact and Challenges of Heterogeneous NUMA Architectures for HPC". In: *Architecture of Computing Systems: 37th International Conference, ARCS 2024, Potsdam, Germany, May 14-16, 2024, Proceedings*. Potsdam, Germany: Springer-Verlag, 2024, pp. 251-265. ISBN: 978-3-031-66145-7. DOI: 10.1007/978-3-031-66146-4_17.

[43] Lilia Zaourar, Alice Chillet, and Jean-Marc Philippe. "A-DECA: An Automated Design Space Exploration Approach for Computing Architectures to Develop Efficient High-Performance Many-Core Processors". In: *2023 26th Euromicro Conference on Digital System Design (DSD)*. 2023, pp. 756-763. DOI: 10.1109/DSD60849.2023.00108.