# HPCQCMark: a new modular HPC-QC benchmarking framework

Brice Chichereau, Stéphane Vialle, Miwako Tsuji, Patrick Carribault,
Mitsuhisa Sato

► To cite this version:

Brice Chichereau, Stéphane Vialle, Miwako Tsuji, Patrick Carribault, Mitsuhisa Sato. HPCQC-Mark: a new modular HPC-QC benchmarking framework. 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Aug 2025, Albuquerque, United States. pp.8-14, 10.1109/QCE65121.2025.10285. hal-05426530

HAL Id: hal-05426530

https://hal.science/hal-05426530v1

Submitted on 19 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
HPCQCMark: a new modular HPC-QC benchmarking framework

**Brice Chichereau**

LIHPC, CEA,
Université Paris-Saclay
Bruyères-le-Châtel, France
brice.chichereau@cea.fr

**Stéphane Vialle**

LISN UMR-9015, CNRS
Université Paris-Saclay,
CentraleSupélec
F-91405 Orsay, France

**Miwako Tsuji**

RIKEN Center for
Computational Science
Kobe, Hyogo, Japan
miwako.tsuji@riken.jp

stephane.vialle@centralesupelec.fr

**Patrick Carribault**

LIHPC, CEA,
Université Paris-Saclay
Bruyères-le-Châtel, France
patrick.carribault@cea.fr

**Mitsuhisa Sato**

RIKEN Center for
Computational Science
Kobe, Hyogo, Japan
msato@riken.jp

**Abstract**

Supercomputing centers all over the world have started integrating Quantum Processing Units (QPU) with their supercomputers. These new hybrid architectures require software stacks with specific environments, interfaces, tools, ... This has led to a need for ways to quickly evaluate the readiness and the performance of these new software stacks. While benchmarking methods exist for the individual HPC or QC systems, combined HPC-QC benchmarking is still mostly a blind spot. We thus propose a new modular, easy to customize, benchmarking framework called “HPCQCMark” to evaluate HPC-QC systems and especially their software stack. We added modular components to the framework which cover relevant HPC-QC workflows and widely used QC programming environments. Finally, we perform experiments on Riken R-CCS’s and TGCC’s HPC-QC environments to validate the ability of the framework to evaluate the impact of HPC-QC software stacks on code execution time.

**Keywords**— benchmark, quantum computing, high performance computing, software stack, C++, Python

# 1 Context & background

## 1.1 Integrating High Performance & Quantum Computing

Quantum Computing (QC) offers multiple potential interesting applications for the world of High Performance Computing (HPC), including: linear system solving [1], combinatorial optimization [2,3], computational chemistry [4]. Because of this, multiple supercomputing centers all around the world are acquiring and installing Quantum Processing Units (QPUs) to be used alongside HPC systems. Some examples of these centers are: Riken R-CCS in Japan, LRZ and FZJ in Germany, TGCC in France, ORNL in the United States, and many others.

All of this has lead to the emergence of HPC-QC integration

as a new field of research and development for both hardware and software. Notably, some significant efforts are being put forward regarding the software stacks driving the new hybrid architectures [5-7]. This includes new or accounting and monitoring systems, modified job schedulers and resource allocation strategies & algorithms, programming models, ... While decades of research have gone into HPC software stacks, these brand-new stacks also need to take into account all the specific requirements of current QPUs: limited resources, error rates, availability, ... The reliability and performance of these software stacks could have a real impact on the adoption of hybrid HPC-QC computing.

As QPUs have already started being installed in the supercomputing centers, there is a real need for a way to evaluate the new software stacks. This is needed to evaluate their potential issues and highlight key performance metrics. The benchmarking and evaluation of HPC-QC systems has become a subject of interest for both supercomputing centers and HPC-QC users. This work will aim at identifying and filling a gap in the current evaluation capacities for HPC-QC systems.

## 1.2 Related work

Benchmarking High Performance Computing systems is an established research field with methods to evaluate application, library, or hardware performance. Application-oriented benchmarks often are proxy applications of real HPC codes, some examples include: LULESH [8] or Quicksilver [9]. HPCG [10] and LINPACK [11] are examples of benchmarks evaluating specific common HPC tasks. Some benchmarks focus on the hardware like for example STREAM [12] for memory bandwidth. Finally, there is even some drive for automated, community-driven continuous benchmarking of HPC systems [13].

While evaluating Quantum Computing systems is a more recent field, there are multiple widely used methods to assess the low-level characteristics of machines. These “system-level” benchmarks include for example: Quantum Volume [14], CLOPS [15] and EPLG [16]. High-level, application-oriented benchmarks have also started to recently make an appearance in quantum computing [17-20]. While the use of combined
<figure><img src="image_2.png" /><figcaption>(a) Execution flow of HPCQCMark: benchmarks are compiled and run on the HPC-QC system from the modular components. The execution results are analyzed into useful metrics and charts.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) Software architecture of HPCQCMark's micro-applications. The benchmark applications invoke methods from the abstract class **Backend**. The specific backend implementation of the class is chosen at compile time.</figcaption></figure>

<caption>Figure 1: HPCQCMark's overall architecture, the “QC Backends”, “Benchmarks” and “Job Script Templates” are all modular components made to be easy to extend.</caption>

HPC-QC resources is evaluated in [21] through mini-apps, the HPC side only serves as a support to simulating the QC execution.

Methods to evaluate HPC and QC systems individually have been developed and the evaluation of HPC-QC systems is an emerging field, there is however a lack of tools to comprehensively evaluate these hybrid systems. This is further the case for the evaluation of HPC-QC software stacks, in this paper we aim to fill this gap by developing and validating a new HPC-QC benchmarking framework: HPCQCMark.

The rest of this paper is organized as follows: in Section 2, we give an overview of how we designed HPCQCMark's modular architecture. In Section 3, we describe how modular components can be added to the framework and give an overview of the ones we already implemented. In Section 4, we validate HPCQCMark's ability to provide insights into the impact of HPC-QC software stacks on code execution. Finally, Section 5 concludes this paper.

# 2 HPC-QC benchmarking framework design

## 2.1 Design goals

The core aim of HPCQCMark is to allow the rapid evaluation and subsequent development of HPC-QC software stacks. This will be done through the execution of "micro-applications" for various software stacks on an HPC-QC system. Using this, both scientists and system engineers would be able to quickly diagnose issues in their toolchain. To achieve these goals, we have followed a set of core principles which we apply throughout the design and development of the benchmarking framework: *ease of use*, *relevance*, *modularity*, and *openness*.

### 2.1.1 Ease of use

Using the benchmarking framework should be as easy as possible in order to avoid introducing more issue than what we are trying to help diagnose. So, running the benchmarks should be quick, intuitive, and painless if the framework supports the

relevant software stack components. Wider adoption of this framework will heavily depend on how easy it is to use.

### 2.1.2 Relevance

To provide as much valuable information as possible, the proposed benchmarks should cover both current and expected future HPC-QC use-cases and workflows. While we do not aim to provide extensive application oriented benchmarks like in [17–20], the type of real-world use-cases of HPC-QC should always be kept in mind.

### 2.1.3 Modularity

Due to the variety of HPC-QC software stacks, adding support for new ones to HPCQCMark should be as simple as possible. Ideally, all parts of the framework like the HPC-QC workflows and QC backends should be modular to make additions easier. The framework should thus have clear separation of concerns between the different components: HPC-QC micro-applications, QC backends, HPC job configuration, ...

### 2.1.4 Openness

The entirety of the HPCQCMark source code will be made available under the cea-hpc GitHub organization. Combined with the aforementioned design principles, HPCQCMark should be built with potential external contributors in mind. Adoption from other supercomputing centers will be dependent on their ability to implement their own QC backends and workflows to the framework.

With these goals as core principles, the next section introduces the architecture of our benchmarking framework.

## 2.2 A modular HPC-QC benchmarking framework

In order to achieve the aforementioned goals, the HPCQCMark architecture is split between two main parts: the modular components and the supporting scripts. First is the modular side
with separate components for: HPC-QC benchmark applications, Quantum Computing backends and HPC job script templates. The other part consists of Python scripts that assemble the modular components into HPC-QC micro-applications as well as analyze the produced result. Figure 1a details the global architecture of the framework through the execution flow of a benchmark.

### 2.2.1 Software architecture

The “QC Backend” and “Benchmark” modular blocks of HPC-QCMark contain all the main code for the generated HPC-QC micro-applications. Figure 1b describes the interaction between the two sides: benchmarks call quantum kernels defined as methods in a **Backend** class. These methods are then implemented for each QC backend in separate C++ files, the chosen QC backend will then be linked to the benchmark at compile-time. While most of the code for the QC backends and the benchmarks is written in C++, some require some amount of Python code. To help with their implementation we provide a basic wrapper for the Python C API which allows to call a Python method easily for example.

### 2.2.2 Generating HPC-QC benchmarks

To use the benchmarking framework, a user provides a configuration file describing which modular component they want to use as well as some other execution parameters. In this file, the user selects a QC backend, which HPC-QC benchmarks to run, using which HPC job scheduler, for circuits with how many qubits, etc... From this file, the `generate_benchmarks.py` script will then create a folder for the micro-applications containing all files required for their execution on a supercomputer. This includes source code, compiler configuration, and HPC scheduler job scripts, all being generated from the modular components chosen by the user.

The next subsection will describe the various analysis we perform on the benchmark results through another Python script.

## 2.3 Analyzing results and providing insightful data

The first “metric” we collect in HPCQCMark is whether each benchmark-backend combination was executed properly or not. To help with this, error messages from the execution of the micro-applications are captured, aggregated and provided to the user during the analysis step. This can provide rapid feedback on possible critical problems with the currently implemented HPC-QC software stacks.

On the quantitative side, during execution of the HPC-QC codes we measure the start and end times of various code sections. These are recorded into CSV files and can include classical pre-processing, quantum circuit execution, MPI communications, ... These can be first used to visualize “execution timelines” which show the various timing sections for the various threads/processes.

Moreover, the timing measurements also provide the total execution time $T_{exec}$ of the HPC-QC code. This metric can be augmented by normalizing it based on the number of quantum measurement shots. As the latter is expected to directly affect the duration of the quantum circuit execution, this can help

when comparing experiments with different numbers of shots. This leads to a “normalized” execution time defined as follows:

$$T_{norm} = T_{exec}/N_{shots} \quad (1)$$

While this metric can help to compare experiments, it could arbitrarily favor experiments with many shots if the number of shots is not the major impacting factor of execution time. For more precise analysis it would be necessary to compare experiments with the same fixed execution parameters.

These metrics will be used extensively in Section 4 to validate HPCQCMark’s ability to evaluate the impact of HPC-QC stack on code execution.

Now that we have designed and developed a modular benchmarking framework for HPC-QC software stacks, we will describe how we implemented components such as benchmarks and QC backends that cover relevant HPC-QC workflows and programming frameworks.

# 3 Implementing relevant benchmarks and quantum backends

The goal of this Section is to implement relevant modular components for HPCQCMark, including both HPC-QC benchmark workflows and QC backends.

## 3.1 Benchmarks

The “benchmarks” are the first of the two modular components that become the evaluated HPC-QC micro-applications. The benchmarks are designed to be as vendor-agnostic as possible to allow the separation of concerns with the QC backends. Multiple versions of each benchmark are implemented to cover various HPC programming models like MPI or OpenMP. There are currently two main “benchmarks” implemented in HPCQCMark, each with various HPC programming model implementations.

The first benchmark, which we call “Split Shots”, has been designed to represent future applications where quantum work is distributed over many QPUs. This benchmark captures a workflow where a user wants to run $S$ shots of a quantum circuit and splits those across $N$ tasks (i.e., MPI processes or OpenMP threads). The quantum circuit here prepares a simple GHZ state [22] over $Q$ qubits.
<figure><img src="image_4.png" /><figcaption>Figure 2a provides an overview of the execution pattern of this benchmark type and Listing 1 describes what the code for this benchmark looks like. In the aforementioned Listing, we use the OpenMP programming model and `Back.runGHZ` is a method which is implemented by the various QC backends.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 2: Global flow of the implemented HPC-QC benchmarks. The arrows represent communications between tasks. The rounded boxes represent quantum jobs that are run on one or multiple QPUs.</figcaption></figure>

The second benchmark type will be closer to more short-
term HPC-QC workflows where the limited quantum resources
have to be used more sparingly. In this benchmark each task
has a QUBO problem instance that needs to be solved; the
instances are then gathered across “task groups” of arbitrary
size in order to limit the number of submitted quantum jobs.
This decrease in the number of quantum jobs comes at the cost
of each job having to run a larger circuit to solve the combined
QUBO instances. Figure 2b shows the execution flow of this
other type of benchmark which we call “Funneled QUBO”.
The QUBO problem instance for each task *t* is defined as a
QUBO matrix *Q*<sup>*t*</sup> which has the following form:

$$
\min_{x \in \{0,1\}^n} x^T Q^{(t)} x = \sum_{i=1}^{n} \sum_{j=1}^{n} Q_{ij}^{(t)} x_i x_j \quad (2)
$$

The aggregated problem instances are solved simultaneously by

solving a single combined block-diagonal QUBO matrix $Q = \text{diag}(Q^{(1)}, Q^{(2)}, \dots, Q^{(N)})$.

These two benchmarks represent very different types of
HPC-QC workflows and should cover many short- and long-
term use-cases.

3.2 Backends

The other main modular components of the benchmarking
framework are all the different “QC backends” that imple-
ment the various quantum kernels for various quantum soft-
ware stacks. Code-wise, the backends simply implement the
few methods defined in the **Backend** class related to the ex-
ecution of quantum circuits. The currently implemented QC
backends are listed in Table 1, their name corresponds to the
underlying quantum software stack. There are differences be-
tween these backends: some of these backends are emulator-
only, some are QC hardware-only, and some use some Python
code in addition to C++.

<table><caption>Table 1: Currently implemented QC backends in HPCQC-Mark as of the time of writing. Each QC backend's name corresponds to the name of the underlying QC framework.</caption>
   <thead>
    <tr>
     <td>
     </td>
     <td>
      Hardware
     </td>
     <td>
      Emulator
     </td>
     <td>
      Language
     </td>
    </tr>
   </thead>
   <tbody>
    <tr>
     <td>
      Qulacs [23]
     </td>
     <td>
      <span style="color:red;">
       X
      </span>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      C++
     </td>
    </tr>
    <tr>
     <td>
      QCRPC-IBM
      <sup>
       1
      </sup>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:red;">
       X
      </span>
     </td>
     <td>
      C++
     </td>
    </tr>
    <tr>
     <td>
      QCRPC-RQC
      <sup>
       2
      </sup>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      C++
     </td>
    </tr>
    <tr>
     <td>
      IBMQ-REST
      <sup>
       3
      </sup>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:red;">
       X
      </span>
     </td>
     <td>
      C++
     </td>
    </tr>
    <tr>
     <td>
      Qiskit [24]
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      Python
     </td>
    </tr>
    <tr>
     <td>
      Qnexus [?]
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      Python
     </td>
    </tr>
    <tr>
     <td>
      Pulser-myQLM [25]
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      <span style="color:green;">
       ✓
      </span>
     </td>
     <td>
      Python
     </td>
    </tr>
   </tbody>
  </table>

Adding a new quantum backend to HPCQCMark is as sim-
ple as implementing all the required kernels from the **Backend**
class. Some helper functions are provided to interact with the
Python C API if required by the QC framework. If this frame-
work supports the execution of quantum circuits specified in
the commonly used OpenQASM [14] representation, the im-
plementation of the backend becomes simpler. You can simply
implement a single method that executes a OpenQASM cir-
cuit and use provided circuit-generation methods for all the
relevant quantum kernels. An example of such a method is
given in Listing 2 and Listing 3 for the “Qiskit” QC backend.

<footnote n="1">¹RPC-based quantum circuits execution framework calling the IBM-Q API.</footnote>

<footnote n="2">²RPC-based quantum circuits execution framework calling the RQC API.</footnote>

<footnote n="3">³IBM-Q API client using the libcurl library.</footnote>
<figure><img src="image_6.png" /><figcaption>Listing 3: Python side of the Qiskit backend's `runQASM` implementation.</figcaption></figure>

<figure><img src="image_7.png" /></figure>

# 4 Validating HPCQCMark's ability to evaluate HPC-QC software stacks

Using the implemented benchmarks and backends, we are able to perform experiments on real HPC-QC architectures. With this we want to validate the ability of our benchmarking framework to evaluate of the influence of the software stacks on HPC-QC code execution.

## 4.1 Evaluating the impact of grouping QC jobs

As mentioned in Section 2.3, HPCQCMark can generate execution timelines from the measured timings of various code sections. We want to use this ability to evaluate the impact of grouping quantum jobs in the “Funneled QUBO” benchmark for the “Pulser-myQLM” QC backend.

We ran experiments on the TGCC’s Joliot-Curie supercomputer using their HPC-QC software stack based on Pasqal’s Pulser [25] analog QC framework and Eviden’s Qaptiva (previously known as QLM) system. The “Funneled QUBO” benchmark was run on this HPC-QC system using the MPI programming

model using Qaptiva-based emulation for the execution of the analog quantum jobs. The execution timelines for two different group sizes over 8 MPI processes, each with a size 4 QUBO instance, are shown on Figure 3.

In Figure 3a, each MPI process sends its own 4-qubits quantum job, while in Figure 3b the 2 “group leader” processes send a 16-qubit job each. The larger number of qubits in the second case lead to an increased Job Preparation time, this includes for example the placement of the neutral atoms in a 2D plane for the analog computation. The reduced number of parallel tasks in 3b leads to a reduced initialization time for the Python runtime (Python Init), while the larger circuits lead to longer circuit emulation time (Execution). The quantum jobs are emulated here meaning there is zero queue time, with a real QPU, each job will have to wait a certain amount of time depending on congestion. This type of analysis using HPCQCMark could enable users to more easily find a good compromise between the time spent in queue and the time spent preparing larger quantum jobs.

## 4.2 Detecting slow-downs caused by the software stack

We also installed and ran HPCQCMark on the supercomputer Fugaku at Riken R-CCS in order to evaluate the current state of its HPC-QC software stacks. Using this, we ran experiments to test all the installed HPC-QC software stacks, we will focus here on the “QCRPC-IBM” backend which sends jobs to a cloud-based IBM QPU through the IBM-Q API and a custom RPC-based middleware.

We noticed low performance when using the OpenMP parallel programming model, this can be seen by comparing the normalized execution time between OpenMP and MPI on Figure 4. This result led us to investigate the reasons behind this issue, we found cases where a global lock in the middleware was acquired by each thread. This lock meant that in the OpenMP version, a new quantum kernel is only submitted when the previous one finished executing. While in the MPI version all the jobs are submitted simultaneously and are thus run close to one another, in the OpenMP version every submitted kernel has to go through the job queue all over again. From what we saw, having to go through the queue every time led to the vastly increased execution time. The synthetic metric provided by HPCQCMark allowed to us to detect this notable behavior in a part of an HPC-QC software stack which affects execution time.

# 5 Discussion and conclusions

## 5.1 Summary & Discussion

In this paper we presented HPCQCMark, a benchmarking framework aimed at evaluating the impact of HPC-QC software stacks on the execution of codes. We designed a modular benchmarking framework aimed at providing an easy to use and to modify tool for the evaluation of the software stacks. This evaluation is done through metrics computed and provided by the framework such as execution time normalized by the number of quantum measurement shots. While this metric can help compare different experiments, it could be biased when the number of shots has a limited impact on execution time.
<figure><img src="image_8.png" /><figcaption>(a) Group size = 1</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) Group size = 4</figcaption></figure>

Figure 3: Execution timelines for the MPI version of the “Funneled QUBO” benchmark on TGCC with the PulsermyQLM backend. The timelines describe the execution of the benchmark for two different task group sizes (1 and 4 respectively). In both cases, each task has a QUBO problem of size 4 that needs to be solved. The larger group size in (b) leads to wider quantum circuits and thus slower job preparation for this backend.

<figure><img src="image_10.png" /></figure>

Figure 4: Normalized Execution Time using the QCRPC-IBM backend on Fugaku for the MPI and OpenMP versions of the “Split Shots” benchmark. The quantum circuits were all run on a single IBM-Q QPU for 8192 shots split across 8 tasks.

We implemented a number of HPC-QC workflows as benchmarks and QC backends for multiple quantum programming frameworks. These were meant to capture predicted application workflows as well as multiple software stacks that could be deployed shortly, especially at TGCC and Riken R-CCS. These cover many use cases, but new ones will probably need to be added as new types of HPC-QC workflows and software stacks are designed and deployed in supercomputing centers.

The ability of the benchmarking framework to evaluate the impact of HPC-QC software stacks was validated through experiments on the R-CCS and TGCC HPC-QC systems. While the executed benchmarks allowed us to detect slow-downs and interesting behaviors for some HPC-QC workflows and software stacks, HPCQCMark does not provide detailed application execution time analysis like some existing application-oriented benchmarks [17–20]. HPCQCMark aims to capture high-level behaviors of the software stacks currently being developed and deployed in supercomputing centers across the world.

## 5.2 Conclusion & Future work

HPCQCMark is a modular, easy to customize and easy to use benchmarking framework for the evaluation of HPC-QC software stacks. Our main goal when developing this framework was to enable rapid evaluation of these stacks and rapid investigation of potential problems. We validated the software stack evaluation ability of the framework by highlighting potential points of investigation and improvement in multiple deployed HPC-QC software stacks.

While we implemented support for multiple HPC-QC workflows and programming frameworks, many others exist, and we want to increase the size of our component portfolio in the future. We invite interested parties to keep an eye on the cea-hpc GitHub organization for the release of HPCQCMark if you'd like to try it on your system or even contribute. It would be interesting to work on components corresponding to the software stacks of other supercomputing centers integrating HPC-QC systems.

To improve the relevance of HPCQCMark for current HPC-QC systems, we want to add the ability to mix multiple QC backend types in a single micro-application. This would help test some new workflows for HPC-QC systems where QPUs from multiple vendors are present. We would also like to add support for full variational workflows as those can generate many QC jobs and thus pressure on the QPUs. Finally, we want to deploy HPCQCMark on more HPC-QC systems to further validate its real-world relevance and identify potential areas of improvement.

## Acknowledgments

This work is part of HQI initiative (www.hqi.fr) and is supported by France 2030 under the French National Research Agency award number “ANR-22-PNCQ-0002”.

## References

[1] A. W. Harrow, A. Hassidim, and S. Lloyd, “Quantum Algorithm for Solving Linear Systems of Equations,” Physi-
cal Review Letters, vol. 103, no. 15, p. 150502, Oct. 2009.

[2] E. Farhi, J. Goldstone, and S. Gutmann, "A Quantum Approximate Optimization Algorithm," Nov. 2014.

[3] S. Hadfield, Z. Wang, B. O'Gorman, E. G. Rieffel, D. Venturelli, and R. Biswas, "From the Quantum Approximate Optimization Algorithm to a Quantum Alternating Operator Ansatz," *Algorithms*, vol. 12, no. 2, p. 34, Feb. 2019.

[4] J. Liu, H. Ma, H. Shang, Z. Li, and J. Yang, "Quantum-centric high performance computing for quantum chemistry," *Physical Chemistry Chemical Physics*, vol. 26, no. 22, pp. 15831-15843, Jun. 2024.

[5] T. S. Humble, A. McCaskey, D. I. Lyakh, M. Gowrisankar, A. Frisch, and T. Monz, "Quantum Computers for High-Performance Computing," *IEEE Micro*, vol. 41, no. 5, pp. 15-23, Sep. 2021.

[6] T. Beck, A. Baroni, R. Bennink, G. Buchs, E. A. C. Pérez, M. Eisenbach, R. F. da Silva, M. G. Meena, K. Gottiparthi, P. Groszkowski, T. S. Humble, R. Landfield, K. Maheshwari, S. Oral, M. A. Sandoval, A. Shehata, I.-S. Suh, and C. Zimmer, "Integrating quantum computing resources into scientific HPC ecosystems," *Future Generation Computer Systems*, vol. 161, pp. 11-25, Dec. 2024.

[7] P. Seitz, A. Elsharkawy, X.-T. M. To, and M. Schulz, "Toward a Unified Hybrid HPCQC Toolchain," in *2023 IEEE International Conference on Quantum Computing and Engineering (QCE)*, vol. 02, Sep. 2023, pp. 96-102.

[8] I. Karlin, J. Keasler, and R. Neely, "LULESH 2.0 Updates and Changes," Livermore, CA, Aug. 2013.

[9] D. F. Richards, R. C. Bleile, P. S. Brantley, S. A. Dawson, M. S. McKinley, and M. J. O'Brien, "Quicksilver: A Proxy App for the Monte Carlo Transport Code Mercury," in *2017 IEEE International Conference on Cluster Computing (CLUSTER)*, Sep. 2017, pp. 866-873.

[10] M. A. Heroux, D. W. Doerfler, P. S. Crozier, J. M. Willenbring, H. C. Edwards, A. Williams, M. Rajan, E. R. Keiter, H. K. Thornquist, and R. W. Numrich, "Improving performance via mini-applications," *Sandia National Laboratories, Tech. Rep. SAND2009-5574*, vol. 3, 2009.

[11] J. J. Dongarra, C. B. Moler, J. R. Bunch, and G. W. Stewart, "LINPACK Users' Guide." Society for Industrial and Applied Mathematics, Jan. 1979.

[12] J. McCalpin, "Memory bandwidth and machine balance in high performance computers," IEEE Technical Committee on Computer Architecture Newsletter, Tech. Rep., Dec. 1995.

[13] O. Pearce, A. Scott, G. Becker, R. Haque, N. Hanford, S. Brink, D. Jacobsen, H. Poxon, J. Domke, and T. Gamblin, "Towards Collaborative Continuous Benchmarking for HPC," in *Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis*, ser. SC-W '23. New York, NY, USA: Association for Computing Machinery, Nov. 2023, pp. 627-635.

[14] A. W. Cross, L. S. Bishop, J. A. Smolin, and J. M. Gambetta, "Open Quantum Assembly Language," Jul. 2017.

[15] A. Wack, H. Paik, A. Javadi-Abhari, P. Jurcevic, I. Faro, J. M. Gambetta, and B. R. Johnson, "Quality, Speed, and Scale: Three key attributes to measure the performance of near-term quantum computers," Oct. 2021.

[16] D. C. McKay, I. Hincks, E. J. Pritchett, M. Carroll, L. C. G. Govia, and S. T. Merkel, "Benchmarking Quantum Processor Performance at Scale," Nov. 2023.

[17] T. Lubinski, S. Johri, P. Varosy, J. Coleman, L. Zhao, J. Necaise, C. H. Baldwin, K. Mayer, and T. Proctor, "Application-Oriented Performance Benchmarks for Quantum Computing," *IEEE Transactions on Quantum Engineering*, vol. 4, pp. 1-32, 2023.

[18] T. Lubinski, C. Coffrin, C. McGeoch, P. Sathe, J. Apanavicius, and D. E. B. Neira, "Optimization Applications as Quantum Performance Benchmarks," Feb. 2023.

[19] T. Lubinski, J. J. Goings, K. Mayer, S. Johri, N. Reddy, A. Mehta, N. Bhatia, S. Rappaport, D. Mills, C. H. Baldwin, L. Zhao, A. Barbosa, S. Maity, and P. S. Mundada, "Quantum Algorithm Exploration using Application-Oriented Performance Benchmarks," Feb. 2024.

[20] N. Quetschlich, L. Burgholzer, and R. Wille, "MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing," *Quantum*, vol. 7, p. 1062, Jul. 2023.

[21] N. Saurabh, P. Mantha, F. J. Kiwit, S. Jha, and A. Luckow, "Quantum Mini-Apps: A Framework for Developing and Benchmarking Quantum-HPC Applications," in *Proceedings of the 2024 Workshop on High Performance and Quantum Computing Integration*, ser. HPQCI '24. New York, NY, USA: Association for Computing Machinery, Aug. 2024, pp. 11-18.

[22] D. M. Greenberger, M. A. Horne, and A. Zeilinger, "Goming Beyond Bell's Theorem," in *Bell's Theorem, Quantum Theory and Conceptions of the Universe*, M. Kafatos, Ed. Dordrecht: Springer Netherlands, 1989, pp. 69-72.

[23] Y. Suzuki, Y. Kawase, Y. Masumura, Y. Hiraga, M. Nakadai, J. Chen, K. M. Nakanishi, K. Mitarai, R. Imai, S. Tamiya, T. Yamamoto, T. Yan, T. Kawakubo, Y. O. Nakagawa, Y. Ibe, Y. Zhang, H. Yamashita, H. Yoshimura, A. Hayashi, and K. Fujii, "Qulacs: A fast and versatile quantum circuit simulator for research purpose," *Quantum*, vol. 5, p. 559, Oct. 2021.

[24] A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Cross, B. R. Johnson, and J. M. Gambetta, "Quantum computing with Qiskit," Jun. 2024.

[25] H. Silvério, S. Grijalva, C. Dalyac, L. Leclerc, P. J. Karalekas, N. Shammah, M. Beji, L.-P. Henry, and L. Henriet, "Pulser: An open-source package for the design of pulse sequences in programmable neutral-atom arrays," *Quantum*, vol. 6, p. 629, Jan. 2022.