QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Multi-ranged Context Learning Baseline
Authors: Tianyi Bao
Ruizhe Zhong
Xinyu Ye
Yehui Tang
Junchi Yan
Venue: ICML 2025 poster (2025)
Keywords: Quantum Error Mitigation
Dataset Benchmark
Graph Learning
Primary area: applications->chemistry_physics_and_earth_sciences

Abstract: Quantum Error Mitigation (QEM) has emerged as a pivotal technique for enhancing the reliability of noisy quantum devices in the *Noisy Intermediate-Scale Quantum* (NISQ) era. Recently, machine learning (ML)-based QEM approaches have demonstrated strong generalization capabilities without sampling overheads compared to conventional methods. However, evaluating these techniques is often hindered by a lack of standardized datasets and inconsistent experimental settings across different studies. In this work, we present **QEM-Bench**, a comprehensive benchmark suite of *twenty-two* datasets covering diverse circuit types and noise profiles, which provides a unified platform for comparing and advancing ML-based QEM methods. We further propose a refined ML-based QEM pipeline **QEMFormer**, which leverages a feature encoder that preserves local, global, and topological information, along with a two-branch model that captures short-range and long-range dependencies within the circuit. Empirical evaluations on QEM-Bench illustrate the superior performance of QEMFormer over existing baselines, underscoring the potential of integrated ML-QEM strategies.

QEM-Bench:
Benchmarking Learning- Quantum Error Mitigation and QEMformer as a Multi-ranged Context Learning Baseline
Tianyi Bao 12 Ruizhe Zhong'
Xinyu Ye Yehui Tang' Junchi Yan 12
Abstract
2019), noise significantly impedes the practical realization of quantum systems, limiting their performance and relia- bility. Quantum Error Miligation (QEM) techniques have therefore emerged to restore quantum advantages on noisy hardware by algorithmically suppressing noise-induced bi- ases via post-processing (Temme et al., 2017; Li & Ben- jamin, 2017; Kandala et al., 2019; Huggins et al., 2021; Czarnik et al., 2021; Bravyi et al., 2022; Daley et al., 2022). While these strategies are crucial stepping stones toward achieving near-term quantum utility beyond classical su- percomputers, they often suffer from drawbacks such as high sampling overheads (Cai et al., 2023) or large qubit overheads in methods like virtual distillation (VD) (Huggins et al., 2021). Morcover, some approaches rely heavily on prior knowledge of the noise models (Liao et al., 2025), limiting their generalization capabilities.
Quantum Error Mitigation (QEM) has emerged as a pivotal technique for enhancing the reli- ability of noisy quantum devices in the Noisy Intermediaae-Scall Quantum (NISQ) era. Re- cently, machine learning (ML)-based QEM ap- proaches have demonstrated strong generaliza- tion capabilities without sampling overheads com- pared to conventional methods. However, evaluat- ing these techniques is often hindered by a lack of standardized datasets and inconsistent experimen- tal settings across different studies. In this work, we present QEM-Bench, comprehensive bench- mark suite of (wenyy-wwo datasets covering diverse circuit types and noise profiles, which provides a unified platform for comparing and advancing ML-based QEM methods. We further propose a refined MI--based QEM pipeline QEMFormer, which leverages a feature encoder that preserves local, global, and topological information, along with a two-branch model that captures short-range and long-range dependencies within the circuit. Empirical evaluations on QEM-Bench illustrate the superior performance of QEMFormer over existing baselines, underscoring the potential of integrated ML-QEM strategies.
Machine learning (ML) has recently emerged as a promising solution to these limitations, leveraging neural networks for ideal outcome prediction without qubit or sampling over- heads (Kim et al., 2020a; Liao et al., 2024). Nevertheless, current ML-based methods face several challenges. First, existing feature encoders either scale exponentially with the number of qubits (Kim et al., 2020a) or focus on coarse global information (e.g.. counts of gates with rotation an- gles between 0 and 5) (Liao et al., 2024). Second, prevail- ing architectures rarely capture multi-range dependencics. Both Kim al. (2020a) and Liao et al. (2024) adopt vanilla MLP structures that fail to effectively encode circuit topol- ogy. Although Liao al. (2024) additionally explores GNN architectures, this approach offers a less direct means of learning node attributes, resulting in reduced performance. Further, the MI-QEM field lacks standardized benchmarks: current studies frequently employ distinct and narrowly fo- cused experimental settings. Such variability in datasets and evaluation protocols has led to a limited coverage of diverse noise models, insufficient examination of general- ization capabilities, and difficulties in comparison among approaches.
1. Introduction
Quantum computing promises to revolutionize fields such as cryptography (Gisin et al., 2002) and machine learn- ing (Biamonte et al., 2017) by efficiently solving problems that are intractable on classical hardware. However, during the Noisy Intermediate-Scale Quantum (NISQ) era (Brooks,
'Sch. of Computer Science Sch. of Artificial Intelli- gence, Shanghai Jiao Tong University "Shanghai Innovation Insti- tute. Correspondence to: Junchi Yan <yanjunchi @ww...edu.cn>.............. This work supported by NSFC (92370201, 62222607). Code and datasets are at: https://gittie wtt..b.ooobblllllccoo QEM-Bench- ICML.
To address these shortcomings, we unify a collection of twenty-two ML-QEM datasets, named QEM-Bench, that standardize key experimental factors and provide a compre- hensive range of benchmarking scenarios. These datasets
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Baseline
encompass diverse noise configurations (incoherent, coher- ent, fake-provider, and real IBM devices) and various circuit types (structured 1-D Ising models (Suzuki et al., 2013), QAOA (Zhou et al., 2020), and random unstructured cir- cuits), with qubit counts ranging from small (4 qubits) to large (50 qubits). Furthermore, unlike Kim et al.; Liao et al., we propose an ML-QEM pipeline, named QEMFormer, that encodes quantum circuits as directed acyclic graphs (DAGs), preserving both gate-level information mode--baed features), circuit attributes (global-level features) and graph topology. Simulaaneussyy in contrast to prior MI--based QEM methods, we develop two-branch model that lever- ages both MLPs for capturing short-range gate-specifie con- text and Graph Transformers for modeling long-range de- pendencies and topological structure, thereby offering more robust approach error mitigation. Our contribu- tions are:
rors, where random Pauli operator is applied to each qubit (Nielsen & Chuang, 2000). Coherent errors arise from unintended or imperfect unitary rotations within quan- tum circuits. These errors are typically associated with slow noise processes (Huang et al., 2023; Beale et al., 2018). Ad- ditionally, other physical errors such as amplitude damping occur when energy is dissipated from the system into the environment, leading to state decay (Blume-Kohour et al., 2022). In contrast, Markovanv errors exhibit memory effects, where the noise depends on the system's 's history. A key indicator of non-Markovian noise is the oscillation of qubit's coherence and purity over time. The purity p of a qubit state, defined as = Tripp], can reveal such memory effects when it varies periodically (Agarwal et al., 2024).
2.2. Quantum Error Mitigation
Among non- -on-Me- QEM techniques, Zero-Noise Ex- trapolation (ZNE) is widely adopted. It estimates noise- free expectation values by intentionally increasing the noise levels and executing quantum circuits with measurements. While enjoying the capability of generalization, ZNE de- mands significant computational resources due to repeated circuit executions. Clifford Data Regression (CDR) is an- other prominent method (Czarnik et al., 2021). CDR ap- proximates noise-free expectation values by replacing most Cli-ffifford gates in the target circuit with Clifford gates, which are efficiently classically simulable. This replace- ment facilitates the generation of training data, enabling a linear regression model to map noisy expectation values to their ideal counterparts. Although CDR leverages the effi- cient simulation of Clifford circuits, it requires substantial number of training samples and may not generalize well to circuits with high density of non-Clifford gates.
1) Unifying and releasing datasets for QEM. We introduce QEM BeM-Bench, a collection of twenty-two ML-QEM datasets for standardized evaluation: (i) nine standard datasets for general-purpos testing, (ii) nine advanced datasets probing generalization capabilities, and (iii) four large-scale datasets with 50- and 63-qubit circuits executed on real IBM quantum devices.
2) The proposed QEMFormer approach. We propose pipeline that encodes quantum circuits as directed acyelic graphs (DAGs), preserving both gate-level, global-level, and topological information. Our two-branch model, QEM- Former, integrates MLP modules for short-range context and Graph Transformers for long-range dependencies.
3) Benchmarking baselines across datasets. We imple- ment existing ML-based baselines and thoroughly compare them against QEMFormer on QEM-Bench. QEMFormer consistently outperforms other methods across the standard, advanced, and large-scale real-device datasets.
Machine Learning (ML)-based QEM approaches have re- cently gained attention. For instance, (Kim et al., 2020b) utilizes neural networks (NN) and concatenated neural net- works to predict errors in the measurement outcomes of quantum states. However, their approach faces scalability issues as the input feature space, consisting of noisy mea- surement outcomes, grows exponentially with the number of qubits. Similarly, (Liao et al., 2024) evaluates several ML models, including linear regression, random forests, multi-layer perceptrons, and graph neural networks, aiming to minimize the sum of squared errors between mitigated and ideal expectation values. These models, however, often struggle to capture both short-range and long-range depen- dencies within quantum circuits, resulting in limited mitiga- tion performance. Overall, while ML-based QEM methods offer promising improvements in efficiency and scalability, they are hindered by challenges in feature encoding and the ability to model complex circuit dependencies, necessitating further advancements to fully realize their potential.
2. Related Works
2.1. Quantum Device Noise
Quantum device noise can be classified into two categories, Markovian and non-Markovian, based whether the vironment retains memory of eenteemmnviionment interac- tions (Zhang et al., 2024). Detailed descriptions of the noise types considered in this study are provided in Appendix C.
A noise process is deemed Markovian if the system's 's state transformation p p p' depends solely on the current state p, independent of previous operations or temporal context. Markovian errors can be further subdivided into two main types. Incoherent (stochastic) errors include bit-flip er- rors, which flip a single qubit state between (0) and [1), phase-flip errors, which alter the relative phase without changing the probability amplitudes, and depolarizing er-
2

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Baseline
3. Building Benchmarking Datasets for QEM
Let D = ((9,, Xx, A 3/& ye W) j= denote dataset of N quantum circuits, where each circuit Ck is represented as directed acyclic graph (DAG) Gn = (Vs, Ex). Nodes and edges in the graph are associated with feature vectors X E RIPL/xd and an adjacency matrix A, E RIM//XMM respectively. The noisy and ideal expectation values are denoted as V/
This section begins by revisiting the definitions and nota- tions of learning- based QEM in Sec. 3.1. The construction of QEM-Bench, including circuit selection, noise configu- rations, and dataset statistics is presented in Sec. 3.2 3.4. Experimental settings are outlined in Sec. 3.5.
noisy (0)k maisy and ya = (0)k
3.1. Problem Revisiting
The objective to learn a graph transformer model f that maps each circuit's ' graph representation and its noisy mea- surement to prediction ÿJ: of the ideal expectation value:
We first provide a brief overview of the key concepts related to Quantum Error Mitigation (QEM). For details about quan- tum information and computing, please refer to the seminal textbook (Nielsen & Chuang. 2000).
U = [(k, Xx, Ak: Uk
noisy)
by minimizing the mean squared error (MSE) between pre- dictions and true labels:
3. QUANTUM STATE VECTOR
A quantum state is described by a state vector (or ket) in complex Hilbert space. For qubit, the state is: |eb)
N min
yk 0z)²
N
a]0) + 1) where a, 3 E C and |a|2 + 1/812 -
=1 1. For an
n--ubit system, the state vector resides in a 2"-dimensional
Without loss of generality, we focus on a single sample (g,X, A, g y) for the remainder of the paper.
Hilbert space: (xb) = Σi=
i=1 04 (i), satisfying Σi= |a,|²
1. Quantum states evolve via quantum circuits (QC), which consist of applying quantum gates in sequence.
QEMFormer is divided into two major subsets: standard datasets and challenging datasets. As the QEM benchmark is constructed based on the quantum eircuits, we first intro- duce the circuits we used for the benchmark.
3.1.2. EXPECTATION VALUE IN QUANTUM CIRCUITS
Consider a quantum circuit represented by the unitary U acting on an initial state (2)).. The ideal expectation value of an observable O is: { ideal (N/ UTOU))o)- In prac- tice, the measured value (0) maisy deviates from the ideal due to factors such as decoherence, gate imperfections, and other noise sources (Cai et al., 2023). QEM aims to recover (O) ideal from these noisy measurements (0) noisy.
3.2. Circuit Selection
We evaluate our approach across three representative classes of quantum circuits, each distinguished by unique structures and parameters: (i) Trotterized one-dimensional transverse- field Ising model (TFIM) circuits, (ii) randomly generated circuits, and (iii) QAOA circuits designed for the MaxCut problem. Each class is described in detail below.
Observables, Pauli Operators, and Hamiltonians. Ob- servables are typically represented by Hermitian operators, with Pauli operators (e.g., X, Y, Z, I, also named as Pauli matrices) serving fundamental basis for single-qubit measurements. For multi-qubit systems, operators are often expressed as tensor products of Pauli matrices, enabling the decomposition of more complex observables. A Hamil- tonian, another crucial Hermitian operator, specifies the energy structure of a quantum system and governs its dy- namics (Preskill, 2018). Many quantum applications, such as quantum chemistry and materials science simulations, involve estimating the expectation value of Hamiltonians or sums of Pauli operators (e.g., in variational quantum eigensolver (VQE) frameworks) (Peruzzo et al., 2014).
Trotterized TFIM Circuits. Consider time-evolution cir- cuits for the one-dim TFIM, governed by the Hamiltonian:
f = -,, 2,2,+ + , = -J Hzz + h ffxx, (1)
where J is the nearest-neighbor exchange coupling and h is the transverse field strength (Suzuki et al., 2013). To simulate the time evolution over an interval t, one applies the unitary U(t) = exp(-i f t). In practice, we use the first-order -rotter-Suzuki decomposition:
U(I) ~ (o (cx(() E (ih Nuv.
(2)
where Al = 1/NTrot and NTrot is the number of Trotter steps. Each term in Eq. 2 is further decomposed into ele- mentary gates (e.g., CNOT and single-qubit rotations). We randomly sample J, h, and I, varying NTro from 1 to 20 to produce circuits of different depths.
3.1.3. LEARNING- BASED QEM FORMULATION
For machine barming-based approaches, the problem can be framed as a regression task. Specifically, we formulate the prediction of the ideal expectation value (O)idaal as graph- level regression task, although it is not strictly necessary to use graph structures when employing models such as MLPs.
Random Unstructured Circuits. To evaluate the generality of models, we also construct randomized quantum circuits. Then, each gate in the circuit is selected uniformly from:
3

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Baseline
Table I. Overview of datasets in QEM-Bench
Setting Type
Traln/Validation
Test
Qubit Num
(Train/Val)
Key Parameters
#
Key Parameters
Trotter-ttndaad
800/100
Steps 1-20 Size 10-150, 8 20]
300
Steps 1-20
10
Random Sanddo--Stamllle Standard
800/100
300
Size 10-150. 8 e [0.
3-6
QAON- QA0A-Stamdrrd
800/100
Layers 12-18
300
Layers 12-18
Trotter-Stey Step Zero-Sbot
765/135
Steps 1-15, Pauli-2 Z
300
Steps 16-20, Pauli-Z
10
Random-San Sire Scht
658/117
<100 gates, Pnuli-
425
100- 150 gates, Pauli-Z
3-6
Unseen Pauli-Basis Obs
800200
Steps 1-20, random Pauli-basis obs
200
Stops 1-20, New Pauli-basis obs
Kyiv Pre
400/78
Trotter Cires; extreme outlier filtered
113
Trotter Cires: extreme outher filtrre
50
Kyiv Raw
467/50
Trotter Cires
100
Trotter Cires
50
Brishune Pre
500/61
Troiter Cires; extreme outlier filtered
148
Trotter Cires; extreme outlier filtered
63
Brishane Row
800/100
Trottor Cires
300
Trotter Cires
63
QAOA encodes this objective into the cost of Hamiltonian
Two-Qubit-Gak Gate
One-Qubit-Gale
One Params-Gate Tioo Params Gate
Zero Params Gate
Hc
(i 2)- (i.))E
Σ
(4)
Three: Qubit Gate
Three Params
(a)
where 2, is the Pauli-Z operator acting on qubit i. The
MC-
MC 6
QAOA circuit alternates between phase-separation unitaries
2
MC
Uc(1) = exp(-i7 He) and mixing unitaries (B( = exp( (-1 HB), where HB Σド X3, and (7, are
MC-3
MC 8
4
MC
5C=5
variational parameters.
(c)
Figure 1. Distribution of gate types in the quantum circuit among datasets. (a) Proportions of one-, two-, and three-qubit gates in the random circuit dataset. (b) Distribution of zero-parametes one-, two-, and paremparrmeter gates in the random circuit dataset. (c) Distribution of the maximum cut solutions for QAOA circuits in
3.3. Circuit Set Statisties
We construct three distinct sets of quantum eircuits. Statis- tics for each set of circuits are presented in Tab. 1. Each dataset comprises 1,200 unique circuits.
the QAOA dataset, grouped by solution value (MC = I to 9).
For the randomized circuits, the distributions of single- two-, and three-qubit gates are illustrated in Fig. ((a), and the distribution of gates requiring zero, one, two, or three rotation parameters is depicted in Fig. 1(b).
1) Single-qubit gates: {id, u1, u2, u3, x, y, z, h, s, sdg, t, tdg, rx, ry, rz)
2) Two-qubit gates: {cx, cy, cz, ch, orz, cul, cu3, swap, r22]
For QAOAs, we fix the number of qubits to 6, targeting the MaxCut: N₂, = 6. Each circuit corresponds to randomly generated 6-node graph G = (V. F). where |V| 6. The number of QAOA layers for each circuit is randomly set from the integer range [12, 18]. The distribution of MaxCut values across generated graphs is given in Fig. 1(c).
3) Three-qubit gates: fecx, cswap).
For parameterized gates (e.g. u3, rx, ry, r2), rotation gles are sampled from [0, 2T.. The qubit(s) upon which gate acts are chosen uniformly among the available qubits.
3.4. Noise Configurations
For the random circuits, we set the max qubit number over tch This procedure yields a diverse ensemble of circuits with varying structures and depths.
We consider three primary noise configurations:
Fake Providers. To emulate the noise characteristics of real quantum devices, We utilize FakeHannivV and F FakeWashington.
QAOA Circuits for MaxCut. The Quantum Approximate Optimization Algorithm (QAOA) (Farhi et al., 2014; Har- rigan& & Sung, 2021) is designed to solve combinatorial optimization problems by encoding the objective function into a cost Hamiltonian. We focus on QAOAs for MaxCut in this paper, where one seeks to partition the vertices of a graph G = E) into two subsets to maximize the number of edges between them. Labeling each vertex E V with binary variable 2; E {+1,-1) -1), the objective can be written as (Guerreschi & Matsuura, 2019):
Incoherent Errors. Incoherent noise is simulated by apply- ing Pauli-X and depolarizing errors to all gates, based on the Sycamore quantum device (Arute et al., 2019). The error rates are set as follows: single-qubit gate error 0.16%, two-qubit gate error 0.62%, and read-out error at 3.8%.
Coherent Errors. To further examine models' capabilities among types of noises, coherent noise is introduced by adding systematic over-rotation errors to two-qubit gates (CX, CY, CZ, and Swap) with an average over-rotation angle of 0.02m, in combination with the noise model derived from
1 Σ (1 24 22)- (1.j) E
max
(3)
[=]}
4

---

QEM- Benchmarking Learning-basee based Quantum Error Mitigation and QEMFormer as a Baseline
Quantuum Cirouit
Global Feature
3.5.3. LARGE-SCALE SETTINGS
Drtaf
Kadl
We curate datasets: two comprising 50-qubit Trotterized circuits run IEM Kyiv and two comprising 63-qubit Trotterized circuits run on IBM.Brisbane. In the Raw variants (Kyiv Raw, Brisbane Raw), noisy expectation values are taken directly from the measurement outcomes. In the corresponding Pre variants (Kyiv Pre, Brisbane Pre), we remove only the most extreme outliers that reflect severe noise corruption.
Graph Transformrr branch
Disected Acpelli Grogh (DAG))
MLP branch
Node Festare
Vilas
Piciin
By incorporating standard, advanced, and large-scale datasets, QEM-Bench offers comprehensive and multi- faceted evaluation of error mitigators. Standard datasets assess their general performance, advanced datasets exam- ine their ability to extrapolate to new depths, sizes, and observables, and large-scale datasets evaluate their potential for practical application.
Figure 2. The structure of proposed QFMFormer.
the FakeWashington provider.
Real Quantum Devices. For experiments of large-scale circuits, we executed circuits on IBM_Kyiv and IBM Brisbane, two 127-qubit quantum computers provided by IBM.
4. Proposed QEMFormer
To address the limitations of existing ML-based QEM meth- ods, we introduce QEMFormer to enhance feature extrac- tion and leverage the topology of quantum circuits. It en- codes quantum circuits as directed acyclic graphs (DAGs), enabling the extraction of both local (gate-level) and global (circuit-level) features. Our architecture (presented in Fig. 2) employs a two-branch design: one branch utilizes multi- layer perceptrons (MLPs) to capture short-range gate con- texts, while the other leverages graph transformers with self-attention mechanisms to model longerange dependen- cies within the circuit. This integrated approach facilitates the effective interpretation of munti-rangedd contextual infor- mation, leading to improved prediction of EVs.
3.5. Experimental Settings
We construct a total of 22 datasets by combining three main setting types (standard, advanced, and large-scale) with three circuit families (QAOA, random, Trotter) and three noise configurations (incoherent, all noise, provider). Each dataset is partitioned into training, validation, and test sets.
3.5. STANDARD SETTINGS
The 9 standard datasets are derived from all possible com- binations of (circuit family) (noise configuration). Each standard dataset contains circuits with fixed depth/size and predefined set of measurement operators (e.g., all Pauli- Z) that have been previously introduced. These datasets serve as a baseline evaluation for error mitigators under well-controlled conditions.
4.1. Feature Encoder
Let C be a given quantum circuit operating on N, qubits. In line with (Moflic al., 2023; He et al., 2023). we construct directed acyclic graph g (V, from C, where each node v; E V represents a quantum operation (or special "start"/'end" symbol for each qubit), and edges (v; - U3) E E signify the directed flow of operations along qubits in C.
3.5.2. ADVANCED SETTINGS
To examine generalization beyond the configurations seen in training, we introduce three advanced benchmarks, each paired with the same set of three noise configurations. This results in total of 9 advanced datasets:
4.1.1. 1. NODE-LEVEL FEATURES
Trotter-Step Zero-Shot: Trains on Trotter circuits with shallower depths. then tests on circuits with deeper (previ- ously unseen) Trotter steps.
Let N = |V| be the total number of nodes. We endow each node e; with a feature vector 2; E IRd where
x; = g q: Ø.).
Trains on random circuits up to certain gate-count threshold, and tests on larger, more complex circuits outside the training distribution.
Random-Size Zero-Shot:
Here: g: E [0, 1) Noge is a one-hot vector representing the type of the i-th gate (e.g., CNOT, RX, H. .). The dimen- sion Noye equals the total number of unique gate types in the dataset. qi E {0, 1] is a multi-hot encoding that in- dicates which qubits are operated on by the gate associated with node V;- 6, E Rd4 encodes all possible continuous parameters for parameterized gates (c.g., rotation angles). Specifically, if the dataset includes up to de parameters, e.g.,
Unseen Dauli-Basis Observables: Trains on random n- qubit observables drawn from the set [I,X, Y, 2) Dur- ing testing, each circuit is paired with an observable that was not included in the training set, thereby challenging the model to generalize to novel measurement operators.
5

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Baseline
each gate might have zero/single/multiple rotation angles, then 6 reserves one entry for each rotation angle. In this way, x; captures gate properties from diverse perspectives.
MLP branch and Graph Transformer branch. The outputs of these branches are then concatenated together with u, forming aggregated representation that is passed through an additional 2-layer MLP to predict the expectation value.
4.1.2. GLOBAI-LEVEL. FEATURES
(1) MLP Branch. In this branch, we first perform global mean-pooling on the node features to obtain a single vector:
Besides local gate-level features, we also extract a global feature vector u E Rd to capture circuit-wide properties. This consists of two parts:
N X E RR
1
X
(5)
Circuil Statistics. We first extract the count of single- two-, and three-qubit gates; also gather the binned rotation gles including the number of gates whose rotation angle(s) lie in each of the intervals [o, 1 T), [TT, 2 > (3 2T); and additionally we count the total number of gate opera- tions within each circuit. We then normalize these circuit- related features to [0, 1].
We then feed x into multi-layer perceptron (MLP) com- posed of K sequential layers with bath normalization (BN);
x (1) = x, k = K.
(6)
x (9) Dropout( (BN w +b b
At the final layer, we obtain the MLP branch output:
Multiple Pauli-Basis Expectation Values. For a quantum circuit with N₂ qubits, we measure the expectation values of aungle-qubit Pauli operators X, Y, and Z on each qubit, while applying the identity operator I to all other qubits. For each qubit i E {1...., Ng) and each Pauli operator P E (X, Y, Z}, the measurement operator is defined as
Zmlp = (
E R Rddi.
(2) Graph Transformer Branch. In parallel, we process the node features X via Graph Transformer, which gener- alizes the self-attention mechanism (Vaswani al., 2017; Devlin et al., 2019) to graph data. For each Graph Transformer layer / = O, L- L-1 1, we compute multi-head attention as follows: for the e-th head (c = (.....)).
P, = 10 & P & Io(N--))
We then obtain the expectation values (Pt) for all combina- tions of i and P. resulting in a total of 3.,, measurements. These expectation values provide a comprehensive character- ization of the global noise affecting the quantum hardware from the qubit level. Formally, let
W(() 6.4
(1)
(1))
(() C.
w.t x3 +b00 c,ke
x
0,97
(9 (9)
exp
(8)
( O
Vd
=
(q k!ll
u [c. m, y moisy] E 18+
Σ EnEN(()) exp
vd
where e E RR. aggregates circuit-level statistics, including counts and binned angle distributions, m E R3N,, contains the noisy expectation values of the (X, Y, Z} observables measured across N, qubits, and y goisy E IR the raw noisy expectation value of the target observable to predict.
where x;
is the feature of node i at layer 1, and d is the head dimension. We then aggregate messages from neighbors:
(1) o.
w!!)
b
C.UT
x((+D) x
Σ
cxy
Overall, each node i in the graph G is associated with local feature vector X; E RR and the entire circuit C is denoted by a global feature vector u E R8+3N,+1. This encoding scheme via graph effectively captures local gate interactions, circuit topology, and global context, offering rich and scalable representation for QEM tasks. Also, it maintains acceptable dimensionality compared to tradi- tional encoders, ensuring computational eefiraationllffifiiinee...
JEN(()
where [|__||-] denotes concatenation along the head dimen- sion. To further stabilize training, we employ gated resid- ual connection with layer normalization (LN):
r with (1)
+ bbD..
3 (w
[x] (x(++))
r x]
(1) (1+1) -11)) (10)
x (+1) RELU(LN((1 ,(0) xZ (1+1) + B r1 r.1)).
4.2. Model Architecture
We adopt two-branch design, capture the short-range context, the long-range information, and the topological structure of quantum circuits. Denote by X E IR N xd,, the initial node features for N nodes (gates) in the circuit graph, and by u E Rd the global (circuit-level) feature vector. Our model processes X in two parallel branches: an
After L layers, we obtain ( fx( (L) ..... x J. To
(1)
produce a circuit-level representation for the final layer, we apply global readout (e.g., mean-pooling):
N
1
Zutt
E E.daat.
(1))
N
i=1
6

---

QEM-Bench: Benchmarking Learning-baser Quantum Error Mitigation and QEMFormer as a Baseline
Table RMSE comparison over different settings and noise models (our method: QEMFormer). The best- pest-performing method
highlighted in red, while the second- second-best is highlighted in blue. Lower RMSE values indicate better performance.
Motbod
Incoherent
Random Size Zero Shots
Coherent
Provider
Incolaerent
Trotter Step Zero Shots
Coherent
Provider
Incoherent
Unseen Obs
Coherent
Provider
Nony
0.19 0.117
0.125 1 0.081
0.131
0.00
0.313 0.101
0.49% 0.147
0,535
0.170
0.134
0,188 2 0.097
QRMFormer
0000 0:061
8:113
0.078
0.069 + 00000
0.025
0.015
0.151
0
0089
0.051
0.08
0.074
0180 + 0.057
0.096 + 00075
MLP
0.166 0.073
0.119 = 0.080
0.105 + 0.074
0.049 = 0.030
0.183 + 0.148
0.117
+ 0.058
0.139 + 0.110
0.146 = 0.109
0.116 = 0.093
GNN
0.200 0.132
0,162 4 0.105
0.074 4: 0.054
0.024 a: 0.017
0.179 0.148
0.123 0.062
0,102 4 0.078
0.124 1: 0.099
0.104 1: 0.078
OLS
0259:1 0.151
0.164 1: 0,088
0.155 1: 0.093
0.026 1: 0,018
0.215 1: 0.144
0.098 1 0.056
0.131 1 0.087
0.090 1: 0.061
0.125 1: 0.089
ZNH
0175-+:00 0.114
0.157 + 0,110
0,120 + 0.80
0.255 + 0.057
0.479
0.159
0.279 + 0.115
0.154 + 0.133
0,111 + 0.090
0.117 + 0.100
CDR
0.049
0.119 0,082
0.072 +
0.039 0.052
0.295
0.228
0.153
0.128
0.109
+ 0.092
0.123 0.105
RF
0.113 0.075
0.122
+ 0.082
0.086 2: 0.066
0.064 +: 0.044
0.206
4: 0,173
0.244
à: 0,100
0.131 4: 0.105
0,100 +: 0.076
0.110 :: 0.085
GTraQEEM
0.103 0.065
0,118 0,080
0.037 0.023
0.183 0.152
0.165
0.095
0.100
0.061
1.088
Table 3. RMSE for various circuit types and noise models. Multiple implementations of ZNE, CDR, and RF consistently performed weakly on QAOA circuits and are therefore excluded from the miligation results to ensure a fair comparison.
Method
Random
QAOA
Trotter
Incoherent
Coherent
Provider
Incoherent
Coherent
Provider
Incoherent
Coherent
Provider
Noisy
0.192 1 0.132
0.132 a 0.091
0.129 1: 0.090
0,186 1: 0.106
0.174 1: 0.092
0.347 1 0.072
0.278 1 0.139
0.155 1 0,217
0,414 1: 0.216
QRMForrmr
0,87 0.055
0.095
0.064
0.054
0.054 03035
0.061 0.039
0033
0.024 4 0.015
0.189
0.062 0.041
MLP
0.093 + 0.061
0.139 + 0.094
0.063 + 0.996
0.056 + 0.058
0.068 + 0.043
0.047
0.028
0.022 + 0.023
0.200 + 0.139
0.082 + 0.055
GNN
0 0.075
0.096 0.064
0.080 4: 0.089
0.063 +: 0.042
0.070 0.045
0.055
0.037
0.067 + 0.035
0.190 2: 0.156
0,080 =: 0.054
OLS
0217 1: 0.132
0.153 1: 0.097
0.117 1: 0.076
0.100 a: 0.061
0.123 4: 0.078
0.066 1. 0.040
0.054 1 0.039
0.200 1: 0.155
0.090 1: 0,066
ZNE
0,188 4 0.136
0,146 :: 0,103
0,116 1: 0.088
0.247 4: 0,144
0:438 2: 0,220
0,213 1 0.108
CDR
D003 0:043
0.101 0.077
0.059
0.175 0.162
0.182
0.405 0.515
RF
0,127 0.093
0,120
= 0.085
0.111 + 0.090
0.059 + 0.035
0.185 = 0.151
0.096
= 0.066
GTraQEEM
0.089 0.060
0.103 à 0.069
0.054 0.037
0,070 0.014
0.015
0.030
1 0.157
0.033
(3) Feature Concatenation and Regressor. Let mmip E Rd be the MLP branch output from Eq. 6, and Zate E IRd be the final Graph Transformer output. We concatenate these two vectors with the global feature u E Rdv:
RMSE. The Root Mean Squared Error (RMSE) quantifies the square root of the average squared differences between the predicted values and the actual ground truth values. A lower RMSE indicates higher prediction accuracy, reflecting the model's 's effectiveness in minimizing prediction errors.
Emerged Zuulps Zutt; u]
W merge E R Rewersee
(12)
AE and MAE. The Absolute Error (AE) for each prediction measures the absolute difference between the predicted and actual values, i.e., ý and y, and the Mean Absolute Error (MAE) is the average of these absolute errors among dataset. Formulas are provided in Appendix D.
where W merge E R tast +du Xd is trainable.
Finally, we feed Zmerged into an additional 2-layer MLPs (with batch normalization, dropout, nonlinear activation) to obtain the final prediction of the ideal expectation value:
5.2. Compared Methods
ÿ- = MLP regl Zmergoa),
(13)
We evaluate several baseline methods for quantum error mitigation, encompassing machine learning approaches and classical statistical models. Below, we briefly describe each method. For ML-based methods, we implement:
where ğ denotes the error-mitigated expectation value. This two-branch design enables the model to simultaneously focus on gate features (via MLPs) and explore long-range patterns as well as circuit topology (via Graph Transformer), ultimately enhancing the predictive accuracy for QEM.
Vanilla MLPs. We adopt the standard MLP architecture utilized in Kim et al. (2020a) and Liao et al. (2024), two- layer MLP with ReLU activation and batch normalization. The second layer concatenates the noisy expectation value to predict the ideal one.
5. Benchmark Experiments
5.1. Setups and Evaluation Metrics
wempleeent GNNs. Following Liao et al. (2024), we implement a GNN that begins with linear projection of the input features and then applies multiple Transformer- based convolutional layers interleaved with adaptive pooling operations. The layer-wise representations are combined using Jumping Knowledge mechanism to capture multi-scale information. Finally, the aggregated embedding is concatenated with the noisy EV and passed through fully connected layers to produce the ideal EV.
The quantum circuits in our experiments are simulated and executed on IBM's backend, and the 50-qubit circuits are executed on the IBM Kyiv device. See details for system and model configurations in Appendix A.
Table 4. Results of various QEM techniques on real-device
datasets. Best entries are red, second-best entries are blue.
Metool
MAE
Kyrk 2ag
KMSE
STD
MAE
Ksl
RMSE
STD
MAE
Rotdues
XM5E
STD
MAE
Rischre Rese
RMSE
Nakey
0187
0099
11009
0.152
0.254
11.10
4.319
0.261
11:44
13307
5/200
11430
OCS
0.007
0015
0.06
0.105
0.333
1130
0.120
0.183
0.136
0.154
3.390
10333
Random Forest (RF). An ensemble of 100 decision trees trained on bootstrap samples, each split selecting a random
0000
0074
0.000
0166
0.725
072
18096
01661
0131
0133
0.733
03312
XLP
03/20
0002)
0.520
0.151
0.220
0.163
0.213
0.122
0.144
9.27
0.235
25D
0.997
0100
0.135
0.315
03355
0.19
5690
RF
0.000
0337
0.02
0.100
0.277
0.215
0.075
0.174
0.158
0.165
3.29
0.247
GTTangEM
13009
00027
0.00
0.10
0.234
11300
0.107
0171
0.333
01360
3.30
01314
7

---

QEM-Bench: Benchmarking Learning- based Quantum Error Mitigation and QEMFormer as a Baseline
Stadded Treter Prokere
Stndert- Prdlee
Dranded Oible Ocheree
Advarcad cod. Twiter Step thot. incement
Ideence- fannern Sax Fraddr
Idvanced Ureeen
1846-bew Tebor har
Figure 3. Distribution of Absolute Errors across 8 settings, with setting names, circuit types, and noise types indicated above each figure.
Lower AE values (closer to 0) represent better performance.
MLP's RMSE of 0.063 and GNN's RMSE of 0.080. Simi- larly, for Trotter circuits with incoherent noise, QEMFormer exhibits the best performance with an RMSE of 0.024.
6121
1.00
02
On Advanced Settings. QEMFormer also shows strong performance in advanced settings, generalizing and extrapo- lating over circuit size, Trotter steps, and unseen Pauli-basis observables (see Tab. 2). For example, under the Trotter step zero-shot setting with incoherent noise, QEMFormer achieves an RMSE of 0.025, outperforming other methods. Furthermore, it consistently performs best under the co- herent noise setting, which is more complex compared to incoherent and fake provider settings.
Figure 4. AE over circuit size for random circuits (left) and Trotter steps for Trotterized circuits (right) in the zero-shot task. The training uses circuits with sizes < 100 or Trotter steps < 15.
subset of features and using mean squared error reduction as the criterion (Liao et al.. 2024). The final prediction is the average of all tree outputs, allowing for the modeling of complex, nonlinear dependencies.
GTranQEM. A quantum error mitigation framework that employs a non-message-passing graph transformer architec- ture, introduced in (Bao et al., 2025).
On Large-Scale Circuits. Tab. 4 reports results for 50- qubit circuits on IBM Kyiv and 63-qubit circuits on IBM Brisbane. QEMFormer attains the lowest errors on both platforms-
MAE of 0.018 and RMSE of 0.026 on the Kyiv Pre, and an MAE of 0.123 and RMSE of 0.272 on the Brishane Raw the-bhyreby demonstrating its robustness to real-device noise, with or without outlier filtering.
For non-ML-based mitigators, we implement:
OLS. Ordinary Least Squares (OLS) regression model (Liao et al., 2024) is an approach assuming a linear relationship between the target variable and the input features. This model identifies optimal coefficients by minimizing the sum of squared residuals between observed and predicted values.
We employ violin plots to illustrate the distribution of AE across test sets for comprehensive evaluation (see Fig. 3) and examine how AE varies with circuit size and Trotter steps in random and trotter zero-shot settings, respectively (see Fig. 4), demonstrating the stability of QEMFormer performance. Overall, these three categories of experi- ments demonstrate that our approach offers (a) superior or near-best accuracy across varied circuit types and noise conditions, (b) stable performance that generalizes to new operations or eircuit sizes, and (c) strong real-hardware applicability on large-scale circuits.
CDRs. Clifford Data Regression (CDR) (Czarnik et al., 2021) is trained on circuits modified with Clifford gate replacements and is evaluated on the original circuits. CDR leverages this data to learn corrections for error mitigation.
For reference, we also report results from Zero-Noise Ex- trapolation (ZNH) (Temme et al., 2017: Li & Benjamin, 2017), cornerstone non-ML technique in QEM. 5.3. Experimental Results Analysis
6. Conclusion and Outlook
On Standard Settings. Tab. 3 presents the RMSE across nine standard datasets. QEMFormer consistently achieves top or second-best performance across configurations. For instance, on random circuits with fake providers, QEM- Former attains the lowest RMSE of 0.054, compared to
We have presented comprehensive benchmark, QEM- Bench for quantum error mitigation, as well as strong baseline QEMFormer. All resources will be released and the benchmark would be updated continuously.
8

---

QEM-Bench:
Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as Baseline
Impact Statement
Czarnik, P., Arrasmith, A., Coles, J., and Cincio, L. Error mitigation with elifford quantum-cireuit data. Quantum, 5:592, 2021.
Quantum computing holds transformative potential for Al and beyond. yet its practical realization is hindered by quan- tum noise. This paper establishes a standardized benchmark for evaluating quantum emrrr mitigation (QEM) techniques, addressing the need for fair and transparent comparisons. Additionally, we propose novel QEM method, rigorously validated through extensive experiments. By facilitating open benchmarking, we believe that our work potentially fosters the development hardware-agnostic error mitiga- tion strategies, accelerating progress toward fault-tolerant quantum computation and advancing the broader quantum ecosystem.
Daley, A. J., Bloch, I., Kokail, C., Flannigan, S., Pearson, N., Troyer, M., and Zoller, Practical quantum advantage in quantum simulation. Nature, 607:667 676, 2022.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019.
Farhi, E., Goldstone, J., and Gutmann, S. A quantum ap- proximate optimization algorithm, 2014.
Gisin, N., Ribordy, G., Tittel, W., and Zbinden, H. Quantum cryptography. Rev. Mod. Phys., 74:145-195, Mar 2002. doi: 10. 10.110//ewMooPhyy77......
References
Agarwal, A., Lindoy, L. P., Lall, Jamet, F., and Rungger, I. Modelling non-markovian noise in driven supercon- ducting qubits. Quantum Science and Technology, 9(3): 035017, 2024.
Guerreschi, G. G. and Matsuura, Y. Qaoa for max- cut requires hundreds of qubits for quantum speed-up. Scientific Reports, 9(), May 2019, ISSN 2045-2322. doi: 10.1038/s41598 100386641598-019-431666664
Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., Biswas, R., and Boixo, S. Quan- tum supremacy using a programmable superconducting processor. Nature, 574(7779):505-511 Oct 2019. ISSN 1476-4687. doi: 10.1008//41186-019991
Harrigan, M. and Sung, K. J. Quantum approximate optimization of non-planar graph problems on pla- nar superconducting processor. Nature Physics, 17 (3):332 February 2021. ISSN 1745-2481. doi: 10. 10.10338/41167-0000011 0110y-
Bao, T., Ye, X., Ruan, H., Liu, C., Wu, W., and Yan, Be- yond circuit connections: non-message passing graph transformer approach for quantum error mitigation. In The Thirteenth International Conference on Learning Representations, 2025.
He, Z., Zhang, X., Chen, C., Huang, Z., Zhou, Y., and Sim, H. A gnn-based predictor for quantum architecture search. Quantum Information Processing, 22(2):123 Feb 2023. ISSN 1573-1332 doi: 10.1007/s1 10011112--2-0000 023-03881--2
Beale, S. J., Wallman, J. J., Gutiérrez, M., Brown, K. R., and Laflamme, R. Coherence in quantum error-correcting codes. arXiv preprint arXiv: 1805.0880 2018.
Huang, H.-L., Xu, X.-Y., Guo, C., Tian, G., Wei, S.-J., Sun, X., Bao, W.-S., and Long, G.-L. Near-term quantum computing techniques: Variational quantum algorithms, error mitigation, circuit compilation, benchmarking and classical simulation. Science China Physics, Mechanics & Astronomy, 66(5):250302, 2023.
Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., and Lloyd, S. Quantum machine learning. Nature, 549(7671):195-202 2017.
Huggins, W. J., McArdle, S., O'Brien, E., Lee, J., Ru- bin, N. C., Boixo, S., Whaley, K. B., Babbush, R., and McClean, J. R. Virtual distillation for quantum error miti- gation. Physical Review X, 11(4), November 2021. ISSN 2160-3308. doi: 10.1 103/physrexx. 1 11.00m00ee.....101666.6 11.041036.
Blume-Kohout, R., da Silva, M. P., Nielsen, E., Proctor, T., Rudinger, K., Sarovar, M., and Young, K. A taxonomy of small markovian errors. PRX Quantum, 3(2)002,335, 2022.
Bravyi, S., Dial, O., Gambetta, J. M., Gil, D., and Nazario, Z. The future of quantum computing with superconducting qubits. Journal of Applied Physics, 132(16), October 2022. ISSN 1089-7550. doi: 10. 00082955
Kandala, A., Temme, K., Córcoles, A. D., Mezzacapo, A., Chow, J. M., and Gambetta, J. M. Error mitigation ex- tends the computational reach of a noisy quantum pro- cessor. Nature, 567(7749):491 -455,99, Mar 2019. doi: 1040-7.
Brooks, M. Beyond quantum supremacy: the hunt for useful quantum computers. Nature, 174 2019.
10.
Kim, C., Park, K. D., and Rhee, J.-K. Quantum error mit- igation with artificial neural network. IEEE Access, 8: 188853-188860, 2020a. doi: 10.1109/ACCESS.200
Cai, Z., Babbush, R., Benjamin, S. C., Endo, S., Huggins, J.J., Li, Y., McClean, J. R., and O Brien, E. Quantum error mitigation, 2023.
3031607.

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as Baseline
Kim, C., Park, K. D., and Rhee, J.-K. Quantum error mit- igation with artificial neural network. IEEE Access, 8: 188853-188860, 2020b.
Zhou, L.,, Wang, S.-T., Choi, S., Pichler, H., and Lukin, M. D. Quantum approximate optimization algorithm: Performance, mechanism, and implementation term devices. Physical Review X, 10(2). June 2020. ISSN 21 2060-33. doi: 10. 103/hhysrevx 10.021062
near-
Li, and Benjamin, C. Efficient variational quantum simulator incorporating active error minimization. Phys- ical Review X. 7(2), June 2017. ISSN 2160-3308. doi: 10.1103pphysrevx.7.220000
Liao, H., Wang, S., Sitdikov, I., Salcedo, C., Seif, A., and Minev, Z. Machine learning for practical quantum error mitigation. Nature Machine Intelligence, 6(12): 1478-1486, December 2024.
Liao, M., Zhu, Y., Chiribella, G., and Yang, Y. Noise- agnostic quantum error mitigation with data augmented neural models. npj Quantum Information, 1 11(1):8, Jan- uary 2025.
Moflic, I., Garg, V., and Paler, A. Graph neural network autoencoders for efficient quantum circuit optimisation, 2023.
Nielsen, M. A. and Chuang, L. Quantum Computation and Quantum Information. Cambridge University Press, 2000.
Peruzzo, A., McClean, J., Shadbolt, P., Yung, M.-H., Zhou, X.-Q.. Love, P. J., Aspuru-Guzik, A., and O'Brien, J. L. A variational eigenvalue solver on a photonic quantum pro- cessor. Nature Communications, 5(1), July 2014. ISSN 2041-1723. doi: 10.10338ncomms521..
Preskill, J. Quantum computing in the nisq era and beyond. Quantum, 2:79, August 2018. ISSN 2521-327X. doi:
(Q-233314-2018-08--06
Suzuki, S., Inoue, J.-i., and Chakrabarti, B. Quantum Ising Phases and Transitions in Transverse Ising Models, vol- ume 862. Springer, 01 2013. ISBN 978-3-642-33038-4, doi:
Temme, K., Bravyia S., and Gambetta, J. M. Error mitigation for short-depth quantum circuits. Physical Review Letters, 119(18), November 2017. ISSN 1079-7114, doi: 10. 103/ 1
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, I.., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Zhang, H., Han, X., Zhang, G., Li, L., Cheng, L., Wang, J., Zhang, Y., Xia, Y., and Xia, C. Non-markovian noise mitigation in quantum teleportation: enhancing fidelity and entanglement. Scientific Reports, 14(1)223885, 2024.
10

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as Baseline
A. System and model configuration details
Our pipeline QEMFormer is implemented on a system running Ubuntu 20.04.4, with CUDA 12.2, PyTorch 1.13.0, and PyTorch Geometric 2.3.1. Most experiments are conducted on a server with NVIDIA GeForce RTX 4090 GPUs with 24 GB CUDA memory, two AMD Ryzen Threadripper 3970X 32-Core Processors at 3.70 GHz and 128 GB RAM. Additionally, detailed model hyper-parameters are presented in Tab. 7.
B. Circuit Structure Visualization
Visualization examples of different types of quantum circuits is shown Fig. 5.
Trotterized TFIM Circuits
Random Unstructured Circuits
QAOAA Circuits for MaxCut
U1
RX
RZ
RZ
RX
RX
RZ
X
RZ
RX
RZ
RZ
Figure 5. Examples of three types of quantum circuits.
C. Quantum Errors
Quantum error sources pose significant challenges to the advancement of quantum computing. These errors can be catego- rized as coherent or incoherent and as systematic or random, arising from factors such as imperfect qubit control, unwanted interactions, and measurement inaccuracies. In this work, conduct quantum enror mitigation (QEM) experiments targeting incoherent errors, coherent errors, and readout errors.
C.1. Incoherent Errors
Incoherent errors result from stochastic interactions between qubits and their environment, leading to non-unitary dynamics that cause decoherence and the loss of quantum information. These errors typically modeled random Pauli operators acting on the physical qubits under the assumption of a memoryless environment. We consider two primary types of incoherent errors: bit-flip errors and depolarizing errors.
Bit-Flip Errors. A bit-flip channel induces transitions between the computational basis states (0) and [1) with probability of p. This channel is described by the Kraus operators:
1 0
Es = = = vp
(14)
0
1
0
E = -1=1--pX- VI-P
(15)
where is the Pauli-X operator. The action the bit-flip channel on quantum state p is given by:
hal(p] = p XpX.
(16)
Depolarizing Errors. The depolarizing channel represents scenario where a single qubit replaced by the maximally mixed state with probability p. while remaining unchanged with probability p. Mathematically, the depolarizing channel dD acts on state p
Pelp = P1 + (1 (1-p)
(17)

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as a Baseline
Table Mean Ahsolute Error (MAF) across different circuit types and noise models for the standard setting. Lower MAF values indicate better performance. Multiple implementations of ZNE and CDR consistently performed weakly on QAOA circuits and are therefore excluded from the mitigation results to ensure a comparison.
Random
QAOA Coherent
Trotter Coherent
Method
Incoherent
Coherent
Provider
Incoherent
Provider
Incoherent
Provider
Noisy
0.140
0.092
0.147
0.128
0.241
0.354
QEMFormer
0.067
0.071
0.036
0.039
0.047
0.026
0.019
0.102
0.047
MLP
0.073
0.103
0,042
0.041
0.053
0.037
0.023
0.145
0.061
GNN
0.082
0.072
0,054
0.047
0.054
0,040
0.023
0.110
0.059
OLS
0.172
0.118
0.089
0.080
0.096
0.052
0.038
0.126
0.061
ZNE
0.130
0.103
0.076
0.245
0.379
0.184
CDR
0.047
0.079
0,042
0.066
0.129
0.255
RF
0.086
0.085
0,066
0.048
0.108
11.69
GTranQEM
0.074
0.076
0.048
0.040
0.054
0.034
0.024
0.122
0.038
Table 6. MAE across different circuit types and noise models
Random Size Zero Shot
Trotter Step Zero Shot Coherent
Unseen Obs
Method
Incoberent
Coherent
Provider
Incoberent
Provider
Incoberent
Coherent
Provider
Noisy
0.135
0.095
0.095
0.296
0.476
0.507
0.072
0.067
0.074
QEMFormer
0.077
0.088
0.048
0.017
0.090
0.058
0.060
0.050
0.062
MLP
0.091
0.093
0.074
0.039
0.107
0.101
0.085
0.097
0.069
GNN
0.151
0.123
0.051
0.018
0.100
0.107
0.065
0.075
0.068
OLS
0.210
0,132
0.123
0.019
0.162
0.081
0.098
0.066
0.087
ZNE
0.132
0.112
0.078
0.296
0.451
0.254
0.078
0.064
0.072
CDR
0.087
0.088
0.057
0.022
0.184
0.083
0.072
0.061
0.073
0.084
0,090
0.055
0.046
0.112
0,222
0.078
0.064
0.069
GiraaQHM
0.080
0.101
0.070
0.076
Recognizing that for any P.
p+ XpX + YpY + ZZZ
2
4
we can express the depolarizing channel as:
$ D [pl
(1
3p
p (XpX +YX ZpZ).
P 4
(18)
This formulation reveals that the depolarizing channel comprises the operators
3p 4
vi X,
vp
C.2. Coherent Errors
Coherent errors stem from unintended or imperfect unitary operations within quantum circuits. These errors can be modeled by unitary operators of the form:
U = e
(19)
where 0 = (01,..., .,.....,,x,,, quantifies the magnitude of the coherent error across the N, Pauli basis operators. Coherent errors transform pure quantum states into other pure states while maintaining quantum coherence due to their unitary nature. Despite preserving the purity of states, coherent errors can significantly undermine the reliability and accuracy of multi-qubit quantum computations.
D. Evaluation Metrics
In this section, we provide a comprehensive overview of the evaluation metrics employed in this manuscript. These metrics are essential for interpreting and assessing the performance of the proposed error mitigation techniques.
12

---

QEM-Bench: Benchmarking Learning-based Quantum Error Mitigation and QEMFormer as Baseline
D.I. Root Mean Square Error (RMSE)
Root Mean Square Error (RMSE) is widely used metric that measures the average magnitude of the prediction errors. provides insight into the model's accuracy by penalizing larger discrepanctes more heavily than smaller ones, Mathematically. RMSE is defined as:
1 N
RMSE
U4 y/ )
where:
NN is the total number of quantum circuits evaluated,
di represents the predicted value for the i-th circuit,
Vi denotes the actual ground truth value for the i-th circuit.
RMSE provides a single scalar value that summarizes the model's predictive performance, making it easier to compare different error mitigation strategies.
D.2. Absolute Error (AE)
Absolute Error (AE) quantifies the absolute difference between the predicted and actual values for each data point. Unlike RMSE, AE does not square the errors, thus treating all errors uniformly regardless of their direction or magnitude. is defined
AE
1
Val
where:
ÿi is the predicted value for the i-th circuit,
Vi is the actual ground truth value for the i-th circuit.
AE provides straightforward measure of prediction accuracy for each quantum circuit, highlighting the exact deviation without emphasizing larger errors disproportionaaely.
D.3. Mean Absolute Error (MAE)
Mean Absolute Error (MAB) aggregates the absolute across all data points provide overall measure of prediction accuracy. It offers an interpretable average of the absolute discrepancies between predicted and actual values. MAE is mathematically expressed as:
MAE
AE
10 - 34l
N
N
where:
N is the total number of quantum circuits evaluated,
AE, is the absolute error for the i-th circuit.
MAE provides an easily interpretable metric that reflects the average prediction error across all evaluated circuits, facilitating the comparison different error mitigation techniques in terms of their overall accuracy.
13

---

QEM-Bench: Benchmarking Learning-basse based Quantum Error Mitigation and QEMFormer as Baseline
Standard Toteer Cohumet
Standard Tuster Proider
Standard Aandem
Scandar:: fandon Conumrt
23-4
10-4
34-4
GAGA Ihamaart
OABA
Advanced Trater
Tetter Stp Zan Shadk
Adweeedd Kander Shet
ADVATENE Randon Son Zaro Srat
Adurced: Incchenet
Mtnuced- Urase Nut houberr
1344
2944
10-4
3844
Raw
Drtzane
Telle
2
Figure Violin plot depicting distribution of AE, The setting names, circuit types, noise configurations are annotated at top each figure.
D.4. Discussion of Metric Selection
The chosen
AE, and MAE MAB-collectively offer robust framework for evaluating the performance of error mitigation methods. RMSE emphasizes larger errors, making it suitable for identifying models that may have occasional significant deviations. AE provides granular view of individual prediction errors, while MAE offers balanced average that is less sensitive to outliers compared to RMSE. Together, these metrics ensure a comprehensive assessment of the error mitigators' effectiveness across various scenarios.
E. Additional Experimental Results
We present additional supplementary results alongside those in Section 5.3. Specifically, summarize the Mean Absolute Error (MAE) for both the standard and advanced settings in Tabs 5 and 6, respectively. QEMFormer consistently outperforms other baselines, achieving the best or second-best results across both datasets. Furthermore, we include a violin plot illustrating the distribution of AE in Fig. 6, complementing the violin plots presented in Fig. 3 of the main text. Together, the tables and violin plots demonstrate the efficacy of QEMFormer compared to other baselines, highlighting its strong generalization capabilities and potential for handling large-scale circuits executed on real quantum devices.
14

---

QEM-Bench: Benchmarking Learning-baser Quantum Error Mitigation and QEMFormer as Baseline
Table 7. Model architecture configurations for different experimental settings.
Setting
MLP Layers
Graph Trans Conv Layers
Hidden Dim
Real Pre
128
Real Raw
4
128
QAOA-Coherent
128
QAOA-Incoherent
128
QAOA-Providee
4
128
Random-Coherent
Random- Coherent
128
Random-Provider
128
Random-Incoherent
256
Random-Sizee Sere--err-Shho Chot-Coherent
512
Random-Size Shot-Inccherent
512
Random-SSie--Znn-Shor Swie
512
Trotter-Coherent
128
Trohrer-Incoheeen
64
Trotter-Provider
64
PooteerSteep-erov Coherent
128
64
Trotter-Step-7Zen-SSoo- Provider
128
Drneen--Ob-Inccheree
4
128
Coherent
64
Unseen-Ohne Saree--Ohs-Provdee
128
15


Reviews:
[{"review_id": "EkXvZLmAdw", "reviewer": "Reviewer_fTJA", "summary": "The authors make two primary contributions in their manuscript. First, they compile QEM-Bench, a set of twenty datasets that the community can use to benchmark ML-based approaches to quantum error mitigation (QEM). Second, they introduce a new ML-based approach to QEM called QEMFormer, which combines multi-layered perceptrons (MLPs) and graph transformers to predict the true expectation value of a quantum circuit based on the noisy measurement statistics. Comparisons between QEMFormer and other ML-based QEM methods are made on QEM-Bench. The authors claim that these results show that QEMFormer is generally superior to existing methods across QEM-Bench.\n\n## update after rebuttal\nMost of my concerns were addressed with post-review edits. I’ve raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and inquiries. **We summarized all Tabs. and Figs. of the newly added experiments at [this link](https://anonymous.4open.science/r/Rebuttal-fTJA-433C/rebuttal_%20fTJA.pdf).** \nBelow are our responses.\n\n> **1: RF [1] and GTraQEM [2] on QEM-Bench.**\n\nWe apologize for the oversight and now implement the RF in [1] and GTraQEM in [2] and evaluate them on 20 datasets from QEM-Bench (**Figs. 1&2, Tabs. 1-3**).\n\nAlthough [1] reported that the RF outperformed the MLP and GNN on a simple 4-qubit random circuits dataset, we consider it not convincing to state that \"the RF routinely beats the MLP and GNN in [1]\" as no further comparisons are conducted in other settings in [1]. Our comprehensive evaluation shows that RF's performance degrades in advanced settings such as trotter zero-shot.\n\nAlso, though GTraQEM shows competitive performance in certain settings, its non-message passing aggregation incurs high computational costs with increasing circuit depth, as constructing its structural matrix has an $O(n^3)$ complexity.\n\n> **2. How to obtain the ideal EV on the Kyiv device.**\n\n**The ideal EVs were computed solely based on the circuit, independent of any quantum device.** For 50-qubit circuits, we use the IBM Aer simulator for ideal simulation, specifically with:\n\n    simulator_ideal = AerSimulator(method='matrix_product_state')\n    estimator = Estimator(mode=simulator_ideal)\n    job = estimator.run([(circs, observables)])\n\n> **3. Large Error Bars?**\n\nThere may be confusion in our previous writing. To clarify, the reported error bars (**in Tab. 2 & 3 of the original manuscript**) refer to\n\n$$\n\\sigma_{\\text{dataset}} = \\sqrt{\\frac{1}{N} \\sum_{i=1}^{N} \\left(y^{\\text{miti}}_i - y^{\\text{ideal}}_i\\right)^2},\n$$\n\nwhere $N$ is the number of test data points. This metric quantifies the deviations of noisy (or mitigated) results from the ideal values across the dataset. **It is not a measure of model reproducibility across different random seed runs**, computed by:\n\n$$\n\\sigma_{\\text{stability}} = \\sqrt{\\frac{1}{K} \\sum_{k=1}^{K} \\left(\\text{MAE}_k\\right)^2} \\quad \\text{(or similarly for RMSE)},\n$$\n\nwith $K$ representing the number of runs.\n\nTo show this distinction, we executed QEMFormer under five different random seeds, summarized in **Tab. 4**.\n\nWe will clarify it in our main paper:\n\n1. The models show a small $\\sigma_{\\text{stability}}$, indicating that the mean performance is reliable for comparison.\n2. The relatively large $\\sigma_{\\text{dataset}}$ reflects the variable impact of noise on different circuits, leading to diverse deviations from the ideal outcomes.\n\n> **4. Difference between GNN in [1] and QEMFormer** \n\nWe would like to note that **QEMFormer is an integrated architecture rather than a mere refinement of the GNN in [1]**. A comparison is shown in **Tab. 5**. Importantly, **QEMFormer experimentally outperforms the GNN in [1] among most settings**, demonstrating its two-branch design's inherent well-suitedness for the quantum system.\n\n> **5. How QEM-Bench came up with?**\n\nWe apologize for the unclear expression. Regarding the design of the QEM-Bench:\n\n1.  QEM-Bench is built on insights from existing QEM and quantum computing research **to ensure it includes the key concerns of the community**. \n   \n2. **Key Enhancements:**  \n   - **Structural Diversity:** Incorporates representative QAOA circuits.  \n   - **Circuit Complexity:** Enriches gate types and parameter selection in random circuits.  \n   - **Noise Characterization:** Include coherent noise.  \n   - **Evaluation Scope:** Expands zero-shot settings to test generalization ability.  \n   - **Real-World Data:** Construct large-scale circuit datasets executed on quantum devices with ideal EVs as labels.\n\nBy integrating these enhancements with established research, QEM-Bench aims to address the need for a standardized benchmark evaluation dataset for ML-QEM techniques. \n\n> **6. Are noise parameters fixed across error models?**\n\nWe politely clarify that **QEM-Bench does incorporate different noise strengths across error models.** Incoherent noise parameters are derived from the real device, the Sycamore [3], and the parameters of real devices and simulators provided by IBM are **not manually set**, thereby capturing a realistic and diverse range of noise regimes.\n\n> **7. How far do the noisy EVs differ from the ideal EVs?**\n\nWe respectfully note that **the error distributions, MAE, and RMSE of raw data are detailed in Tabs 2-4 and Figs 3, 4 & 6 of the original manuscript**. \n\nWe hope the reply eases your concern. Should you have any further inquiries, we would be pleased to offer responses.\n\n**References:**\n- [1] Machine learning for practical quantum error mitigation. *Nature Machine Intelligence, 2024*\n- [2] Beyond circuit connections: a non-message passing graph transformer approach for quantum error mitigation. *ICLR 2025*.\n- [3] Quantum supremacy using a programmable superconducting processor, *Nature 574, 2019*", "claims_and_evidence": "The authors provide modest evidence for their claim that QEMFormer outperforms the other ML-based QEM methods that they tested. QEMFormer routinely achieves the lowest or second lowest root mean squared error on the various datasets in QEM-Bench. However, the large error bars make it difficult to ascertain if this performance is statistically significant. I would like to see more rigorous analysis to support their claim. \n\nThe authors also overlook several competing ML-based QEM methods that have shown good performance. They chose not to include the random forest model from [1] in their paper, despite the random forest model routinely beating the MLP and GNN in [1] (which are both included). The authors also do not compare their method to the graph transformer approach in [2].", "methods_and_evaluation_criteria": "The overall method of comparing several competing models on multiple datasets is sound. The use of RMSE, absolute error (AE), and mean absolute error (MAE) is also typical, as is reporting error bars. However, the large error bars mean that you need to perform additional statistical analyses to show that QEMFormer truly outperforms the other models.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I greatly appreciated that the authors included data simulated using both incoherent (i.e., stochastic) and coherent error models. A lot of papers overlook coherent errors, despite evidence that they are much harder for ML approaches to model. Nonetheless, I question the value of benchmark sets simulated under fixed noise parameters. By not varying the noise strengths across error models it is hard to get a good sense for how models perform in a variety of noise regimes, and the community risks training to the standard rather than truly probing their models. \n\nIt is also hard to judge how hard these benchmarks are from the data presented. For instance, no analysis is presented showing how far the noisy expectation values differ from the true expectation values in each dataset.", "relation_to_broader_scientific_literature": "See my comments in the “Claims and evidence” and “Other comments or suggestions” section about missing models and less-than-ideal descriptions of other works.\n\nAlso, it isn’t really explained how they came up with these benchmarks. For instance, both one-dimensional transverse field Ising model circuits and mitigating unseen observables were used in [1]. You should credit that work!", "essential_references_not_discussed": "[1] H. Liao, D. Wang, I. Sitdikov, C. Salcedo, A. Seif, and Z. Minev. Machine learning for practical quantum error mitigation. Nature Machine Intelligence, 6(12), December 2024.\n[2] T. Bao, X. Ye, H. Ruan, C. Liu, W. Wu, J. Yan. Beyond circuit connections: a non-message passing graph transformer approach for quantum error mitigation. ICLR 2025.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "The description of the GNN modelled on [1] fails to convey how similar it is to the GNN branch of QEMFormer. Yes, they use an undirected acyclic graph instead of a direct acyclic graph and a slightly different architecture, but when phrased this way QEMFormer doesn’t seem like that big of an advancement.", "questions_for_authors": "1. How does the random forest model in [1] perform on QEM-Bench?\n\n2. How does the graph transformer model in [2] perform on QEM-Bench?\n\n3. How did you get the true expectation values for the one-dimensional TFIM circuits run on ibm_kyiv? Are exact solutions known?\n\n4. Given the large error bars, how do you know if the benchmarks have enough resolving power to meaningfully distinguish between different approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hIYOlPVncI", "reviewer": "Reviewer_6Drn", "summary": "The paper introduces QEM-Bench, a benchmarking suite designed to evaluate machine learning-based Quantum Error Mitigation (QEM) techniques.\nThe benchmark includes 20 datasets covering various circuit types and noise models to standardize QEM evaluation.\nFurthermore, the paper proposes QEMFormer, a novel learning-based QEM method that improves quantum error mitigation by leveraging both short-range and long-range dependencies with quantum circuits.\nThe paper evaluates QEMFormer against various baselines, showing its superior performance across different circuit families, noise configurations, and real quantum devices (IBM Kyiv 50-qubit experiments).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful questions and positive evaluation of our work. **We summarize the Tabs. and Figs. for newly added experiments at [this link](https://anonymous.4open.science/r/Rebuttal-6Drn-C2D8/rebuttal_%206Drn.pdf).**\n\nBelow are our responses to each question.\n\n> **Q1: Why does the paper evaluate IBM Kyiv only? There are other quantum computers in the IBM Quantum system.**\n\nInitially, we evaluated only IBM Kyiv due to the significant financial and time costs associated with executing large-scale quantum circuits on real quantum devices. Although multiple IBM devices are available, these constraints still limited our testing.\n\nTo improve the diversity of QEM-Bench and strengthen your confidence in our work, we have expanded our evaluation to include two datasets from the **IBM Brisbane** device: one with extreme outlier filtering (Brisbane Pre) and another with raw, unfiltered data (Brisbane Raw). \n\n- Detailed statistics for the two datasets are provided in **Tab. 1**\n- The performance of various QEM techniques is shown in **Tab. 2 and Fig. 1**. \n\nNotably, even with the more severe noise effects on the Brisbane device (compared to Kyiv), QEMFormer consistently outperforms the baseline methods.\n\nWe shall include these datasets and results in the revised manuscript.\n \n> **Q2: Is the evaluation on a 50-qubit system good enough? Why? If the circuits are deployed on a 127-qubit system, will the performance stay the same?**\n\nWe would like to clarify that the construction of our datasets on 50-qubit systems is limited by **the prohibitive computational cost of obtaining ideal EVs for circuits with over 100 qubits.** Although devices like Kyiv and Brisbane support up to 127 qubits, their outputs are inherently noisy. Hence, ideal EVs, which serve as the dataset labels, have to be obtained through classical simulation. Yet, to the best of our knowledge, the IBM simulators that do not restrict circuit structure to provide ideal simulation are limited to 63 qubits (namely, Aer matrix_product_state simulator), making 100-qubit simulations difficult under the current implementation structure of QEM-Bench.\n\nFurthermore, to the best of our knowledge, the only ML-QEM method exploring circuits beyond 100 qubits, [1], also demonstrates such difficulty in ideal EV obtaining in their work and thus **uses ZNE-mitigated results as training labels.** However, our experiments applying IBM’s built-in ZNE to both 50-qubit and 63-qubit circuits (on the Kyiv and Brisbane devices) show only marginal improvements over noisy outcomes, with significant residual errors (see **Fig. 1–2 and Tab. 2–3**). This suggests that using ZNE outcomes as labels may not provide a fair or reliable benchmark for larger systems.\n\nAccordingly,  due to the current time constraint, we plan to extend the inclusion of systems exceeding 100 qubits to future work. Nevertheless,  to enhance your confidence and further evaluate the scalability of various ML-QEM methods, we have developed the Brisbane Pre and Brisbane Raw datasets on **63-qubit systems** (results summarized in **Tab. 2 and Fig. 1**).  \n\nWe hope the reply eases your concern. If you have any additional questions, we would be pleased to provide further responses.\n\n**References:**\n\n[1] Machine learning for practical quantum error mitigation, *Nature Machine Intelligence, 2024*", "claims_and_evidence": "- QEM-Bench provides a comprehensive and standardized benchmarking suite for machine learning-based QEM techniques.\n\n- Based on experimental results, QEMFormer outperforms other machine learning-based and traditional QEM methods.\n\n- The two datasets with 50-qubit circuits executed on IBM Kyiv might not be flexible. This is because, in practice, the real quantum systems can have higher than 50 qubits. The benchmark on real quantum systems should show the scalability.", "methods_and_evaluation_criteria": "- The proposed benchmarking is well-motivated since prior QEM methods might have different evaluation protocols, which could make the evaluation unfair in some perspectives or unable to be compared because of a lack of reproducibility.\n\n- The proposed QEMFormer utilizing the long-range and short-range dependencies in Graph Transformer is well-aligned with quantum circuit structure, which can be represented as Directed Acyclic Graphs (DAGs).", "theoretical_claims": "The paper does not have any explicit theoretical claim.", "experimental_designs_or_analyses": "The proposed QEM-Bench shows the diversity of circuit designs and scenarios to evaluate the QEM methods.\nBesides, as mentioned in \"Claims And Evidence\", the evaluation on the IBM Kyiv system should be considered since it does not convince that the QEM methods can work in multiple systems.", "relation_to_broader_scientific_literature": "The paper can potentially standardize the evaluation of the QEM methods, making the comparisons fair and transparent.", "essential_references_not_discussed": "There are no additional related works that are essential to understanding the key contributions of the paper.", "other_strengths_and_weaknesses": "I don't have any comments on other strengths and weaknesses of the paper.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "I think the paper is solid in terms of the benchmark and proposed model.\nWhat I am concerned about is the evaluation on the real quantum devices:\n\n- Why does the paper evaluate on IBM Kyiv only? There are other quantum computers in the IBM Quantum system.\n\n- Is the evaluation on a 50-qubit system good enough? Why? If the circuits are deployed on a 127-qubit system, will the performance stay the same?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wETJ8WcH1T", "reviewer": "Reviewer_Qjn1", "summary": "This paper introduces a dataset for benchmarking quantum error mitigation techniques, as well as a graph-transformer model to serve as baseline. The dataset consists of three evaluation settings - each with different levels of added noise; Standard (general purpose testing with Trotterized TFIM Circuits, Random Circuits and MaxCut QAOA Circuits), advanced (testing aspects of generalization capabilities) and two large-circuit datasets, executed on real quantum hardware. The circuit data-samples are encoded directed acyclic graph representations and are provided with statistical information about the circuit itself (e.g. nr-gates, nr-parameters etc.), as well as the noisy and ideal measurement expectation values. To leverage the graph-representations and the informational feature vectors, a graph-transformer (QEMFormer) is introduced and compared on the proposed benchmark against a set of QEM algorithms from the literature.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and positive evaluation of our work. We acknowledge that quantum error correction is aimed to be solved on the hardware side ultimately. However, the significant qubit overhead associated with QEC renders it less feasible in the near term, especially for large-scale circuits, as illustrated in [1] and [2]. Therefore, during the NISQ era, ML-QEM methods offer an effective and efficient interim approach, and we hope our work lays a foundation for future studies. \n\nWe will revise the manuscript to clearly reflect this nuanced perspective and incorporate your valuable suggestions.\n\n**References:**\n\n[1] Quantum Error Mitigation, *Reviews of Modern Physics, American Physical Society, 2022*\n\n[2] Near-term quantum computing techniques: Variational quantum algorithms, error mitigation, circuit compilation, benchmarking and classical simulation, *Science China Physics, Mechanics & Astronomy, 2023.*", "claims_and_evidence": "Since the paper is mostly focused on introducing and specifying the benchmark dataset, I’d say the claims are rather on the neutral side. The QEMFormer is compared against other algorithms from the literature, where it performs consistently strong across most of the dataset settings. The evaluation is thorough, with a good selection of comparable algorithms of both ml-based and non-ml-based error mitigators.", "methods_and_evaluation_criteria": "Both the proposed dataset as well as the QEMFormer are well motivated. The evaluation criteria of (root, mean) absolute errors mitigated is ultimately the logical metric, although a more critical discussion on other factors such as run-time or complexity of the compared algorithms would have been beneficial.", "theoretical_claims": "No theoretical claims are made in this paper.", "experimental_designs_or_analyses": "The experimental comparison in Ch.5 is in itself quite simple, a direct comparison of the final absolute errors mitigated across different circuit-type and noise settings. All results are reported with mean and std.div. and appear to be sound.", "relation_to_broader_scientific_literature": "The benchmark should be helpful for a more standardized quantum evaluation, which is currently indeed rather lacking. A comprehensive, maintained dataset as proposed here would certainly help the field. The QEMFormer seems to be a performant, but on this data specialized baseline (i.e., the graph structure). \n\nUltimately, quantum error correction is a technical problem that needs to be solved to make QC in itself a technically sound computing device. As an intermediary solution ML-based mitigators may have their merits, but on a practical level QEM is a problem that will (need to) be solved on the hardware side, which is why I would not expect learning based QEM methods to stay around for too long.", "essential_references_not_discussed": "None that I am aware of. The discussion on related work covers the field decently well.", "other_strengths_and_weaknesses": "I generally have very little to critique on this paper, it is well written, formalized and visualized. While the contribution is generally rather “short-term”, as mentioned above, until QC hardware handles QEM natively, I find this benchmark a good current contribution to a current problem.", "other_comments_or_suggestions": "The colors in Fig.1 could be stronger, and Fig. 4 is too small to read.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2o6KvX3ufG", "reviewer": "Reviewer_iomM", "summary": "This paper presents QEM-Bench, a benchmarking suite for machine learning-based quantum error mitigation (ML-QEM), addressing the lack of standardized datasets in the field. The benchmark includes twenty datasets spanning different circuit types and noise models to enable consistent evaluation of ML-QEM techniques. The authors also propose QEMFormer, a two-branch architecture incorporating MLPs for short-range dependencies and Graph Transformers for long-range dependencies, leveraging directed acyclic graph (DAG) representations of quantum circuits. Empirical evaluations show QEMFormer outperforms other ML-QEM baselines across diverse settings, reinforcing the claim that a structured representation enhances mitigation performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful comments and inquiries, and the positive evaluation of our work. **We have summarized the newly added Figs and Tabs at [this link](https://anonymous.4open.science/r/Rebuttal-iomM-B8AB/rebuttal_iomM.pdf).**\n\nBelow are our responses.\n\n> **1. Real devices other than IBM Kyiv and traditional QEM techniques like ZNE on real devices.**\n\nWe apologize for the earlier oversight. We have now incorporated the following enhancements:\n\n- Two datasets derived from 63-qubit Trotter circuits executed on **IBM Brisbane** have been included—one with extreme outlier filtering (Brisbane Pre) and one without (Brisbane Raw).\n  - Dataset statistics are detailed in **Tab. 1**.\n  - The performance of various QEM techniques is provided in **Tab. 2** and **Fig. 1**.\n- We have evaluated ZNE across the datasets from Kyiv (Pre and Raw) and Brisbane (Pre and Raw), as detailed in **Fig. 1-2** and **Tab. 2-3**.\n\nWe would like to note that the CDR approach entails significant time costs due to the need to construct training sets for each circuit individually. Consequently, we plan to include its results in future revisions.\n\n> **2. Ablation studies of QEMFormer**\n\nWe now conduct two sets of analyses. \n- **Tab. 4** compares the performance impact of the MLP and Graph Transformer modules.\n- **Tab. 5** evaluates our multi-ranged feature extractor.\n\n> **3. RMSE as the sole evaluation metric?**\n\nWe apologize for any potential confusion. **In the original manuscript, we report RMSE (Tabs. 1 & 2), MAE (Tabs. 5 & 6), and standard deviation (Tabs. 1 & 2)** to provide a comprehensive evaluation. RMSE emphasizes larger deviations and is sensitive to outliers, while MAE directly measures the average error magnitude. Additionally, violin plots (**Figs. 3 & 6 of the original manuscript**) depict the full error distribution along with the STD.\n\n> **4. About prior QEM benchmarks.**\n\n**To the best of our knowledge, there are currently no standardized benchmarks for the ML-QEM task.** This gap, as also noted by reviewers Qjn1 and 6Drn, has been a primary motivation for this work.\n\n> **5. Dataset curation process and whether it reflects real-world noise** \n\nWe detail the dataset curation process in **Tab. 6**. Our design intentionally reflects real-world noise characteristics in two key ways:\n\n- **Diverse Circuits:** A single type of noise impacts various circuits differently. By including types of structured circuits and random unstructured circuits, QEM-Bench aims to gather a comprehensive depiction of real-world conditions.\n- **Broad Noise Modeling:** We incorporate multiple realistic noise sources, including data from real devices (Kyiv), fake providers (published by IBM) resembling the real devices, and manually set configurations based on statistics from representative real devices (e.g., Sycamore [1] for incoherent settings).\n\nThis design aims to let the QEM-Bench effectively mirror the diverse noise characteristics encountered in practice and bridge the current gap for further research. \n\n> **6. Hyperparameter tuning details.**\n\nWe include an analysis of how the number of layers and hidden dimensions in the MLP and Graph Transformer modules affect QEMFormer's performance (see **Fig. 3**). Our findings indicate that increasing the model size initially enhances model capability; however, excessively large models can lead to overfitting and degraded performance. We also politely noted that hyperparameter settings are provided in **Tab. 7 of the original manuscript**.\n\n> **7. Formal theoretical analysis.**\n\nWe appreciate the reviewer's suggestion. Our primary focus in this work is to introduce a comprehensive benchmark dataset that spans a variety of circuits and noise configurations, addressing a significant gap in ML-QEM studies. \n\nOn the method side, we specifically design a 2-branch neural network architecture in QEM tasks. The idea is that the multi-range context and dependency as frequently occurred in the quantum system can be better captured, and the effectiveness is also verified by our extensive empirical results. We plan to incorporate further qualitative analysis in the revised manuscript.\n\nDue to the inherent complexity of quantum systems and noise profiles, we leave a rigorous theoretical analysis of QEMFormer’s effectiveness for future work.\n\n> **8. Discussion about hardware-specific noise mitigation and circuit optimization techniques.**\n\nWe will expand the related works section to include discussions on hardware-specific noise mitigation techniques, hybrid classical-quantum optimization strategies, and ML-based circuit optimization methods. We will clarify the differences and relationships between these approaches and ML-QEM techniques, especially QEMFormer. \n\nWe hope the reply eases your concern. If you have any further questions, we would be pleased to respond.\n\n**References:**\n\n[1] Quantum supremacy using a programmable superconducting processor, *Nature volume 574, 2019.*", "claims_and_evidence": "5/10\n\nWhile the paper makes strong claims about the benefits of QEM-Bench and QEMFormer, the supporting evidence is limited in some areas. The benchmark provides a well-structured dataset, but its generalizability to real-world quantum hardware is not thoroughly demonstrated. The experimental results highlight improvements over prior ML-based approaches, but the paper does not compare against traditional QEM techniques like Zero-Noise Extrapolation (ZNE) in real-device settings. The justification for QEMFormer’s performance lacks ablation studies isolating the contributions of different architectural components.", "methods_and_evaluation_criteria": "6/10\n\nThe proposed evaluation framework is well-motivated, and QEM-Bench is a meaningful contribution. However, there are inconsistencies in evaluation settings—while the benchmark includes diverse noise models and circuit types, real hardware evaluations are limited to IBM Kyiv, raising concerns about applicability to other quantum architectures. The choice of Root Mean Squared Error (RMSE) as the primary evaluation metric is standard but insufficient, as the robustness of the mitigation method under varying noise intensities remains unexplored.", "theoretical_claims": "4/10\n\nThe paper lacks formal theoretical analysis of QEMFormer’s effectiveness beyond empirical results. The use of DAG representations and multi-range feature extraction is intuitively justified but not rigorously analyzed. Claims regarding the preservation of circuit topology and feature locality should be accompanied by theoretical bounds or complexity analyses. The paper cites relevant works on graph-based representations, but it does not explicitly demonstrate why QEMFormer outperforms existing methods from a theoretical standpoint.", "experimental_designs_or_analyses": "6/10\n\nThe experiments are comprehensive in terms of dataset coverage. The results confirm QEMFormer’s superiority over prior ML-QEM methods. While the authors benchmark across multiple noise models, hyperparameter tuning details are unclear, and there is no discussion on whether the performance gains hold for circuits larger than those evaluated. Furthermore, real-device evaluations are minimal, limiting the reliability of the reported findings.", "relation_to_broader_scientific_literature": "7/10\n\nThe paper is well-positioned within the ML-QEM literature, citing relevant works on machine learning for quantum error mitigation, benchmarking efforts, and graph-based circuit representations. However, there is no engagement with broader ML-based circuit optimization techniques, which could provide useful insights.", "essential_references_not_discussed": "6/10\n\nMost of the relevant references are cited. However, works on hardware-specific noise mitigation techniques and hybrid classical-quantum optimization strategies are not discussed comprehensively.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tQEM-Bench provides a standardized evaluation suite, which is a valuable asset for the field.\n2.\tQEMFormer’s hybrid approach to short- and long-range dependency modeling is innovative.\n3.\tThe inclusion of different noise models and circuit types enhances the credibility of QEM-Bench.\n4.\tThe experiments cover multiple baselines, demonstrating QEMFormer’s competitive performance.\n5.\tThe paper is well-organized and presents technical details in a clear manner.\n\nWeaknesses:\n\n1.\tLack of real-hardware validation beyond IBM Kyiv: Generalizability to other quantum platforms is not demonstrated.\n2.\tLimited ablation studies: The contribution of each architectural component in QEMFormer is not isolated.\n3.\tMinimal theoretical justification: Claims about circuit topology preservation and information retention are not rigorously analyzed.\n4.\tUnclear evaluation criteria: The choice of RMSE as the sole metric does not provide a full picture of mitigation effectiveness.\n5.\tLimited engagement with non-ML QEM methods: The paper does not sufficiently compare QEMFormer with traditional mitigation techniques.", "other_comments_or_suggestions": "1.\tInclude additional real-device evaluations beyond IBM Kyiv for broader applicability.\n2.\tProvide detailed ablation studies on QEMFormer’s feature encoding and architecture.\n3.\tCompare against non-ML-based QEM techniques more rigorously.\n4.\tOffer hyperparameter tuning details to improve reproducibility.", "questions_for_authors": "1.\tHow does QEMFormer perform on hardware other than IBM Kyiv?\n2.\tCan you provide ablation studies showing the contribution of each component of QEMFormer?\n3.\tWhy was RMSE chosen as the primary evaluation metric, and how does it compare to alternative metrics?\n4.\tHow does QEM-Bench compare to prior QEM benchmarks, if any exist?\n5.\tCan you clarify the dataset curation process and whether it reflects real-world noise characteristics?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The rebuttal addressed the concerns from the reviewers and provided comprehensive feedback. It is useful for assessing the paper's contributions. All reviewers recommend acceptance after discussion, and the ACs concur. The final version should include all reviewer comments, suggestions, and additional discussion from the rebuttal.
Decision: Accept (poster)