Benchmarking Quantum Reinforcement Learning
Authors: Nico Meyer
Christian Ufrecht
George Yammine
Georgios Kontes
Christopher Mutschler
Daniel Scherer
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Quantum Computing
Benchmarking
Sample Complexity
Quantum Machine Learning
Primary area: reinforcement_learning->everything_else
TL;DR: We introduce a statistical estimator for benchmarking sample complexity in quantum reinforcement learning.

Abstract: Benchmarking and establishing proper statistical validation metrics for reinforcement learning (RL) remain ongoing challenges, where no consensus has been established yet. The emergence of quantum computing and its potential applications in quantum reinforcement learning (QRL) further complicate benchmarking efforts. To enable valid performance comparisons and to streamline current research in this area, we propose a novel benchmarking methodology, which is based on a statistical estimator for sample complexity and a definition of statistical outperformance. Furthermore, considering QRL, our methodology casts doubt on some previous claims regarding its superiority. We conducted experiments on a novel benchmarking environment with flexible levels of complexity. While we still identify possible advantages, our findings are more nuanced overall. We discuss the potential limitations of these results and explore their implications for empirical research on quantum advantage in QRL.

Benchmarking Quantum Reinforcement Learning
Nico Meyer" 12
Christian Ufrecht
George Yammine'
Georgios Kontes'
Christopher Mutschler'
Daniel D. Scherer
Abstract
Model
Benchmarking and establishing proper statistical validation metrics for reinforcement learning (RL) remain ongoing challenges, where no consensus has been established yet. The emergence of quan- tum computing and its potential applications in quantum reinforcement learning (QRL) further complicate benchmarking efforts. To enable valid performance comparisons and streamline cur- rent research in this area, we propose novel benchmarking methodology, which is based statistical estimator for sample complexity and a definition of statistical outperformance Further- more, considering QRL, our methodology casts doubt on some previous claims regarding its su- periority. We conducted experiments on a novel benchmarking environment with flexible levels of complexity. While still identify possible ad- vantages, our findings are more nuanced overall. We discuss the potential limitations of these re- sults and explore their implications for empirical research on quantum advantage QRL.
small Classical
small Quantum
200k
large Classical
sample complexity
150k
100k
50k
0.35
0.8
0.4 threshold probability
relative error
0.25
0.6
0.15
Figure 1. Comparison of empirical sample complexities Š of dou- ble deep Q-learning and quantum version of the algorithm (lower is better), Sample complexity the number of environment- agent interactions to surpass performance threshold - with probabil- ity 8. The figure shows the result for the BeamManagement 6G environment introduced in this work. In order of decreasing sam- ple complexity: a small classical neural network with 2 hidden layers of width 16, i.c., 387 parameters; small quantum circuit with layers on 14 qubits, i.e., 437 variational parameters, inte- grated between fully connected classical layers with additional 101 parameters; a large classical neural network with hidden layers of width 64, i.e., 4611 parameters; The hybrid quantum model consistently outperforms the similar-sized classical network, and is also competitive with the 10-fold larger classical model.
1. Introduction
Reinforcement learning (RL) is a powerful algorithmic prim- itive increasingly applied across multiple domains (Arulku- maran et al., 2017; François-Lavet al., 2018). Quantum reinforcement learning (QRL) (Meyer al., 2022) a col- lection of RL algorithms developed for quantum computers, an emergent paradigm of computing that exploits the laws of quantum mechanics (Nielsen & Chuang, 2010). While some QRL algorithms with provable advantage over tra- ditional (classical) algorithmss have been proposed (Wang et al., 2021; Cherrat et al., 2023; Dunjko et al., 2016), these
algorithms are currently far out of reach for any existing quantum computing hardware. Hence, most of the work has focused on hybrid algorithms in which the deep neural net- work of traditional RL algorithm either approximating the policy, the value function, or both is replaced by vari- ational quantum circuit (VQC) (Bharti et al., 2022; Chen et al., 2020). However, they (while being less hardware- intensive) are heuristic in nature. Hence, there no proof of intrinsic advantage of the quantum version of the algorithm.
'Equal contribution Fraunhofer IIS. Fraunhofer Insti- tute for Integrated Circuits IIS. Nürnberg. Germany *Pattern Recognition Lab, Aaadrich-Alexannee--nniieeeee Nürnberg, Erlangen, Germany. Correspondence to: Nico Meyer <nico.meyer iir.merrre@....aaroorrrde... fraunhofferde>>.
Erlangen-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In the search for heuristic quantum advantage QRL, we encounter issues similar to classical RL: sensitivity to hyper-

---

Benchmarking Quantum Reinforcement Learning
parameter choices, various randomness sources, and even random seed and codebase (Henderson al., 2018; Jordan et al., 2024). QRL faces similar, if not more severe, repro- ducibility issues than traditional RL (Bowles et al., 2024; Franz et al., 2023) due to additional randomness. Therefore, any claim of QRL's 's superiority over classical RL should be taken with great care.
algoritimm
threshold
Inserquartile over 100 runs
pecformanco averaged
algoritiun
"How do we meaningfully assess a QRL agent outper- forms its classical counterpart and what does outperform mean in the context of quanium udvantage?" We adopt sample complexity. i.e., the number of interactions between the agent and the environment to achieve a certain perfor- mance (Kearns& & Singh, 1999; Kakade, 2003), as the central benchmarking metric due to inherent costly implications in real-world applications. As classical RL is notorious for its sample inefficiency (François-Land Lavet et al., 2018), po- tential quantum advantage in sample complexity presents an intriguing prospect of QRI..
number of Eiporithm-emvironment interactions
Figure 2. Inadequate reporting of two (Q)RL agents' performance can lead to false conclusions about sampling complexity. Although the curves may seem exaggerated. it is common practice in QRL studies to benchmark with such limited number of runs.
lem class for quantum computing, characterized by small input and output dimensions of classical data (Hoefler et al., 2023). Therefore, we opt for new benchmark environment to better suit the specific needs of quantum algorithm eval- uation. (3) We perform the most extensive computational analysis of classical RL vs. QRL done so far to the authors' knowledge. In our study, we compare different problem and algorithm configurations of different scales and complexity. Statistically robust results are obtained by performing 100 training runs per configuration, by far the largest population size found in the quantum computing literature.
In addition to common sources of randommess known in RL (Henderson et al., 2018), such as random weight ini- tialization, randomness in the environment, etc., QRL is subject to additional sources of randomness such as shot noise hardware imperfections due to current limitations. Therefore, performance comparisons need to be based on a sound statistical evaluation. However, most studies perform only a small, potentially insufficient number of training runs for robust inferences under stochasticity. As a result, state- ments are potentially misleading or insignificant. Figure?> 2 exemplifies the comparison of two RL algorithms w.r.t some threshold the evaluated return. It illustrates common flaws prevalent in the QRL literature, e.g., averaging learn- ing curves over only 5 seeds, inconsistency in statistical ranges, etc. Figure indicates that algorithm 1 is more sample-efficient. However, the interquartile ranges (shaded areas) estimated with a much larger sample size of 100 runs rather support the opposite statement.
The remainder of this paper is structured as follows. Sec- tion 2 reviews related work. Section 3 provides background on (Quantum) RL. Next, Section 4 introduces our statistical estimator. Section 5 describes the experimental setup and introduces a novel environment with flexible complexity. Section 6 evaluates QRL using our statistical estimator. We discuss the implications on quantum advantage in Section 7.
2. Related Work
This paper strongly advocates robust stochastic modeling backed by significance testing to meet reproducibility crite- ria (Henderson et al., 2018). The main contributions of this paper are as follows. (1) We propose a formal evaluation procedure by a statistical estimator for sample complexity, complemented by a robust notion of outperformance based on statistical significance. Importantly, both are designed (2) We design and implement a fast and flexible benchmarking suite based on problem inspired by real-world wireless communica- tion tasks. While community benchmarks (Brockman et al., 2016; Tassa et al., 2018b) are widely used, benchmarking quantum algorithms requires the option to flexibly scale difficulty and instance size of the task to allow extrapolation beyond classical simulatability of quantum algorithms. The task introduce belongs to potentially significant prob-
For classical deep RL, in general, no rigorous analysis of al- gorithms is possible, necessitating a computational approach to algorithmic comparison and representative benchmark- ing environments (Brockman et al., 2016; Todorov et al., 2012; Tassa et al., 2018a) for mature implementations of widely-used deep RL algorithms (Hill et al., 2018; Duan et al., 2016; Fujimoto et al., 2019; Wang et al., 2019b). However, benchmarking results are heavily influenced by (sometimes seemingly trivial) implementation details (En- gstrom et al., 2019; Andrychowicz et al., 2021; Huang et al., 2022), as well as randomness in environment transitions and network initialization (Henderson et al., 2018). Re- search has focused on determining the necessary number of seeds for robust comparison (Agarwal al., 2021; Colas et al., 2018) and on adopting statistical analysis over simple point estimates (Agarwal et al., 2021; Colas et al., 2018;
for the benchmarking of heuristic algorithms.
2

---

Benchmarking Quantum Reinforcement Learning
Patterson et al., 2024; Huang et al., 2024) (e.g., reporting
observation o
the interquartile mean instead of the average performance). However, benchmarking algorithms with increased number of seeds (Laskin et al., 2019; Gorsane et al., 2022; Bettini et al., 2024) can quickly become a bottleneck (Jordan et al.,
U0
algorithms, like model-based (Wang et al., 2019b) or safe RL (Zhao et al., 2024). In addition, (Jordan et al., 2020) emphasizes that standard evaluations often overlook the difficulty of hyperparameter tuning, which can obscure the true usability of an algorithm in real-world applications.
2024), especially for computationally-ineee
Figure 3. Hybrid -uasttul neural network for an exem- plary 4-qubit quantum layer. The dimensionality of the observation is mapped to the number of qubits in the quantum circuit using a fully-connceted layer. The variational quantum eircuit for details on the ansatz see Figure 18 in Appendix D) acts as a hidden layer, and all qubits are measured individually in the Pauli-Z basis. These measurement results are post pesseproeesed using another fully- connected layer, mapping to the number actions.
QRL can solve artificial tasks (without practical relevance) with exponentially smaller sample complexity compared to any known classical algorithm. Recently, (Jerbi al., 2021; Liu et al., 2021) constructed artificial problems which are widely believed to be classically intractable. When the type of the algorithm is fixed to e.g. policy iteration, quantum versions exist with polynomially reduced sample complexity (Wang et al., 2021; Ganguly et al., 2023; Zhong et al., 2024; Wiedemann et al., 2023). However, these algorithms require resources that far exceed the capabilities of current quantum hardware. An alternative line of research (Chen al., 2020; Skolik et al., 2022; Lockwood & Si, 2020), more aligned with the limitations of current hardware, substitutes the classical neural network used as a function approximator for policy and value function in classical RL algorithms by a variational quantum circuit VQC (Bharti et al., 2022). Here, Jerbi et al. (2021) construct artificial problems based on the same VQC architecture later used by the learner. The superiority found empirically for the QRL algorithm can then be attributed to the inductive bias introduced in the problem. Several studies have compared classical RL and QRL on toy problems inspired by real-worid tasks. Some of these studies (Chen et al., 2020; Drägan et al., 2024; Reers, 2023; Hohenfeld et al., 2024; Eisenmann et al., 2024) have documented empirical superiority of QRL on metrics closely related to sample complexity. This work proposes robust methodology to compare these types of algorithms.
- "r from time step t onwards. With the state- action value function Q= a) = E, [G,]s, = s, a a). the optimal policy is given by x =s(s = argmax, Q a). The optimal Q-value function is the unique solu- tion to the Bellman optimality equation (*(s, Eg p(s' [s, a) [R(s, a, s') + max,' (s'(s'aa)), for all s E S, a E A (Sutton & Barto, 2018).
The state-action value function is typically represented using some type of function approximator, as tabular approaches are only suitable for small problem instances. Reinforce- ment learning based classical deep neural networks (DNNs) has been first successfully realized in the deep Q-networks (DQN) algorithm (Mnih et al., 2015). In our work, we employ an extension of this concept, i.e. double deep Q-networks (DDQN) (Van Hasselt et al., 2016). This algorithm employs function approximators Qo. Qo,, and performs updates of the parameters towards following loss:
(() = [r + y Qo(s', argmax, Qu (s', b)) - Qo(s,a)
where target parameters O' are synchronized after a hyperparaneter--deeeedeett number of update steps.
3. (Quantum) Reinforcement Learning
It is also possible additionally parameterize the policy (i.e. the actor), in addition the value function (i.e. the critic), and train the respective function approximators with proximal policy optimization (PPO) (Schulman et al., 2017). Details on both the algorithm and the hyperparameter tuning performed can be found in Appendix C.
RL is a framework for solving complex lime-dependent decision- maainn problems. It relies on a Markov Decision Process (MDP), represented as 5-tuple AS, R. p. y),, where S is the set of states and A is the set of actions. The reward function R: S A x S + R assigns scalar value to performing action a in state s and transitioning to state s'. The dynamics are governed by p: SXS X + [0, 1], which gives the probability of transitioning from state s to state s' after taking action a. The discount factor 7, ranging between and 1, determines the importance of immediate versus fu- ture rewards. The objective is to find policy (s)) a that maximizes the discounted long-term reward G, t
We compare two function approximation approaches in these algorithms: (i) the standard version with a classical DNN and (ii) the quantum version, which has variational quantum circuit between two small classical fully-connected layers. This structure is called hybrid classical-quantunn quantum, as depicted in Figure 3. As discussed in Appendix D.2, the clas- sical layers have few trainable parameters, and small fully classical DNNs are not competitive, so the hybrid models'
3

---

Benchmarking Quantum Reinforcement Learning
evaluated expected return. In case of our environment, the maximal value of Vi can be calculated, thus without loss of generality, we constrain V2 E [0, 1]. We generalize the definition of the estimator in Appendix B to environments for which the optimal value of the expected return is un- known. We fix E (0, 1] and & E [0, 1]. and define sample complexity as
algorithm 1
evalnated return
VT
V22
algorithm 2
T
S= Σ
I < <8,
(I)
number of algorithmn-cnvirommeent interactions
where II[-] is the indicator function which is 1 exactly if the probability P. = P(V, 2 1 - E) is smaller than 8 and otherwise O. In words. Pt is the probability that the algorithm performs better than the threshold - é at time step t. Since Pe is unknown, we model N training runs by a collection of stochastic processes (V:(%, , - 1,..,, T3 with 1..., N, where V,(1) i.i.d. for given t. Next, we replace Pt by its unbiased estimator
Figure 4. The figure exemplarily shows two learning curves gener- ated by different algorithms or algorithmic settings (algorithm I and algorithm 2). While algorithm 2 exhibits lower sample complexity with respect threshold V/2 than algorithm 1 (for this particular training run), the converse is true for threshold Vi", which algorithm may even never reach. Consequently, if con- vergence to optimality cannot be proved for the algorithm, sample complexity is well defined only with respect to a given threshold.
N 2
P.
Σ T[
21
21
performance originates from the quantum sub-module. The VQC acts like a quantum counterpart to a classical DNN, with trainable weights parameterizing unitary operations, usually single-qubit rotations (Bharti al., 2022). The quantum state is measured to approximate complex func- tions. Past research suggests VQCs may have advantages over DNNs, including better accuracy for certain tasks (Liu et al., 2021) and smaller model size (Chen et al., 2020). More details on VQCs and the ansatz used here are shown in Appendix D.1. Such models have been used in value- based (Chen et al., 2020; Skolik et al., 2022) and policy- based (Jerbi et al., 2021; Meyer et al., 2023a) RL routines.
Now the empirical sample complexity can be defined as: Definition 4.1 (Estimator empirical sample complexity). Given N training runs {V/(8, t = 1,... with i I,,., N and v,(0) i.i.d for given f, probability thresh- old 5 (0, 1] and a performance threshold value e [0, 1].
we call
S =
Σ
11
P 8],
where P. is defined in Equation (2), the empirical sample complexity.
4. Sample Complexity Estimator
Appendix B provides more details on the intuition of this definition. Additionally, we prove consistency, that is lim P(I S > n)
The sample complexity S of an RL algorithm is the number of samples s' ~ p(-|s, a) required to meet a specified per- formance criterion (Kearns & Singh, 1999; Kakade, 2003) with high probability. When S is expressed by relevant problem parameters such state- and action-space size, discount factor etc., it serves as an effective instrument to compare different learning algorithms with performance guarantees (Lazaric et al., 2012; Lattimore et al., 2013; Liu et al., 2024). However, assessing sample complexity of heuristic RL algorithmss requires empirical evaluation pro- cedures. In particular, as illustrated in Figure 4, sample complexity for algorithms without performance guarantees has to be defined with respect to given threshold V*. In the following, we introduce a statistical estimator for sam- ple complexity. To this end, for a given algorithm, we view each training run as the realization of a stochastic process {V,, L,..., T). In our setting, we choose V, to be the
for n > 0. Moreover, using the central-limit theorem, it is shown that S is asymp- totically unbiased. Based on the analysis in the appendix we choose N = 100 throughout this work. To the author's knowledge, this number far exceeds the population size used in any other study on QRL benchmarking. Based on Definition 4.1 we define:
Definition 4.2 (Significant Outperformance). We say that algorithm outperforms algorithm 2 [on task), if (i) it has significantly lower sample complexity (with respect to definition of significance) for some error threshold e and probability threshold 5. (ii) Algorithm must not have a significantly higher sample complexity than Algorithm 2 for any e-S-configuration.
In this work, we consider difference in sample complexity to be significant, if the respective 5th and 95th percentile
4

---

Benchmarking Quantum Reinforcement Learning
ranges do not overlap (i.e., extended interdecile ranges (De- Groot, 2005)). To guarantee robustness, we perform 100 runs for each setup and use cluster re-sampling (Cameron et al., 2008) for estimating the quantiles.
O position
intensity 1e?
intensity I
previous position
5. Experimental Setup
previous trajectory
As QRL algorithms exhibit some robustness against hard- ware noise (Skolik et al., 2023) (which is expected to further decrease in the future (Kim et al., 2023; Acharya et al., 2025)) we decided to perform all experiments in a noise- free simulation. We reduced the impact of parameter initial- ization by considering 100 random seeds per environment and model configuration. We initialized the classical neural networks using He initialization and the VQC parameters uniformly at random within [0, 2.... We tuned the hyperpa- rameters, see Appendix C, before running the actual exper- iments. The configurations of the models are reported in terms of three metrics: (i) model width, i.e., the number of neurons in hidden layers for classical DNNs, and in case of quantum architectures the number of qubits; (ii) model depth, iee. the number of hidden layers for classical net- works, and the number of layers of the quantum models; and (iii) model complexity, parameterized by the number of trainable parameters, which will be justified further below. The reported configurations were chosen by an extensive ablation study, see Appendix D.
A beam B
antenna with beams
antenna A
Figure 5. The BeamManagement 6G enviromment consists of a set of antennas A E Antenna, for which at any point time only one is active. Furthermore, cach antenna is equipped with multi- ple beams, also referred to as codebook element B E Codebook, which selected automatically. A user moves through the en- vironment, is targeted by one of the antennas, and receives some intensity 1 E Intensity. Based on this observation, i.e., the active amtenna A₁-1, beam B,- and intensity 7 at the previous time step t the task to select the optimal antenna for the next timestep f, i.c., the At providing the greatest intensity value / to the user. The objective to maximize the sum of received intensi- ties over the entire trajectory. Note, that the spatial position of the user unknown, as Incalization induces unreasonable real-world overhead, and furthermore collides with user privacy concerns.
BeamManagement 6G Environment. Solutions to 'in-
both antenna and codebook indices are re-scaled, and the intensity value is in [0. 1] by construction in our model. Following this paradigm, we developed a fast simulator that allows ficxible placement of multiple antennas and to sample random movement trajectories of varying complexity. For simplicity we task the agent to select the base station (i.e. the antenna) but assume the optimal codebook element to be found automatically by the antenna, i.c.,
dustrial use cases' with quantum machine learning (QML) (Dunjko & Briegel, 2018) and QRL (Meyer al., 2022) currently limited toy problems, far from generating commercial value. This limitation is due to constraints of current hardware (i.e.. the number and quality of qubits) and the input-output bottleneck (Hoefler et al., 2023).
We propose a novel benchmarking environment that fo- cuses on beam management in wireless communication. Next-generation communication networks feature anten- nas capable of forming directional beams to serve mobile phones (Enescu, 2020). Beam management is the task of se- lecting the antenna and beam direction that maximize beam quality (its intensity) at the position of a moving phone. The (discretized) antennas and beam directions are precoded into a codebook.
A= = Antenna.
Selecting a (close to) optimal beam can be solved via effi- cient beam search algorithms, cf. (Yammine al., 2023). The reward is the intensity received after selecting the re- spective antenna and codebook element. A sketch of this environment is given in Figure 5, details are deferred to Appendix A and Appendix E.
Promising solutions to beam management are based on RL (Wang al., 2019a; Yammine et al., 2023). Without explicit knowledge of the specific trajectory of the mobile phone, the RL agent is trained to select optimal antenna index and codebook element, only given the selections of previous time steps. This describes the RL state space as
We specifically employed this environment because it has a small state and action space, yet exhibits complex dynam- ics, enabling meaningful analysis without overwhelming encoding complexity. Moreover, its state space mostly continuous, reflecting realistic energy variations and beam selections as continuous angles. Finally, there is ongoing debate on suitable methods for beam management, with RL identified as a strong contender (Maggi et al., 2024; Voigt et al., 2025), making this environment an excellent choice for benchmarking classical RI. and QRL in set-
S = Antenna X Codebook X Intensity,
where Antenna is the set of antenna indices, and Codebook the set of codebook elements. As input to the model,
5

---

Benchmarking Quantum Reinforcement Learning
Model (width-depth):
Quantum (10-4)
Classical (16-2)
Quantum (14-4)
Classical (32-2)
Classical (64-2)
0.20
c 0.15
0.10
200k
0.50
150k
100k
sample complexity
50k
200k
P
0.75
150k
100k
50k
313
387
437
1283
4611
313 model trainable parameters
387
437
1283
4611
313
387
437
1283
4611
Figure Empirical sample complexities $ of double deep Q-learning for various relative errors and threshold prohabilities on BeamManagement 6G environment. The model width denotes the number of neurons in classical hidden layers, and the number of qubits quantum models, respectively, Moreover, model depth refers to the number of hidden layers in DNNs, and number of ansatz repetitions in VQCs. The estimate based on 100 training runs, with validation granularity of 2000 steps. The error bars denote the 5th and 95th percentiles, estimated with cluster resampling. The results follow the performance pattern already highlighted in Figure 1. Both quantum models on par with the 3- to 4-fold larger medium-sized classical model, with the 14-qubit model having slight edge. Configurations where multiple runs do not achieve the targeted relative error rate before the cut-off 200k steps are shown transparent.
ting inspired by practical 6G challenges. Nonetheless, we highlight that this environment is merely one example to empirically demonstrate our methodology, which can also be applied to other, entirely different tasks.
evaluating the sample complexity depending on the relative error threshold E. In contrast, the agent itself has access only to the intensity values, in accordance with the real-world conditions. For practical reasons, we evaluate the sample complexities on only the subset of e-ff-coffiuuration, which are meaningful for real-world performance.
6. Experiments
In Figure 1, plot the empirical sample complexity for three instances of the DDQN algorithm on the standard con- figuration of three antennas of the BeamManagement 6G environment. Overall, the large classical model with 461 trainable parameters (width 64, depth 2) exhibits the lowest, i.e., best, sample complexity. The small quantum model with only 336 classical and quantum parameters (14 qubits, layers) nearly matches this performance. In contrast to that, the approximately equally-sized small classical model with 387 parameters (width 16, depth 2) clearly requires much more samples for convergence. This hierarchy is preserved across a wide range threshold probabilities and error thresholds e. Note, that sample complexity of 200, 000 indicates, that some runs failed converge the desired emror threshold.
We evaluate different (QRL algorithms BeamManagement 6G environment using our proposed statistical sample complexity estimator Section 6. Sec- tion 6.2 studies the relationship between sample complexity and (RL) model complexity. Section 6.3 we compare the performance of the trained policies.
novel
6.1. Sample Complexity
We extract the sample complexity of a model via post- processing of the validation logs for 100 randomly seeded runs. A training epoch collects trajectories from 10 environ- ments, each with horizon of 200 steps. Training is con- ducted for 100 epochs in total, i.e., overall 200, 000 agent- environment interactions are conducted in each run. After each epoch, validation is performed on 100 environment in- stances, reporting the ratio received beam intensity vs. the optimal intensity. This validation with relative intensities is done enhance the stability of the estimate, and allow for
We extend this analysis by conducting cluster resam- pling (Cameron et al., 2008), variant of bootstrap resam- pling which captures the correlations time the learning process. We employ this method to estimate the 5th and
6

---

Benchmarking Quantum Reinforcement Learning
95th percentiles of the estimator S for sample complexity. The percentile ranges are indicated by error bars throughout this work, and used to determine outperformance according to Definition 4.2.
c
0.15
200k
Quantum (4 layers)
T
251
0
150k
189
313
370
100k
50k
sample complexity
In Figure we summarize our main results, and addition- ally consider two more models: An even smaller quantum model with 313 parameters (10 qubits, 4 layers). It ex- hibits a performance close to, but not quite competitive with the 14-qubit hybrid model. The restriction to 10 qubits in most experiments optimally utilizes limited computational resources, see Appendix D.2. Overall, we conclude, that hybrid classical-quantuue models significantly outperform similar-sized fully classical approaches w.r.t. sample com- plexity. Furthermore, we include medium-sized classical model with 1, 283 parameters (width 32, depth 2), which performs slightly worse than the large quantum model for most, but not all e-6-configurations. Moreover, the large quantum model is not significantly outperformed by the largest DNN, demonstrating signs of competitiveness.
437
12
14
qubits
5 0.15
Classical (depth 2)
200k
0.75
387
150k
1283
100k
4611
17411
67587
50k
16
32
64 width
128
256
Figure 7. Correlation of model and sample complexity in the ReamManagement.66 6G environment with the DDQN algorithm. The upper plot depicts quantum models with increasing number of qubits in the variational quantum circuit, the lower plot shows classical DNNs with increasing width of the hidden layers. The number next to the markers denotes the number of trainable pa- rameters in the respective model. All results are averaged over 100 seeds, and error bars denote the 5th and 95th percentiles, estimated with cluster resampling. A more extensive version of this plot can be found in the appendix in Figure 19.
A similar analysis of the PPO algorithm, where both policy and value function are approximated with DNNs and VQCs, respectively, supports our findings. Details can be found in Appendix C.2. While the results qualitatively match pre- vious observations, typical sample complexities are much higher across all models. However, this is not surprising, as on-policy approaches like PPO are known to be less sample-efficient than off-po routines like DDQN. This superiority stems from structures like, e.g., the experience replay buffer, which allows the latter algorithm to re-use previous experience (Sutton & Barto, 2018).
be found in Appendix D.2.
Most crucially, the sample complexity of the classical mod- els saturates once the hidden layer width reaches 64. In contrast, increasing the qubit number of the quantum model up to 14 does not exhibit similar saturation behavior.
6.2. Scaling with Model Complexity
We now explore the relationship between sample efficiency and the complexity of the RI. and QRL model, respectively. There exist different measures for model complexity both in the classical (Hu et al., 2021) and quantum (Abbas et al., 2021) domain. In this work we define model complexity based on the number of trainable parameters. The parameter count defines a sequence of models in model space (assum- ing additional hyperparamter search for given parameter count). This sequence can then be used to identify potential trends, for example in the behavior of sample complexity as we scale to more powerful quantum models.
At this point, two questions arise which addressed in the following: (i) Are models with lower complexity but similar performance generally preferrable, i.e., how should sample and model complexity be balanced? (ii) Is outperfomance of the quantum model achievable by further increasing qubit numbers? The first question (i) cannot be answered in gen- eral as it depends on concrete practical considerations and objectives. If the primary goal is to minimize the number of revermmenvirommet interactions, regardless of training and inference costs, the classical models are superior. However, when memory requirements are considered, the reduced parameter count of the -ennuue-enhhnced model might be advantageous. Assuming scalability of this approach, also runtime improvements are conceivable. The second question (ii) cannot be answered conclusively at the current stage. The saturation behavior in Figure 7 suggests that further scaling the quantum model could lead to compara- ble or superior performance relative to the largest classical model, which, however, can only be substantiated by more experimental evaluations. We stress that a naive extrapola-
In the previous section, we experimentally demonstrated that a hybrid classical-quantum model significantly outperforms purely classical models of similar complexity and competes with much larger ones. In Figure 7, we analyze this behavior more systematically by plotting sample complexity against both classical and quantum model complexities. Specifi- cally, we vary the width of hidden layers for the classical DNNs, and the number of qubits for VQCs. A deeper analy- sis including investigations into different model depth can
7

---

Benchmarking Quantum Reinforcement Learning
Quantum Model (14-4): 13.29 1 0.16
similar-looking histograms of the received intensities. The reported mean values of the intensity of the quantum model (green) and the classical model (blue) do not indicate signif- icantly different behavior by visual inspection. Furthermore, we observed comparable results for various different en- vironment configurations in Appendix E. Therefore, we conclude that the performance of the QRL and classical RL models on par for the BeamManagement 6G environ- ment.
1000
500
0
Classical Model (64-2): 13:24 1 0.19
1000
500
count (oul. of 10000)
These closely match the ground-truth intensity distribution (yellow) obtained by brute force. Moreover, all approaches clearly improve upon a random strategy. Note that the poli- cies of the trained models correspond to non-trivial behavior patterns. We stress that simply selecting the closest antenna is sub-optimal, as highlighted by the complex spatial inten- sity patterns shown in Figure 13 in the appendix. Moreover, as highlighted in Figure 5, the position of antenna and the mobile phone are unknown to the agent and only learned implicitly.
Optimal Actions: 13.99
1000
500
0
Random Actions: 5.09
2000
1000
6.4. Standard CartPole Benchmark
5
10
15
20
25
To further test our benchmarking scheme, we extend our analysis to the widely used CartPole-vl environment. This benchmark regularly appears QRL studies (Lockwood Si, 2020; Skolik al., 2022), often accompanied by claims that quantum models outperform classical approaches with fewer number of trainable parameters. However, these state- ments rely small number of training runs and are based on simple visual comparisons. In contrast, here we apply our statistical sample complexity estimator to CartPole-ve with the results shown in Figure 9.
intensity distribution (+0.5)
Figure 8. Cumulative intensities over trajectories achieved in the BeamManagement environment. For both, trained hybrid quantum and classical model, we select the 10 best- performing instances out the trained models and evaluate each on 1000 random trajectories. The quantur model (green) and the classical model (blue) closely match the ground-truth performance (yellow), with a large performance gap random behavior (brown). Next to the subplot identifiers we report the mean value observed intensities over the 10 model instances.
We employed vanilla policy gradient algorithm and per- formed hyperparameter search over similar range as re- ported in Appendix C.I. While generally to find globally optimal settings (Jordan et al., 2020), more advanced training methods such as natural gradients (Meyer et al., 2023b) could potentially further improve performance. Nevertheless, the chosen configurations allows us to com- pare classical and quantum function approximators with the number of trainable parameters ranging from about 30 to 17, 000. The best quantum setup found, single-layer model only four qubits (Meyer et al., 2023a), performs competitively to the best classical model across nearly all e-Ō configurations. The cross-cut, Figure 9(b), of Figure 9()) at an evaluated threshold of ε = 0.05 shows that the perfor- mance gap may not always be statistically significant due to overlapping confidence bounds. This again underscores the importance of sufficient number of runs.
do not expect
tion to higher qubit numbers might be limited in practice by trainability issues (Larocca et al., 2024). Morcover, it is computationally prohibitive to generate simulation results beyond 14 qubits due the substantial overhead required for statistically robust results summarized in Table 3 in Appendix D.2.
6.3. Performance of Trained Policies
In the following, we validate that the RL models trained in Section 6.1 behave in meaningful way. Therefore, iden- tified the 10 best- rest-performing instances with width 64 and 14 qubits, respectively. In Figure 8, we report the histogram of intensities for the standard environment configuration (see Figure 11) received by employing the respective policies on 1000 random trajectories each.
We stress that these results should not be interpreted as ev- idence for quantum advantage. The circuit sizes are very small, allowing straightforward classical simulation, and scaling CartPole- vv1 in a way that requires larger quan-
As consequence of different trajectories, the maximum achievable intensity varies. Both RL strategies produce
8

---

Benchmarking Quantum Reinforcement Learning
our findings show signatures of heuristic quantum advan- lage? Not quite, because despite extensive hyperparameter optimization, we cannot guarantee that we compared the quantum model to the optimal classical one.
Model
Classical
Quantum
100k
sample complexity
75k
If we set aside this caveat for the moment as double Q- learning is known to be comparatively sample-efficient (Sut- ton Barto, 2018), the results are more nuanced. If we steadily increase the number of parameters of the classical model, numerical results demonstrate that its perfor- mance approaches that of the quantum model, but plateaus before significantly outperforming it. Thus, did we verify quantum advantage under the condition of comparable pa- rameter numbers between models? Here, we caution again because a necessary condition for any kind of quantum ad- vantage classical intractability.
50k
25k
0.08
0.4 threshold probability
0.8
relative error
0.05
0.6
0.02
(a) Surface plot of sample complexity over various configura- tions. The classical model contains one hidden layer of width 16, the quantum model operates on 4 qubits.
Evidence for quantum advantage obtained in small-scale experiments must therefore necessarily be accompanied by arguments for its persistence as problem sizes increase. For example, from our studies on model complexity we might extrapolate that the performance of the quantum model could be further improved by increasing the number of qubits. In this sense, small-scale experiments may indicate trends that might might not persist for large problem instances. We therefore encourage more experiments and simulations with the striving to scale, which might offer valuable insights into the behavior of quantum algorithms. Ulimately, the question empirical quantum advantage is, at its core, an empirical one and an answer will come from experimental progress of the future.
Cross-cut at
0.05
100k
Quantum (4-1)
sample complexity
Clussical (16-1)
75k
Classical (64-1)
50k
25k
0.4
0.6
0.8
threshold probability 6
8. Conclusion
(b) A cross-cut of sample complexity at a relative error of & = 0.05. The error bars denote the 5th and 95th percentiles.
In summary, we introduced robust and statistically sound methodology for benchmarking heuristic RL, in particular quantum RL algorithms. Our procedure relies on an empir- ical notion of sample complexity captured by statistical estimator and a rigorous definition of outperformance. We developed benchmarking suite inspired by real-world wire- less 6G communication tasks, which is flexibly adjustable in difficulty and instance size.
Figure 9. Comparing empirical sample complexities S of vanilla policy gradients and quantum version of the algorithm on the CartFole-vi environment. Both classical and quantum models were optimized over wide range of hy perparaameterss and model sizes. The estimates are based on 100 training runs.
tum models is notoriously difficult. These limitations moti- vate our focus on Be aaaaaaaaemennen 6G, whose complex- ity can flexibly scaled.
As an application of our methodology, we compared the performance of double deep Q learning, as well as proximal policy optimization, and their quantum counterparts. In an extensive and statistically robust computational analy- sis, covering many structurally different problem instances and models, we found that the quantum algorithm consis- tently outperforms the classical version when the number of trainable parameters is similar.
7. Discussion of Quantum Advantage
In this section, we discuss the relation of our benchmarking results of the previous section to potential quantum advan- tage. Let us define heuristic quantum advantage as the consistent and statistically significant (Definition 4.1) out- performance (Definition 4.2) of a quantum algorithm over the best (known) classical algorithm for given task. Do
We evaluated the results with respect to potential empirical quantum advantage. We argued that currently no definitive statement can be made but identified trends, that may be corroborated by further experiments on larger scale.

---

Benchmarking Quantum Reinforcement Learning
Acharya, R., Abanin, D. A., Aghababaie-Beni, L.,, Aleiner, thank D. Fehrle for helpful discussions. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAL of the Frietrich-Alexande--nieee
Acknowledgements
I., Andersen, T. I., Ansmann, M., Arute, F., Arya, K., Asfaw, A., Astrakhantsev, N., et al. Quantum error cor- rection below the surface code threshold. Nature, 638: 920-926, 2025. doi: 10.
Erlangen-Nümbberg (FAU), The hardware is funded by the German Research Foundation (DFG). Composing and producing the results presented in this paper required about 40000 core hours of compute.
Agarwal, R., Schwarzer, M., Castro, S., Courville, A. C., and Bellemare. M. Deep reinforce- ment learning at the edge of the statistical precipice. Adv. Neural Inf. Process. Syst. 34: 29304, 2021. URL https://poocecing. neuuipps.o//ppprr//22///.ee E514666f.00448555441177722eeeee2 pdf.
This research was conducted within the Bench-QC project, lighthouse project of the Munich Quantum Valley initiative, which is supported by the Bavarian state with funds from the Hightech Agenda Bayern Plus.
Funding:
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., al. What matters for on-policy deep actor-critic methods? a large-scale study. In Interna- tional Conference on Learning Representations (ICLR), 2021. URL https: /toperppeeeiieeeeeeeeeeeeeeeeeee net forum?
Impact Statement
This paper aims to advance the field of Quantum Reinforce- ment Learning by introducing statistical benchmarking routine for rigorous evaluation. Our work addresses the need for substantiated claims quantum advantage, promoting a standard of thorough assessment. We anticipate that this contribution will enhance the quality and reliability of future research in Quantum Reinforcement Learning. supporting the development of robust and verifiable advancements in the field.
i id=nIAxjsninl
Arulkumaran, K., Deisenroth, M. Brundage, M., and
Bharath, A. A. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34:26, 2017. doi: 10.11099MSP220172274222.
Bergholm, Izaac, J., Schuld, M., Gogolin, C., Ahmed, S., Ajith, V., Alam, M. S., Alonso-Linaje, G., Akash- Narayanan, B., Asadi, A., et al. Pennylane: Automatic differentiation of hybrid quantum-classies computations. arXiv:/811.04968 2022. doi: 10.48550/arXiv.18h 04968.
Data Availability
Implementations of environments, algorithms, and evalua- tion routines described in this paper are available in the repository https:/ //gttttt rgithhb... com n comeyer96/ grl-benchmark. The framework allows for full repro- ducibility of the experimental results in this paper by execut- ing a single script. Usage instructions and additional details can be found the README file. Further information and data is available upon reasonable request.
Bettini, M., Prorok, A., and Moens, V. Benchmarl: Benchmarking multi-agent reinforcement learn- ing. J. Mach. Learn. Res., 25:1, 2024. URL attps://wwe jmlr. amlr.o.. womlr.org/poe wwt.ooogppppp...... httgg/apers//eeee ppapers/volumede volume25/ 23 2331612/23-1 0/////23-612..11 1612.pdf.
Bharti, K., Cervera-Lierta, A., Kyaw, T. H., Haug, T., Alperin-Lea, S., Anand, A., Degroote, M., Heimonen, H., Kottmann, J. S., Menke, T., Mok, W.-K., Sim, S., Kwek, L.-C., and Aspuru-Guzik, A. Noisy intermediate- scale quantum algorithms. Rev. Mod. Phys., 94:015004, 2022. doi: 10. 101110/eeMoMPPy....... .0..pP/hy.99..000000
Statement of Independent Work
We acknowledge the existence of research paper with the same title as ours (Kruse et al., 2025). We wish to clarify that both works were conducted independently and without knowledge of each other.
Bowles, J., Ahmed, S., and Schuld, M. Better than clas- sical? The subtle art of benchmarking quantum ma- chine learning models. arXiv:2403.07059 2024. doi: 10.488550/2Xx..24433770000
References
Abbas, A., Sutter, D., Zoufal, C., Lucchi, A., Figalli, A., and Woerner, The power quantum neural net- works. Nat. Comput. Sci., 1:403, 2021. doi: 10.1038/
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAl Gym. arXiv:1606.00 000.000066000,,, 2016. doi: 10.48550/arXiv.1666 01540.
10

---

Benchmarking Quantum Reinforcement Learning
Burkhardt, F., Jaeckel, S., Eberlein, E., and Prieto-Cerdeira, R. QuaDRiGa: A MIMO channel model for land mobile satellite. In The 8th European Conference Antennas and Propagation (EuCAP 2014), pp. 1274, 2014. doi: 10.1 AA110/FFCAAP2200444900000
Dunjko, V., Taylor, J. M., and Briegel, H. J. Quantum- enhanced machine learning. Phys. Rev. Lett., 117:130501,
2016. doi: 10.1 1..1
Eisenmann, S., Hein, D., Udluft, S., and Runkler, T. A. Model-based offline quantum reinforcement learning. In IEEE International Conference on Quantum Computing and Engineering (QCE), volume pp. 1490-1496, 2024. doi: 10. 01099//0CEE608222222.00111111
Cameron, A. C., Gelbach, J. B., and Miller, D. L. Bootstrap- based improvements for inference with clustered errors. Rev. Econ. Stat., 90:414, 2008. doi: 10. 10.11622 10.1162/rest.90. 414.
Enescu, M. (ed.). 5G New Radio: A Beam-based Air In- lerface. John Wiley & Sons, Inc., 2020. doi: 10.1002/ 9781119582335.
Chen, S. Y.-C., Yang, C.-H. H., Qi, Chen, P.-Y., Ma, X., and Goan, H.-S. Variational quantum circuits for deep reinforcement learning. IEEE access, 8:141007, 2020.
doi: 0.1 109/ACCSSSSSS 2020.00440.
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, Implementation matters in deep RL: A case study PPO and TRPO. In Interna- tional Conference on Learning Representations (ICLR), 2019. URL https::/t//.......... /wwpprpeeeeeeeeeeeeeeeeeeeeeeeee net. forum? id=rletNlrtPB.
Cherrat, E. A., Kerenidis, I., and Prakash, A. Quantum rein- forcement learning via policy iteration. Quantum Mach. Intell., 5:30, 2023. doi: 10.1007/s42484-022 023-0002 16-1.
Colas, C.. Sigaud, O., and Oudeyer, P.-Y. How many random seeds? Statistical power analysis in deep reinforcement learning experiments. arXiv: 180600898 2018. doi:
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., and Pineau, J. An introduction to deep reinforce- ment learning. Found. Trends Mach. Learn., 11:219, 2018. doi: 10. 10.1561/2200000711.
10.48550/arXiv. 180.ii888000002255.
Corici, M.-I, Franke, N., Heyn, T., Kontes, G., Leyh, M., Magedanz, T., Maaß, U., Mikulla, M., Niemann, B., Peter, M., Roth ManhuMandutz, E., Schubert, C., and A fraunhofer 6g white paper: On the road to 6G: Drivers, challenges and enabling technologies. Technical report, Fraunhofer IIS, 2021. URL https: http::://wwwwtte htrt::://ww..ss...uubbeeee. fraunhofer. de/en/ ff http://mmmmmoleee/ooo/osseeeeeer 6g- sent inee- www.tw wwt.--ttp-peepeptttt.. html.
Franz, M., Wolf, L., Periyasamy, M., Ufrecht, C., Scherer, D. D., Plinge, A., Mutschler, C., and Mauerer, W. Un- covering instabilities in variationallquantumm deep Q- networks. J. Frankl. Inst., 360:1382) 2023. doi: jf.aain 200.2.in.222008.0....
Yammine,
Fujimoto, S., Conti, E., Ghavamzadeh, M., and Pineau, J. Benchmarking batch deep reinforcement learning algo- rithms. arXiv:1910.0110 2019. doi: 10.48550/arXiv. 1910.01708.
DeGroot, M. H. Optimal statistical decisions. John Wiley & Sons, NJ, 2005.
Drägan, T.-A., Tandon, A., Strobel, C., Krauser, J. S., and Lorenz, M. Quantum multi -aget reinforcement learn- ing for aerial ad-hoc networks. arXiv:2404.17499 2024. doi: 10.48550/axXiv.2444..
Ganguly, B., Wu, Y., Wang, D., and Aggarwal, V. Quantum computing provides exponential regret improvement in episodic reinforcement learning. arXiv:2302.08617, 2023. doi: 104485550arXXi.2222288666
Du, Y., Huang, T., You, S., Hsieh, M.-H., and Tao, D. Quantum circuit architecture search for variational quan- tum algorithms. upj Quantum Inf., 8:62, 2022. doi: 141108//41334-0220000077
Gershman, B., Sidiropoulos, N. D., Shahbazpanahi, S., Bengtsson, M., and Ottersten, B. Convex optimization- based beamforming. IEEE Signal Processing Magazine, 27:62, 2010. doi: 10. 10.100//MSP.200.936615.
Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning (ICML., volume 48, pp. 1329. PMLR, 2016. URL https: /ttpp//rcceenggg.eeee////// mlr htt.psesssm...p..////m.mm 148/duanne 6. pdf..
Gorsane, R., Mahjoub, O., de Kock, R. J., Dubb, R., Singh, S., and Pretorius, A. Towards standardised performance evaluation protocol for cooperative MARL. Adv. Neu- ral Inf. Process. Syst., 35:5510, 2022. URL https: / poopenevveewnneee. Forrmmiidaaamn 6qcwErJm.
Griffiths, D. J. Introduction Electrodynamics. Pearson, fourth edition, 2013.
Dunjko, V. and Briegel, H. J. Machine learning &; artificial intelligence in the quantum domain: a review of recent progress. Rep. Prog. Phys., 81:074001, 2018. doi: 10. 1088/13611-6333/abbbd0
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that

---

Benchmarking Quantum Reinforcement Learning
matters. In Proceedings of the Thirly-Second AAAI Con- ference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Press, 2018.
Kakade, S. M. On the Sample Complexity of Reinforcement
Learning. Phd thesis, University College London, 2003.
Kandala, A., Mezzacapo, A., Temme, K., Takita, M., Brink, M., Chow, J. M., and Gambetta, J. M. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. Nature, 549:242, 2017. doi: 10.1038/ nature23879.
Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., and Wu, Y. Stable Baselines. https: / / thub. com/
Kearns, M. and Singh, S. Finite-sample convergence rates for Q-learning and indirect algorithms. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II, pp. 996, Cambridge, MA, USA, 1999. MIT Press.
hill Mall-a/se a/ htwttable-baaseteeeeeeeeeeeeeeee -ee--baaeeienn..e 2018.
Hoefler, T., Häner, T., and Troyer, M. Disentangling hype from practicality: On realistically achieving quantum advantage. Commun. ACM, 66:82, 2023. doi: 3571725.
Kim, Y., Eddins, A., Anand, S., Wei, K. X., Van Den Berg, E., Rosenblatt, S., Nayfeh, H.,, Wu, Y., Zaletel, M.. Temme, K., et al. Evidence for the utility of quantum computing before fault tolerance. Nature, 618:500, 2023. doi: 10.1038/s4 1586-023-06066-3.
Hohenfeld, H., Heimann, D., Wiebe, F., and Kirchner, F. Quantum deep reinforcement learning for robot nay- igation tasks. IEEE Access, 12:87217, 2024. doi:
10.1
Kruse, G., Coelho, R., Rosskopf, A., Wille, R., and Lorenz, J. M. Benchmarking quantum reinforcement learning. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence Volume 1: QAIO, pp. 773-782, 2025. doi:
Hu, X., Chu, L., Pei, J., Liu, W., and Bian, J. Model complexity of deep learning: A survey. Knowl. Inf. Syst., 63:2585, 2021. doi: 10.1007/100115-21--111
Huang. S.. Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023. 2022. URL https: // //iclr-bloo-trrack......
Larocca, M., Thanasilp. S., Wang. S., Sharma, K., Biamonte, J., Coles, J., Cincio, I.., McClean, R., Holmes, Z., and Cerezo, M. review of barren plateaus in variational quantum computing. arXiv:2405.00781, 2024, doi: 10. 48550aaXii.2205.00777
03/ hog.
Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang. C., Pinto, L., and Abbeel, P. URLB: Unsupervised reinforcement learning benchmark. In Deep RL Workshop NeurIPS 2021, 2019. URL https: http:///ppereiie.e// net/forume lieedidd=wwle www.dddwwwwppsi......ppss
Huang, S., Gallouédec, Q. Felten, F., Raffin, A., Dossa, R. F. J., Zhao, Y., Sullivan, R., Makoviychuk, V. Makovi- ichuk, D., Danesh, M. H., et al. Open RL benchmark: Comprehensive tracked experiments for reinforcement learning. arXiv:2402.03046 2024. doi: 10.48550/arxiv. 2402.03046.
Lattimore, T. Hutter, M., and Sunehag, sample-complexity of general reinforcement learn ing. In International Conference ing (ICML), volume 28, pp. 28. PMLR, 2013. URL https:/ ptttps://proceediie /ppppeeeennnnsseeeeeeeeeeeeeeeee wwl.s.pres//vee latt inttimmoel.. 3. html.
The
Jerbi, S., Gyurik, C., Marshall, S., Briegel, H., and Dun- jko, V. Parametrized quantum policies for reinforcement learning. Adv. Neural Inf. Process. Syst., 34:28362, 2021.
Machine Learn-
Jordan, S., Chandak, Y., Cohen, D., Zhang, M., and Thomas, P. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning (ICML), volume 119, pp. 4962-4973. PMLR, 2020. URL https: http///eeeeigg...c mal.prress/ v119 (119joorda220a. html.
Lazaric, A., Ghavamzadeh, M., and Munos, R. Finite- sample analysis of least-squares policy iteration. J. Mach. Learn. Res., 13:3041, 2012.
Liu, Q., Weisz, G., György, A., Jin, C., and Szepesvári, C. Optimistic natural policy gradient: A simple efficient pol- icy optimization framework for online RL. Adv. Neural Inf. Process. Syst. 36, 2024.
Jordan, S. M., White, A., Da Silva, B. C., White, M., and Thomas, P. S. Position: Benchmarking is limited in reinforcement learning research. In International Conference on Machine Learning (ICML), volume 235, 2024. URL https: pttpp//ecceegg.s.e/eep//pc/...se mlr press/ v2 v 4a html.
Liu, Y., Arunachalam, S., and Temme, K. A rigorous and robust quantum speed-up in supervised machine learning. Nat. Phys., 17:1013, 2021. doi: 10.1038/ 141567-021-01287-z.
12

---

Benchmarking Quantum Reinforcement Learning
Lockwood, O. and Si, M. Reinforcement learning with quantum variational circuit. Proceedings of the AAAI Conference Artificial Intelligence and Interactive Dig- ital Entertainment, 16:245, 2020.
https::/www.jjml org/ wwtp:/pppps///oooeeeeeee 23-0183/23-0183.
Pérez-Salinas, A., Cervera-Lierta, A., Gil-Fuster, E., and Latorre, J. I. Data re-uploading for universal quan- tum classifier. Quantum, 4:226, 2020. doi: 10.22331/ q- 0-2020-02-06-2226.
Maggi, L., Andrews, M., and Koblitz, To RL not to RL.? an algorithmic cheat-sheet for Al-based radio resource management. arXiv:2405.19045, 2024. doi: 10448550/anXiv22455599000
Reers, V. Towards performance benchmarking for quantum reinforcement learning. In INFORMATIK 2023 Design- ing Futures: Zukünfite gestalten, pp. 1135. Gesellschaft für Informatik eV, 2023.
Meyer, N., Ufrecht, C., Periyasamy, M., Scherer, D. D., Plinge, A., and Mutschler, C. A survey on quantum reinforcement learning. arXiv:2211.03464 2022. doi: 10.48550/arXiv.221 22..04004 .03464.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization algorithms. arXiv:1707.06347, 2017. doi: 10.48550/arXiv.100) 06347.
Meyer, N., Scherer, D., Plinge, A., Mutschler, C., and Hart- mann, M. Quantum policy gradient algorithm with op- timized action decoding. In International Conference on Machine Learning (ICML), volume 202, pp. 24592. PMLR, 2023a. URL https: / /proceddings.m. mllr. html.
Shepherd, D. and Bremner, M. J. Temporally unstructured quantum computation. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 465:1413 2009, doi: (0.1098/rspa.2008..4 04443.
Meyer, N., Scherer, D. D., Plinge, A., Mutschler, C., and Hartmann, M. J. Quantum natural policy gradients: To- wards sample-efticient reinforcement learning. In 2023 IEEE International Conference on Quantum Computing and Engineering (QCE), volume 2, pp. 36. IEEE, 2023b. doi: 10. N0.1109/QCE57702.20000 2077702.202.11111
Sim, S., Johnson, P. D., and Aspuru-Guzik, A. Expressibil- ity and entangling capability parameterized quantum circuits for hybrid caaantu--classeee algorithms. Adv. Quantum Technol., 2: 2:1900070, 2019, doi: 10.1002/qute. 201900070.
Skolik, A., Jerbi, S., and Dunjko, V. Quantum agents in the gym: A variational quantum algorithm for deep Q-learning. Quantum, 6:720, 2022. doi: 10.22331/ q- 2-2022-05-24-720
Meyer, N., Ufrecht, C., Periyasamy, M., Plinge, A., Mutschler, C., Scherer, D. D., and Maier, A. Qiskit-torch- module: Fast prototyping of quantum neural networks. In IEEE International Conference on Quantum Computing and Engineering (QCE), volume 1, pp. 817-823, Mon- treal, QC, Canada, 2024. doi: 10. 0..100/00E6022522222 00101.
Skolik, A., Mangini, S., Bäck, T., Macchiavello, C., and Dunjko, V. Robustness of quantum reinforcement learn- ing under hardware errors. EPJ Quantum Technol,, 10:1, 2023. doi: 10.1 14144000//444555000000000666
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., al. Human-level control through deep reinforcement learning. Nature, 518:529, 2015. doi: 1038/nature
Sutton, R. and Barto, A. Reinforcement Learning: An Introduction. Cambridge, MA, MIT Press, 2018.
Tassa, Y., Doron, Y., Muldal, A., Frez, T., Li, Y., Casas, D.
d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., et al. Deepmind control suite. meXivv/80/. 000,006 2018a. doi: 10.48550/arXiv. 1801.80690
Nielsen, M. A. and Chuang, I. Quantum Computa- fion and Quantum Information: 10th Anniversary Edi- tion. Cambridge University Press, 2010. doi: 10.1017/ CBO97805 11976667.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A.. Merel, J., Lefrancq. A., Lillicrap. T., and Riedmiller, M. Deep- mind control suite. arXiv:1801.00690 2018b. doi: a0.48550/arxiv. wwt...ivvv....066.11 1801.00690.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Ady. Neural Inf. Process. Syst., 32, 2019.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026. IEEE, 2012. doi: 10.1109/IROS Www.HRROO...02.68811...
Patterson, A., Neumann, S., White, M., and White, A. Empirical design in reinforcement learning. J. Mach. Learn. Res,, 25(318): 2024. URL
13

---

Benchmarking Quantum Reinforcement Learning
Trees, H. L. V. Optimum Array Processing. John Wiley & Sons, Inc., 2002. doi: 10.1002/04712221104
Zhong, H., Hu, J., Xue, Y., Li, T., and Wang, L. Provably efficient exploration in quantum reinforcement learning with logarithmic worst-case regret. In International Con- ference on Machine Learning (ICML), volume 235, Pp. 61681. PMLR, 2024. URL https: /esttts:/pppreee proceedings. mir. ALE.press/v2333 zhong2 4b. html.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double Q-learnin. In Proceedings of the AAAI conference on artificial intelligence, volume 30, pp. 2094, 2016. doi: 10. 10.1609aaaai.ooo watiaaaiiv0i1.1022533
Voigt, J., Gu, P. J., and Rost, P. M. deep reinforcement learning-band based approach for adaptive handover protocols. In 14th International ITG Conference on Systems, Com- munications and Coding (SCC), pp. 1-6, 2025. doi:
10.1 LL00/IEEEEEO05222000000.00000000
Wang, D., Sundaram, A., Kothari, R., Kapoor, A., and Roet- teler, M. Quantum algorithms for reinforcement learning with generative model. In International Conference Machine Learning (ICML), volume 139, pp. 10916. PMLR, 2021. URL https:ttppt.... : proccedd ngs mlr. press/v139 /wt////wa/lww//am// html.
Wang, R., Onireti, O., Zhang, L., Imran, M. A., Ren, G.,
Qiu, J., and Tian, T. Reinforcement learning method for beam management in millimeter-wave networks. In 2019 UK/ China Emerging Technologies (UCET), 1, 2019a.
doi: 10.1109/UCET.2019. 888118411
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Lan- glois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J. Benchmarking model-bases reinforcement learning. 1007 2019b. doi: 10.485500/rxiv.19900 02057.
Weng, J., Chen, H., Yan, D., You, K., Dubureq, A., Zhang, M.. Su, Y., Su, H., and Zhu. J. Tianshou: A highly modularized deep reinforcement learning library. J. Mach. Learn. Res., 23:1, 2022. URL https:://///.... /www. jmlr. brg/papers/ae Setaperssv23/20 vt////232//2/ 1127. 7 html.
Wiedemann, S., Hein, D., Udluft, S., and Mendl, C. B. Quan- tum policy iteration via amplitude estimation and grover search - towards quantum advantage for reinforcement learning. Trans. Mach. Learn. Res., 2023. ISSN 2835- 8856. URL https: hwtpttt/ttttetteeee http:///ppeeeeeeeee/eeeeeee /ttt/ id=HG1 PAmWQ6.
Yammine, G., Kontes, G., Franke, N., Plinge, A., and Mutschler, C. Efficient beam search for initial access using collaborative filtering. 2023 IEEE Wireless Com- munications and Networking Conference (WCNC), pp. 2023. doi: 10.1 20B9WWCCC5533.22233.000
Zhao, W., Sun, Y., Li. F., Chen, R., Liu, R., Wei, T., and Liu, C. GUARD: safe reinforcement learning bench-
mark. Trans. Mach. Learn. Res., 2024. ISSN 2835-
8856. URL https: wwwp///weee.e..eeoo/ee//. net wwwppereeeew..et/torrr... id- (d=k2FKwal KwApeQQ..
14

---

Benchmarking Quantum Reinforcement Learning
A. 6G Beam forming and RL environment
This appendix elaborates on the 6G beam management environment used this work as a benchmark environment for different quantum and classical reinforeement learning algorithms.
Beamforming refers to the technique of transmitting signals in spatially selective manner despite interference and noise. There are multiple practical applications for radar and sonar, in communication technology, radio astronomy, seismology and many more (Trees, 2002). In recent years, the interest in beamforming has further increased due to applications in wireless communication. Beamforming is seen as one of the key technologies to accommodate the growing number of users of high data rate services (Corici et al., 2021). Next generation communication networks will feature antennas (in the following also referred to as base stations) capable forming directional beams and spatially targeting user equipment (UE). This process known as beam management.
In this work, consider specific optimization task, known as hand-over management, the task of switching base stations to maintain optimal quality of service for mobile UEs. For given positions of base stations, the transmitted spatial radiation intensity pattern is typically highly complex due to reflection and absorption effects. Selecting the optimal base station (the one with the highest beam intensity at the UE's location) would require knowledge about the exact spatial intensity pattern and geolocation of the UE based on initial high-resolution measurements of the intensity field. Machine learning, particularly reinforcement learning (Gershman et al., 2010), offers a promising simplification by implicitly learning the intensity field and UE position. Different antenna settings (such as transmission angle, phases and amplitudes of the transmitterss...) encapsulated codebook-a discretized mapping between these intrinsic antenna settings and resulting macroscopic intensity patterns. In general, the RL agent's task is to select the antenna and codebook element which achieves optimal service.
The practical problem sizes of interest far exceed what can be addressed by currently available quantum computing technology. Thus, we employ a simplified toy model with the underlying physics simulator taken to be as realistic as possible. This was ensured by pross-alidatting our simplified model with general physics simulator (Burkhardt et al., 2014) for this problem. Our setting confined two spatial dimensions, ignoring reflection, absorption, and beam interference. We assume one beam per antenna but allow variable number of base stations. The task is train reinforcementt-earning agent that selects the base station with the greatest radiation intensity at the current location of the agent each time step. We assume that the optimal beam direction (that is the codebook entry) of the selected base station is automatically determined by the antenna.
L
Lrfcccorn
intensity
Figure 10. typical setting in the toy-model reinforcement- seaarceemmme---ainnnggggg problem inspired by 6G beam management tasks. The UE moves on a randomly generated trajectory (orange line) in the intensity field of three base stations at positions (0, L/2), (L/2, (L/2, L). The main lobe of antennas is focused on UE (darker blue corresponds to lower intensity). Clearly visible are also the weaker maxima of the intensity fields stemming from interference effects within the antenna. By traversing the area between the base stations multiple times on different trajectories the reinforcement learning agent learns to select the hase station with highest intensity without knowledge of the spatial intensity distribution and the form of its trajectory. For better readability, in this figure the typical inversely squared decrease of electric field with the distance the source is removed.
15

---

Benchmarking Quantum Reinforcement Learning
The agent only has access the following information:
the index of the base station A,-1 E Antenna selected in the previous time step - 1, where Antenna is the set of antenna indices,
the active codebook element B;-1 Codebook of the base station selected in the previous time step, where Codebook is the set of codebook elements per antenna, and
the received radiation intensity T E Intensity from the previously selected base station.
Note that the agent neither has access to its current position in the radiation field nor to the specific structure the beam intensity field. While moving along randomly generated trajectory, the agent learns the mapping between this available information and the spatial beam intensity pattern and its current location. Figure 10 visualizes typical intensity field in toy model, showing the trajectory of UE and the beams from three base stations directed towards the UE. Figure 11 exemplarily details an environment instance with three antennas and shows the ground-trutt for the selection task.
A 940
$0*
Aa
A
A 1000
(a) Two instances of threc-antenme environment configuration, with different user trajectories. The dashed gray lines depict the environment boundaries, while the white and black markers indicate the position and indicated
(b) Ground-truth of the optimal intensity distribution, generated by brute-forcing over all available antenna- codebook configurations. At cach point, only the highest received intensity at each point is reported. The colors indicate the optimal antenna, the brightness the intensity magnitude. We emphasize truth solution, i.e. the optimal antenna selection does not just correspond to selecting the antenna with the smallest spatial distance the Moreover, in real-world sce- narios this information would even be accessible the RL agent, as localization the user induces unreason- able overhcad, and furthermore collides with user privacy concerns.
orientation of antennas A1, A2, and Ag. The trajectories by the green lines, the green crosses mark the current positions of the user. task is to select the antenna (greatest beam intensity) at the position of the user. The active antenna is marked in white, with the active codebook element indicated by (-). The (symmetric) discretized directions of the main beam indicated the white dotted lines. The secondary heams are caused by interference effects, but can also be used to serve users, significantly complicating the task of optimal selection. In cach timestep, the agent has to base its decision solely on the previously selected antenna, codebook element, and observed intensity, without access the spatial position of the
leads to optimal service
non-triviality
ground-
Figure different viewpoints the BeamMane nammmaent gementtennntmmm 6G environment used this paper. Figure (a) shows single active antenna at every step in time, which corresponds the task the RL agent has solve. Figure 11(b) visualizes the underlying ground-trui truth solution.
In the following subsections, first detail model of the iidiatiionintennsin field produced by the base stations in Appendix A. followed by discussion of the relation between beam direction, antenna configuration and codebook element in Appendix A.2. We conclude with the description of the procedure for sampling random trajectories in Appendix A.3.
A.1. Beam intensity field
We now study the antenna model used in this work and visualized in Figure 12. The base station is equipped with linear array of antenna elements senders (orange dots in Figure 12), with the jth sender at position ry. A sender is dipole radiation source, which we assume to be pointing in perpendicular direction to the 2d plane we consider.
Further, we assume the distance - ryl between observer at position r and the jth sender to much larger than the distance between the individual senders and introduce the effective location rg of the antenna such that r, ro + d,. Below,
16

---

Benchmarking Quantum Reinforcement Learning
dj
ro
sender
Fj
0
r Fj
Figure Model of an antenna used in the two-dimensional setting used this work. The senders a linear phased array are shown as orange dots in the figure. The electric field emanating from the jth point source is modeled in the far-field as spherical wave. A tunable phase offsct of each sender can used to direct beams exploiting interference.
To will point to the sender in the middle of the array (assuming an odd number of senders). Far away from the jth sender it's electric field position and time approximately a spherical wave (Griffiths, 2013)
A r-r
(r,
sin (k/rr - -ryl - wt + Φ5).
(4)
Here, A is a constant, w the frequency of the light wave, and h = w/c where is the speed of light is the wave number. We disregard the vector character the electric field since the far field of the antenna the electric field component of each sender will approximately point in the same direction. As we will see below, by tuning the phase offset Φy. directed high-intensity beams can be sent by exploiting interference effects. Next, we define R r ro. and R |R]. The following approximations valid in the far field, that is whenever |d,l/R < 1 (Griffiths, 2013),
r r- r-ryl= = R R R.d
(5)
R
R (d, -di)
k|[r
rıl - Ir - ry1 = k
R + O(R
(6)
R + 2)
rr r-rr ry]
The average field intensity at point in space is proportional to the square of the sum of all electric field components averaged over time, that is
o, 2
((r) lim 1 S'" dt (EEE))) T-000
A
Σ exp
E(K
R R.d
2R
The final equality follows by inserting Equation (4), making use of the approximation in Equations (5) to (7), and subsequent straightforward calculation. We model the base stations as a linear phased array of N, (N₂ odd) senders at position d, = jd with j -N,, 1)/2, ..., (N, 1)/2 and Φ, jop. Furthermore, |d| /k for maximal constructive interference. Substituting these definitions into Equation (8) yields the final result
B² sin (N.E/2)
1 = 21
(9)
sin (C/2)
with the proportionality constant B where
R d R 6
< k
(10)
Since Equation (9) diverges for R - O, we introduce a cut-off at the value of the intensity at R = 0.001. As the number of senders N₂ increases, Equation (9) strongly peaks at & = Measuring the direction of the beam by the angle with respect d, we find
cos =
(11)
17

---

Benchmarking Quantum Reinforcement Learning
In simple model, Equation (11) the mapping between the macroscopic field config configuration (the direction 0 of the beam) and the specific physical settings (the phase gradient yp of the antenna). The codebook then is vector (01,-.., ON. of discretized angles evenly distributed over the interval [0, 2/r) with the underlying mapping (p; cos(O;) with i = 1, No. In our experiments, we set N 17 and Ne- 9.
A.2. Antenna configuration
To generate instance of the environment, randomly sample the positions for a predefined number of antennas. Inspired by economic considerations in the real world, we avoid that pairs of antennas are located too close to each other. To enforce this, we first sample antenna coordinates R, uniform at random within the 2d environment [o, L] x [o, L). By default, we choose 6. We resample positions until the Euclidean distance between any of the antennas is greater than a threshold, i.e. RR; - R > dmin for all and j. The random configurations in our work were created with a value dmin = 1.5. The orientations of the antennas are individually sampled uniformly random, and successively normalized. For a position in the plane the ground truth (optimal antenna and codebook entry) is efficiently calculated by maximizing the intensity at by iterating over all codebook elements fp; for each antenna. Figure 13 illustrates some configurations for different numbers of antennas. The intensity field calculated via Equation (9) together with the discretized codebook gives rise to the complex patterns shown in the figure that the RL agent is tasked to learn. Note the difference between the intensity field pattern for particular codebook element as shown e.g. in Figure 10 and the pattern shown in Figure 13 where the intensity maximized over the codebook elements independently each position.
antennas
antennas
4 antennas
antennas
Figure 13. Different antenna configurations with increasing number of antennas. each position, the colors indicate the antenna with maximum intensity after maximization over the codebook elements. The brightness of the colors show the intensity where darker corresponds lower intensity. For better readability, the plots show R² where is the antenna's intensity according Equation
A.3. Trajectory sampling
In addition to the antenna configuration, we also sample random trajectory the agent moves on. To this end, we uniformly
sample n 2 2 points (z,, y:) for = 1,.... n within the 2d grid [o, L] x [o, L]. At the boundary we choose I1 = 0 and
Jn = L. The points are subsequently interpolated by a cubic spline function s(1) (v(1), x({) with E [o. 1] and s()) (0, y1) and s(1)
y.). We perform rejection sampling until 0 < y(t) < and < xxt) < for all t. The number of support points allows to control the complexity of the trajectory. Figure 14 exemplarily shows trajectories with increasing number of support points. Each episode starts at 0 and ends t 1. The parametrization of the trajectory s()) by í results in 1-dependent velocity. As simplification, we fix the velocity of the agent to which in general is trajectory dependent due to the different are lengths of the trajectories. To achieve a constant absolute value of the velocity, we re-parametrize s(((T)) and differentiate with respect to
ds Ös(t) dt dr Øt dr
(12)
Taking the absolute on both sides of the equation, fixing |ds/dr = and abbreviating v(t) = |0s(t)/Ot), we arrive at the differential equation
dr r(t)
(13)
dc 00
18

---

Benchmarking Quantum Reinforcement Learning
3 support points
4 support points
support points
6 support points
Figure 14. Different trajectory degrees. We sample n points in the two do-edmmmnsinall grid and interpolate them by a cubic spline function. The complexity of the trajectories increases with the number of support points n.. The figure shows sample trajectories for 3, .... 6.
Integration with the initial conditions t(T 0) and t(T = 1) 1 then yields
(tt) = vo [ dt'v(t')
1
(14)
with vo = So d(')((). We use this functional dependence between and т to re-parametrize the spline function of the trajectory.
B. Sample complexity estimator
This appendix provides more details on the statistical estimator for evaluation of sample complexity. Sample complexity, as used this work, follows the intuitive notion: Sample complexity is the number of interactions with the environment the reinforcement learning algorithm needs to achieve certain performance. More formally, following Ref. (Kakade, 2003; Kearns & Singh, 1999), given a threshold V*, the sample complexity the number interactions with the environment until measure of the algorithm's performance such the expected return is larger than the threshold with high probability. This definition is meaningful for RL algorithms with guaranteed monotonic convergence properties to the optimal policy (Kakade, 2003) such as policy iteration in tabular settings (Sutton & Barto, 2018).
However, for algorithms with no performance guarantees, the situation is more involved as showcased by Figure 15. The
algorithm 1
evaluated return
Vi
algorithm 2
www
number of algorithnnenniioomment interactions
Figure 15. The figure shows the learning curves for algorithm and algorithm 2. While algorithm 2 exhibits lower sample complexity with respect to threshold Vi¹ than algorithm 1, the converse is true threshold Vi', which algorithm 2 even never seems to reach. This example shows the following: If convergence to optimality cannot be proved for the algorithm, the definition of sample complexity is only meaningful when defined with respect to a threshold.
figure exemplarily shows the learning curves of algorithms. Since we consider here for example neural-network based algorithms with no convergence guarantees the optimal value, which might even be unknown, the question of which of
19

---

Benchmarking Quantum Reinforcement Learning
these algorithms has lower sample complexity can only be answered with respect predefined threshold. Indeed, in Figure 15 algorithm 2 has lower sample complexity with respect to V/ than algorithm because the learning curve shown first crosses the threshold V2. Conversely, algorithm has lower sample complexity with respect to Vi" which algorithm 2 even seems to never reach. In this case, we assign infinite sample complexity to algorithm 2.
So far the discussion has been for two specific training runs. In the following, we capture the notion of sample complexity more rigorously by viewing the training curve as the realization of a stochastic process. The stochasticity stems from randomness the environment but also from the algorithm itself (random initialization of neural network weights, sampling from action distribution, etc.). Statements about sampling complexity under randomness thus require the definition of statistical estimator introduced in the following:
Consider random process (V,, / 1,.... T). For given ō E (0, we define criterion to decide if V is below or above a threshold V via the probability Pt = P(V, 2 V*) and define the sampling complexity as
S Σ [PP
(15)
Here, [[-] is the indicator function which is one if its argument is true and zero otherwise. The number of time steps T is sent to infinity but in practice taken to be large enough to capture possible instabilities of the algorithm. Obviously, we do not have access to P.. which we therefore estimate from independent training runs. This leads to the following definition of empirical sample complexity:
Definition B.1 (Estimator empirical sample complexity general case). Given collection of N random processes (V:(), , 1, ,,,-. 7} with 1,.... N and Vi i.i.d. for given threshold value V R and threshold probability d E (0, 1]. we call
$ = Σ
uP
(16)
t=l
where
N
P = N 111
Σ
[V 2 v*].
(17)
the empirical sample complexity.
Note that R.P. = Pt, consequently P; is unbiased. The estimator S can be simplified when v, is bounded, e.g. V.
[0, V- In Section we redefine (((1) /Vmxx and V - V Vmax with E [0.1], 1]. In this case Equation (17) becomes
P
(1) 2 - o] T TV
(18)
In the following, we investigate the properties of Š, in particular consistency and bias. Theorem B.2 (Consistency). S is consistent, that is for all y > O we
we find that xim P(IS - S > n) = 0
Proof.
P(I$ 51 n) S E S
(19)
1)
1
B B - <]]
(20)
"
Σ
P( < I[P < 6]
(21)
20

---

Benchmarking Quantum Reinforcement Learning
We first used Markov's inequality, followed by the triangle inequality. In the final step made of the fact that the indicator function is either zero and that E [[-] P(-). With the definition A ò Pt. find
P(P: - Pi -Al1 SO -P(P -P2:
P(P. (
= IPP 8] +
(22)
i
and thus
lim N-000
P(P: < 6) 1|P < 6],
(23)
recalling that P. a consistent estimator for P., that is n P( P. > A) > The claim then follows when taking the limit of Equation (19) using Equation (23).
Let us now consider the large N limit and possible bias of the estimator. To this end, we calculate the probability of I[P: < 8] even though P. > 6.
Theorem B.3 (Bias). For large we have P(I < 8) 1) ((AVN/o) with A 5 Pa and P((11 Pt) where & is the error function.
Proof. We obtain an expression for P(I[Pt < 6] 1) by the central-limit theorem. We find
P( < = 1) = P(P; < 8) *(AvN/o) (N large)
(24)
where o² Var(I[V, 2 v*]) = P, P.) the error function ((z) dze
The estimator therefore is slightly biased around P. where Equation (24) deviates from the step function I(Pt 8) over width of /5(1 5)/ /N. Figure 16 shows the numerical evaluation for N = 100 samples and y() ~ Uniform(O, 1). The smoothed step function is clearly visible for different values of 5. For the uniformly distributed random variables considered here, the central limit theorem closely approximates the simulated curves for N = 100 which corresponds to the chosen number of training runs throughout this work.
1.0
0.8
0.6
8 = 0.2
0.5
8 0.8 contral limit theorem
0.4
0.2
0.0
0.0
0.2
0.4
0.6
1.0
Ph
Figure The figure shows numerical simulations y Uniform(O, 1). The y axis shows probability P(I[A <] <8 P( P. < 8), on the z-axis we sean P. = 1 V*. The figure shows smoothed step functions around P; = 8. For the exemplary distribution v(i) Uniform(O, 1). central limit theorem (dashed lines) approximates the functions well.
21

---

Benchmarking Quantum Reinforcement Learning
C. Experimental Details
This appendix provides details on the experimental setup that was used for producing the results in this paper. Moreover, we will highlight the implemented tools and configuration options. All implementations and raw results are accessible as described in the data availability statement.
We implemented the pipeline in python to be compatible with most of the ongoing research efforts in the ma- chine learning (MI.) and quantum computing (QC) community. The classical routines are mainly based on the PyTorch library (Paszke et al., 2019). For hyperparameter optimization of the quantum models we made use of the (Meyer et al., 2024), a library for fast simulation of quantum neural networks on multi-core systems. These initial experiments (results see Appendix C.I) were conducted on a system with a AMD Ryzen 9 5900X 12-Core CPU. As the successive experiments exceeded the capacity of a single machine, these were executed on the woody cluster of the Erlangen National Performance Computing Center (NHR @ FAU), consisting of 112 nodes of Intel Xeon E3-1240 v6 4-Core CPUs. As these are addressable in single-core granularity, the PennyLane library (Bergholm et al., 2022) was found to be more efficient for simulating the quantum circuits.
giskit-torch-modun
The pipeline itself consists of three main components:
(1) A realization of the BeamVanaagement 6G environment described in Appendix A, following the OpenAl gymnas um API (Brockman et al., 2016): The three-dimensional observation consists of base station index, received intensity, and codebook element of the previous time step. The first and last elements are normalized component -wise to the range [0, 1]. The intensity is inherently restrained to [o, 1j. The implementation allows to stack multiple past observations to provide more information to the agent, but this was not found to improve the overall performance for the setups considered in this paper. During training, the absolute received intensity value is returned as reward value. For testing purposes, also possible to report the fraction of the maximum achievable intensity. However, this internally computes the ground-trun truth solution, and therefore this information should not be used during training phase.
The agent's 's action is to select one of the available base stations by its index. Once selected, the environment internally performs sweep over the available codebook elements and selects that corresponding to the greatest intensity the position of the UE The internal spatial position of the environment is updated according to trajectory object following Appendix A.3. For the experiments in this paper, we selected a constant trajectory length of 200 steps. If not mentioned otherwise, the experiments were conducted with the three-antenna setup displayed in Figure 11. Our framework is implemented in a modular fashion, which allows to easily replace this environment with others available through the gymnasi im library.
(2) A RL algorithm that trains on the environment for defined maximum number of steps. For that purpose, we realized two different algorithms: Double Deep Q-Networks (DDQN) (Van Hasselt et al., 2016), an off-policy value- function based routine; Proximal Policy Optimization (PPO) (Schulman et al., 2017), an on-policy policy-function based routine. Most experiments in this paper refer to the DDQN algorithm, as off-policy algorithms are usually more sample-efficient than on-policy approaches (Sutton & Barto, 2018). However, to provide an additional example of using our sample complexity estimator, added some experimental results for PPO in Appendix C.2. Both algorithms integrated from the Tianshou library (Weng et al., 2022), which makes it straightforward to extend framework with additional RL algorithms. Our framework implements classical neural networks and hybrid quantum-classical networks as function approximators. Details on configuration possibilities and model sizes are provided and analyzed in Appendix D.
During the training procedure, validation is performed in regular intervals. The training and validation trajectories and all associated rewards are logged for successive computation of the sampling complexity. Additionally, intermediate and final parameters of the model are stored, which allows for retrospective fine-tuning and testing.
(3) A tool for evaluating the sample complexity, based on the logged trajectories from the training routine. Additionally, we implement the estimation of percentiles via cluster re-sampling (Cameron et al., 2008). This is realized as a post-processing step and requires only few computational resources once the training data is available.
A pseudocode overview of the end-to-end pipeline for estimating the sample complexity of DDQN can be found in Algorithm 1. A modified version for the PPO algorithm is provided in Algorithm 2.
22

---

Benchmarking Quantum Reinforcement Learning
Algorithm 1 Estimating the Sample Complexity of Double Deep Q-Networks (DDQN)
Input: environment (with horizon of 200 for BeamManagement6G)) state-action value function approximator Q.
Input: training runs Need (defaults 100), epochs to train for Nepuch (defaults 100),
Input: number of training environments New (defaults 10), validation environments Naal (defaults to 100) sample complexity of DDQN on environment & for threshold probabilities óo.... and error thresholds co....
Output:
Set up empty reward buffer of shape Need x Nepoch for seed different (random) initial parametrizations of Q do Initialize a standard DDQN algorithm (Van Hasselt et al., 2016; Weng et al., 2022) with Q and hyperparameters
E epsilon-greedy action selection
ac, aq: classical and (optional) quantum learning
reward discount factor
Nayne: target network synchronization rate
:bulter: experience replay buffer size
ibachi mini-batch size for update
for Napoch training epochs do
Use the DDQN algorithm on New parallel training environments
Employ current policy Nea parallel validation environments
Store the averaged validation result to the seward_buffer end for end for for threshold probabilities du, 51.... do for error thresholds en, e..... do
Determine the sample complexity and respective percentiles from the reward buffer as described Appendix B end for end for
Algorithm 2 Estimating the Sample Complexity of Proximal Policy Optimization (PPO)
actor approximator II, critic approximator V, Input: training runs Noeed (defaults 100), epochs to train for Nepuch (defaults 500),
Input: environment & (with horizon of 200 for ReamMa aaaamMaaagementetcamm
Input: number of training environments New (defaults to 10), validation environments N.a1 (defaults to 100) sample complexity of PPO on environment & for threshold probabilities 6y.... and error thresholds EU....
Output:
Set up empty reward. buffer shape Nwed X Nepoch
for Needd different (random) initial parametrizations TI, V do Initialize a standard PPO algorithm (Schulman et al., 2017: Weng et al., 2022) with 11, V and hyperparameters
Eclipi gradient clipping threshold
oc, aq: classical and (optional) quantum learning rate
reward discount factor
Nhothh mini-batch size for update
Nrepear: update repetition for each batch
for Napoch training epochs do
Use the PPO algorithm on New parallel training environments
Employ current policy Naa parallel validation environments
Store the averaged validation result to the reward.buffe end for end for for threshold probabilities &n SI.... do
for error thresholds Ep, ...... do
Determine the sample complexity and respective percentiles from the reward buffer as described Appendix B
end for end for
23

---

Benchmarking Quantum Reinforcement Learning
C.1. Hyperparameter Optimization
To evaluate and compare the sample efficiency, it important to optimize the hyperparameters of the involved algorithms. Only then statements can be made on the suitability and performance for specifie tasks. While focus algorithmic hyperparameters in this section, ablation study for different underlying models is performed in Appendix D. While it is impossible to consider every degree of freedom with fine granularity, we determine the hyperparameters that have the largest impact on the overall performance and in particular sample complexity.
For both the DDQN and PPO algorithm with classical and quantum function approximators the results are reported in Table 1. The bold values were determined be optimal by using grid-search with 20 runs for each configuration. These the settings that have been used to produce the results in the rest of this paper. While this does not proof that the respective models produce the best sample efficiency possible, believe that the performed comprehensive hyperparamter optimization allows for statements with high certainty.
Classical
Quantum
action selection Egreedy synchronization rate Nsyne replay buffer size Nbuller classical learning rate ac quantum learning rate aq
0.05, 0.1, 0.2 250, 1 1000, 2000, 4000
0.05, 0.1, 0.2
not separately optimized not separately optimized 0.001, 0.0005, 0.0002, 0. 0.0001 0.002, 0.001, 0.0005, 0.0002
DDQN (Algorithm
1000, 10000
0.01, 0.001, 0.0005, 0.0002, 0.0001
N/A
gradient clipping Selip update repetition Nrepeat classical learning rate ac quantum learning rate aq
0.05, 0.1, 0.2, 0.4 1, 5, 10, 50, 100
0.05, 0.1, 0.2, 0.4 not separately optimized 0.002, 0. 0.001, 0.0005 0.002, 0.001, 0.0005
PPO (Algorithm 2)
0.002, 0.001, 0.0005, 0.0002, 0.0001
N/A
discount factor mini-batch size Nhathh activation function"
0.90, 0.95, 0.99
not separately optimized not separately optimized None, ReLU, Tanh
Shared
32, 64
None, ReLU, Tanh
Trefers to the type of activation function used in classical neural networks between each layer; for hybrid classical-quantumn networks, for a sketch see Figure 3, placement is between input layer and quantum circuit, as well as between measurement and output layer.
Table 1. Hyperparameter optimization for the DDQN and PPO algorithm with underlying classical and quantum models. For each hyperparameter, we denote the considered values and the found optimal setup. For guaranteeing robust results, we performed full grid search with 20 seeds for each configuration. Due to the large simulation overhead, for the quantum models some hyperparameters were not separately optimized but taken over from the classical results.
C.2. Sampling Complexity of Proximal Policy Optimization (PPO)
Most experiments in this paper were conducted using the DDQN algorithm (Van Hasselt al., 2016), which can be considered an off-policy approach. This means that the agent can learn from actions that were not produced by the current policy. In DDQN this is typically realized by an experience replay buffer that allows to re-use past experience (Mnih et al., 2015). It is straightforward to see that such re-use of information can be expected to reduce the sample complexity. On the other hand, PPO (Schulman et al., 2017) is an on-policy algorithm. Such approaches can only learn from actions that were taken during the current training epoch. While this allows for more stable learning in some contexts, as well as other advantages (Sutton& & Barto, 2018), it does prevent the use of past information. Consequently, the sample efficiency can be expected to be tentatively lower compared to DDQN. However, it is still possible to quantitatively compare PPO with different underlying models.
In Figure 17, we report the sample complexities for three different classical PPO models and one quantum PPO model on the DeamMannageeene 6G environment. To allow for convergence, we increased the maximum number of interactions to 500 (epochs)- 200 (steps)- 10 (batch) = 1000000. Comparison with Figure 6 validate the above assumption that PPO less sample efficient than DDQN. However, for most --O-configurations we observe superior performance of the small quantum model to the small classical model (width 16), on par performance with the medium classical model (width 32), and only slightly inferior performance to the large classical model (width 64). This is in line with the other observations in this work, i.e. that quantum models can achieve a sample complexity competitive with much larger classical models.
24

---

Benchmarking Quantum Reinforcement Learning
Model (width-depth):
Quantum (10-4)
Classical (16-2)
Classical (32-2)
Classical (64-2)
E 0.20
0.15
0.10
1000k
= 0.50
+
750k
500k
sample complexits
250k
1000k
0.75
750k
500k
250k
604
740
2500
9092
604
740
2500
9092
604
740
2500
9092
model trainable parameters
Figure Empirical sample complexities Š proximal policy optimization (PPO) various relative errors & and threshold prohabilities ó on the BeamManagement 6C environment. The model width denotes the number of neurons in classical hidden layers, and the number of qubits in quantum models, respectively. Moreover, model depth refers to the number of hidden layers DNNs, and number of ansatz repetitions in VQCs. Compared to Figure 6 which shows the same study but for DDQN, the larger number of trainable parameter originates from the use of separate actor and critic networks. All results are averaged over 100 seeds and error bars denote the 5th and 95th percentiles, estimated with cluster resampling. Configurations where multiple do not achieve the targeted relative rate before the cut-off of one million steps are grayed out.
D. Analysis of Classical and Quantum Function Approximators
This appendix complements the discussion of the algorithmic setup from Appendix C with considerations regarding the underlying models. In work, distinguished between two classes of function approximators both for the value function in DDQN and the policy in PPO: On the one hand, classical fully-connected neural networks with varying width and depth of the hidden layers. On the other hand, hybrid classical- networks, i.e. variational quantum circuit with varying qubit and layer counts, encased by single-layer classical networks (sketch see Figure 3).
In Appendix D.I, we describe the hybrid model in more detail and discuss different ansätze for the underlying variational quantum circuit. An ablation study we conducted w.r.t. model complexity of both, classical and quantum approaches, is outlined in Appendix D.2.
D.1. Notes on Quantum Circuit Ansatz
The choice of the quantum circuit ansatz a frequently debated topie in the quantum computing community. It is possible compare different choices based measures such expressibility, trainability, and entanglement capability (Sim et al., 2019; Abbas al., 2021). There also cxist approaches for automatically generating architectures that optimal w.r.t. some of these properties (Du et al., 2022). However, apart from some artificial examples, relating these metrics to task-specific performance has so far been unsuccessful.
We decided for a hybrid instead of pure quantum model which avoids the dependency of the qubit number on the dimensionality of the RL state. This allows the flexibility arbitrarily scaling the VQC - for a sketch see Figure 3, More formally, we use an initial single-layer fully connected network to map the dimension the state observation s (0) to vector compatible with the input dimension of n-qubit VQC:
w pre
(()) bpe
(25)
This intermediate state (1) is encoded into the VQC with a parameterized unitary Uss with details on the trainable
25

---

Benchmarking Quantum Reinforcement Learning
parameters O and encoding procedure described below. The respective quantum state is evolved and the individual qubits measured in the Pauli-Z. basis to get the intermediate state s(2):
s(2)_ (oIU((())))) (z @ 18m-1) ((s0))))00)
(26)
(o (16n-1 () 0010
The intermediate result is post-processed using another single-layer classical neural network. This adjusts the dimensionality to the desired output dimension, usually the number of actions, i.e. antennas in the nageeeett 6G environment:
Dpost
(27)
post
For the DDQN algorithm, this output ( is directly used as an approximation for the Q-value function. In case of PPO, we consecutively append a softmax layer to get a probability density function approximating the policy. While the hybrid model incorporates both, trainable classical and quantum parameters, our ablation study on model complexity in Appendix D.2 demonstrates that the performance of these models can be mainly attributed to the quantum part.
H
U1D0
Uo.0
H
U1.0
Uo.0
U
H
Up,0
!/
Uia
Ug,1
H
U2a1
U0a1
UL
11
U0,1
H
U122
Uo.2
H
U122
U022
И
U1.2
H
Uo.2
(a) "Ent-C": CX entanglement
(b) "Ent-CZ": CZ entanglement
(c) "IQP": instantaneous quantum polynomial ansatz
Figure 18. Three different hardware-efficient ansatz structures compared for this paper. All configurations initially create uniform superposition, followed by potentially multiple layers of variational gates and two-qubit entangling gates. The different instances are realized with: (a) variational single-qube unitaries together with nearest- neighbr controlled-X X gates; (b) variational single-quie qubit unitaries together with saarest-ncihhoo controlled-Z gates; (c) controlled variational unitaries in nearest-ncighbor structure, followed by Hadamard gates; In the notation Ut.g- the index I denotes the layer and the qubit position. Consequently, for two and more layers we employ data re-uploading Pérez-Salinas et al., 2020), enhancing the expressivity of the ansatz. The variational unitaries are realized with one of the parameterizations of a universal single-qubit rotation shown below in Equations (29) to (31).
For studying the measure of sample complexity, we identified the most suitable ansatz out of architectures that are commonly used by the quantum machine learning (QML) community. More concretely, consider 3 different hardware- efficient circuit structures, all featuring most two-qubit interactions (Kandala et al., 2017). This includes (a) nearest- neighbor entanglement structure with controlled-X gates and single-qubit rotations, (b) nearest-neighhor entanglement structure with controlled-Z gates and single-qubit rotations, and (c) the so-called instantaneous quantum polynomial ansatz (Shepherd& & Bremner, 2009) with controlled rotations interleaved between Hadamard gates. visualization of these configurations can be found in Figure 18. In general, the single-qubit unitary U1,g(15 +) acts on qubit in layer I. We realize data encoding in the general form as
(s 0, A) U()Ln22 8y + Ou.,2, ALg. 89 + 0r.g.1, Ang.0 89 + 01,g.));
(28)
where the trainable parameters è are comprised of standard variational parameters 0, and state scaling parameters (Jerbi al., 2021). Furthermore, Sy denotes the q-th entry of the intermediate output from the classical encoding layer, see Equation (25). In this work, we consider 3 different parameterization of variational universal rotation gates:
U (s; 0, A) R3 (N... . Sy Ru : Sq + 01. (A1.g00 Sg + 01.g.0) U (s; X)= = R2(AL. 89 + al.22) Ry Atg.1 8g + 01..1) ) Rs (Ni,,0 + 01. U4 (s; = U3(>i.g.2 8, + +11 A.g. Sy + 01.g11, A Sy + 0r...)
(29)
(30)
(31)
All three expressions parametrize arbitrary single-qubit unitaries. The representation in Equation (29) is typically used by PennyLane. The second on in Equation (30) is the subsequent rotation along all three axis. The final one in Equation (31)
26

---

Benchmarking Quantum Reinforcement Learning
structure
IQP
Ent-CX
Ent-CZ
ROT
baseline throughout paper
+ slightly worse performance
significantly worse performance
e slightly worse performance
i comparable performance
C significantly worse performance
gate
XYZ
- slow simulation
slow simulation
slow simulation
θ extremely slow simulation
significantly worse performance
U3
unstable training curves
3 unstable training curves
e unstable training curves
Table 2. Different quantum circuit ansätze compared to the baseline selected for the experiments in this paper. We considered combinations ansatz layouts Figure 18 variational gates Equations (29) to (31). Overall we observed that IQP and CX-entanglement layout exhibited a performanee clearly superior CZ-entanglemenn. Furthermore, ROT and XYZ gates performed similarly, but the former was much faster to simulate, While the final performance using U3 gates was also comparable, the training procedure was much more volatile.
is the arbitrary single-qubit rotation parameterized as
(0.0,668) cos sin
sin ( el(++) cos (2)
(32)
While in principle all these parameterizations can be used to approximate the same functions, we observed significant differences regarding convergence stability and simulation speed. Our results are summarized Table 2. Overall, we identified the IQP structure with UROT parameterization as most suitable for optimizing the sample complexity on the BeamManagement 6G task. Therefore, this configuration used for all other experiments in this paper. The combination of the CX-Ent structure with UXY% parameterization exhibits almost equivalent performance but increased the simulation times due to not being native in PennyLane. As general rule of thumb, we discovered that the IQP and CX-Ent structure are superior to CZ.-Ent. Moreover, the training performance with ROT and U usually was much more stable compared to U03 We emphasize that this small study by no means should be considered an exhaustive architecture search. It might be possible to develop ansätze that are even more suitable for the considered task. However, such extensions are out the scope of this work.
D.2. Ablation Study of Model Complexity
In the following, we will investigate the impact of model complexity on the sample complexity. For both types of models we examined two degrees of freedom: For the classical model, this incorporates the depth, i.e. number hidden layers, and the width, i.c. the number of neurons in each hidden layer. With input dimension dim and output dimension dimouts the number of trainable parameters scales as:
input layer
hidden layer(s)
outpotlayer layer
output
weight parameters
dim width
+
(depth 1) width width +
width dim
(33)
bias parameters
width
+
(depth - 1) width
+
dimout
(34)
For the standard BeamManagement 6G environment configuration with dim,, = (i.e. antenna index, received intensity, and codebook element of previous timestep) and dimout (i.e. 3 antennas) this simplifies
(depth 1) width + (depth + 6) width +
(35)
overall trainable parameters, i.c. linear scaling in the depth and quadratic in the width. The parameter count approximately doubles for PPO as separate actor and critic networks are used with the critic requiring an output size of
For the hybrid classical-quantum model, the number of classical parameters depends on the in- and output dimensionality, as well as on the number of qubits in the VQC. Additionally, can increase the number of variational parameters by
27

---

Benchmarking Quantum Reinforcement Learning
appending additional layers. Therefore, the number of trainable parameters scales
impal layer
oulpul layer
(classical) weight parameters
dimin qubits +
0
+
qubits dimout
(36)
(classical) bias parameters
qubits +
0
dimout
(37)
(quantum) variational parameters:
+
layers qubits 3
(38)
(quantum) scaling parameters:
0
+
layers qubits 3
+
0 (39)
quantumn circuit
For the standard environment configuration with this simplifies
6 layers qubits + 7 ququtss +3
(40)
quantum parameters
classical parameters
overall trainable parameters. One can see from this expression that for increasing number of layers, the fraction of trainable classical parameters becomes insignificant in comparison to the parameters count of the quantum circuit. This ensures, that the performance of the model originates from the quantum part, while only pre- and post-processing is handled classically. Moreover, further below we show that small purely classical networks are not capable of learning a meaningful strategy in the BeamMaaagement 6G environment.
= 0.15
0.15
depth 2
200k
layers 4
200k
150k
150k
100k
100k
commlesxe Sasiity
50k
sample complexity
50k
16
32
256
12
14
width
qubits
0.15
0.15
sample
width
32
width
200k
qubits
qubits = 10
200k
150k
0.75
150k
100k
100k
50k
50k
2
3
6
depth
layers
(a) Classical neural networks with: (upper plot) increasing width of (b) Hybrid classical-quantum networks with: (upper plot) inereasing hidden layers for depth 2; (lower plot) inercasing depth hidden number qubits for variational layers; (lower plot) incrcasing layers for widths 32 and 64.
number of variational layers for and 10 qubits.
Figure 19. Impact of model size on the sample complexity in the BeamManagement 6G environment with the DDQN algorithm. For simplicity, we show only a single intermediate configuration of threshold probability 0.85 and error threshold y 0.15, but similar results were observed for other setups. In (a) depict the scaling bchavior of classical model with increasing width and depth, (b) refers to a quantum model with increasing qubit count and number of layers. The concrete parameter counts can be determined following Equations (35) and (40). All results averaged over 100 seeds. and error bars denote the 5th and 95th percentiles, estimated with 1000 repetitions of cluster hootstrapping.
We analyze the impact of model complexity, in terms of the trainable parameter count, on the sample complexity in Figure 19, All experiments were conducted on the Be BeamManagement 5G environment from Figure 11, using the DDQN algorithm. We report results for an in-between configuration of threshold probability 0.85 and error threshold æ 0.15.
For the classical models in Figure 19(a). clear performance saturation with model complexity can be observed. When the number of hidden layers is fixed to 2, the best sample complexity is achieved with hidden layer width of 64 (4611 parameters). The performance with a width of 128 (17411 parameters) and 256 (67587 parameters) is nearly equivalent but no improvement could be observed. Consequently, there should be a sweet-spot model size for optimizing the tpec-fee
28

---

Benchmarking Quantum Reinforcement Learning
qubits
6
8
10
12
14
3 total runtime [min:sec]
00:48
01:19
thereof for train test
00:23
00:19
00:44
00:26
total runtime [min:sec]
00:44
00:59
01:35
04:10
12:41
layers
thereof for train| test
00:20 00:18
00:30
00:21
00:55
00:29
02:46
01:07
09:21 | 02:48
5 total runtime [min:sec] thereof for train| test
01:09
01:57
00:37
00:23
01:10
00:33
total runtime [min:sec] thereof for train test
01:57
02:17
6
01:10
00:33
01:24 00:37
Table Expected runtimes for simulating one epoch of DDQN training with hybrid dassical-quantuue models. All times refer single-core performance on an Intel Xeon E3-1240 v6 CPU. Keep in mind that for the results in this work we typically trained for 100 epochs and averaged the performance over 100 runs. We report the total end-to-end times, as well required for actual Iraining, mostly for computing the gradients - realized using the backprop method from PennyLane; the time for intermediate testing, which is necessary for subsequent estimation of the sample complexity; For the classical models of the considered sizes, the time was approximately constant at only 15 seconds per episode.
other values: the time
sample complexity. A similar observation is made for increasing model depth, where close-to-optimal results are achieved for depth 2. Increasing this to depth 3 brings insignificant performance improvements, but significantly increases the parameter count - e.g. from 4611 to 8771 for width 64. Therefore, we selected a depth of 2 for the experiments in this work.
The hybrid classical quiiiccu--quuummmmmmm models Figure 19(b) not exhibit comparable saturation behavior for the considered model sizes. With 4 layers, the sample complexity reduces for qubit counts from 6 up to 14. Moreover, the model size in term of parameters only grows slowly for these instances, i.e. from 189 to 437. However, at this point we are faced with current technical bottleneck: While current quantum hardware is not robust enough to run the quantum models with high enough fidelity, classical simulation costs increase exponentially with qubit count. The simulation time increased additionally by large factor, as we have to execute 100 full training runs for single data point. In Table 3 we summarize runtimes that can typically be expected for training the different models in simulation. While the results suggest that it is possible to reduce the sample complexity even further by increasing the qubit count, experimental validation is currently infeasible. However, already these comparatively small quantum models are competitive with much larger classical models. This highlights the promising potential of scaling the quantum approaches, once the hardware development has caught up. Increasing the number of layers seems have less significant impact the overall performance, but still some gains might be possible there. For the experiments in this work, we selected a moderate size of layers.
Overall, we conclude that the classical baseline with width 64 and depth 2 is the sweet-spot model size w.r.t. sample efficiency. For the quantum model the best performance was achieved with 14 qubits and layers, but is reasonable to assume that further improvements are possible. For saving computational resources, for most experiments we reduced the qubit count to 10. Therefore, the results in this work could be interpreted as comparing an close-to-optimal classical model to only partially optimized quantum model with potential for future improvement.
E. Task Complexity of BeamManagemmnt 6G Environments
In this appendix, we discuss two different ways to adjust the difficulty of the developed BeamManagement 6G enviromment. So far, we conducted our experiments on the environment configuration in Figure 11, which contains 3 antennas. Furthermore, the trajectories were sampled using 3 support points, we also refer to this setup as trajectories of degree 3. In the following, we will vary both these setup parameters and observe the change in task complexity. We interpret the averaged sample complexity, exhibited across various models, as task complexity. While this might not quantitatively capture the ground-trule difficulty across all possible solution methods, it suffices for qualitative comparison. The results are summarized in Figure 20.
First, we use the standard environment configuration but increase the trajectory degree up to an value of 6. Intuitively, this leads to more complicated and unpredictable trajectories, which should complicate the task for the RL agent. Examples of
29

---

Benchmarking Quantum Reinforcement Learning
Model (width-depth):
Quantum (10-4)
Classical (16-2)
Classical (32-2)
Classical (64-2)
0.20
0.15
0.10
200k
6 0.50
150k
100k
sample complexisg
50k
200k
6 0.75
150k
100k
50k
6
3
trajectory support points
(a) Sample complexity of the standard HeamManagement environment with increasing trajectory degree, examples Figure 14.
Model (width-depth)):
Quantum (10-4)
Classical (16-2)
Classical (32-2) 2)
Classical (64-2)
0.30
0.20
0.10
200k
0.50
150k
semple Compann Sovityy
100k
50k
200k
6= 0.75
150k
100k
50k
5
number of antennas
(b) Sample complexity of random BcamManagement 6G environments with increasing number of antennas. The dashed lines depict the average over environments for each antenna count, for plots of the used instances see Figure 13.
Figure 20. Task complexity of different variants of the Be SeamManagement 6G environment in terms of the sample complexity exhibited by various classical and hybrid DDQN algorithms: (a) depicts this scaling behavior for instances of the standard 3-antenna environment from Figure 11; (b) shows the sample complexity over 5 instances with 2, 3, 4, and 5 antennas each, with trajectory degree of 3; The model width denotes the number of neurons classical hidden layers, and the number of qubits in quantum models, respectively. Morcover, model depth refers to the number of hidden layers in DNNs, and number of ansatz repetitions in VQCs. The values are averaged over these instances to for the randomness in antenna placement. All data points are calculated from 100 runs as before.
30

---

Benchmarking Quantum Reinforcement Learning
trajectories with varying number of support points can be found in Figure 14. As expected, in Figure 20(a) can observe an increase of sample complexity over all model instances. While this behavior is less visible for larger threshold probabilities ó and larger error thresholds E, for decreasing values of both the behavior get significant. Moreover, this replicates the behavior from the rest of this work, that the quantum model outperforms the similar-sized classical model, i.c. width 16 and depth 2, on all instances. Furthermore, for most e-6-configurations the performance closely matches that of the larger classical models. Note, that this is only the 10-qubit quantum model, i.c. one can expect performance improvement for 14 qubits, especially for the degree setup. However, due to the large overhead of generating the raw results for this setting, such considerations are too computationally expensive. For a trajectory degree of 6, the movement of the UEs becomes too unpredictable for all models to allow for informed antenna selection. Looking at samples of such trajectories in Figure 14 also suggests that a degree of at most 5 should be used to model human behavior. Overall, it is reasonable claim that with increasing trajectory degree also the underlying task gets more difficult.
Second, we modify the actual placement and orientation of the antennas in Figure 20(b). In all instances the trajectory degree is set to 3. We sample random instances for 2, 3, 4, and 5 antenna positions from [0.0, 6.0) x [0.0, 6.0] as explained in Appendix A.2, i.e. by enforcing an Fuclidean distance of at least 1.5 between pairs of antennas. The direction is assigned uniformly at random. Some of the resulting configurations can be found in Figure 13, all for each antenna count are available in the GitHub repository. As the environments seem become significantly more difficult with increasing antenna count (compared to increasing the trajectory degree) we relaxed the error thresholds e. As there is a lot of randomness involved for the actual environment setup, we averaged the sample complexity over all 5 instances, in order to get more robust estimate. For all models, one can see a clear increase in sample complexity with increasing antenna number. Similar to above, the quantum model exhibits a competitive performance for most e-d-configurations. Only for 8 = 0.75 and & 0.10 the performance on the 2-antenna environments inferior to the similar-sized classical model. However, for this setup also the usually best-performing classical model with width of 64 seems to struggle. Moreover, also for this setting could only simulate sufficient runs for the 10-qubit model. For antennas, all but the large classical model one environment instance fail to reach the desired quality threshold. This is not too surprising, given the complex interference patterns in Figure 13. To improve upon this, one might need to resort to larger models, or provide additional information to the agent. This can e.g. be done by stacking multiple of the past RL states. However, as the reported results are sufficient to see a clear trend w.r.t. task complexity, this is out of the scope of this work. To summarize, there is a clear correlation between the number of antennas placed in the environment and the resulting task complexity.
As discussed, both the trajectory degree, and the number of antennas can be used adjust the difficulty of the BeamMa n aamMManggemenn gement 6G environment. Furthermore, the clear real-world inspiration and the sound physical dynamics enhance the practical relevance. Overall, we are confident that the BeamManagement 6G environment lends itself as sophisticated benchmark for quantum reinforcement learning (QRL) with the possibility to create instances of increasing complexity.
31


Reviews:
[{"review_id": "4najJIk6co", "reviewer": "Reviewer_jcrL", "summary": "The paper presents a benchmarking methodology for quantum reinforcement learning (QRL) by introducing a statistical estimator for sample complexity and a robust definition of statistical outperformance. Through experiments in a novel benchmarking environment inspired by wireless communication tasks, the study finds that hybrid quantum-classical models can outperform similarly sized classical models but do not definitively surpass larger classical models. The results highlight the need for statistically rigorous evaluations in QRL and question some claims of quantum advantage. The study concludes that while QRL shows promise, its superiority remains uncertain and requires further large-scale empirical investigation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the assessment of our paper. We appreciate the constructive comments made by the reviewer and believe that addressing them has further strengthened our work.\n\n- Regarding **Claims and Evidence**:\n\t1. We agree on the concern, that the claim of *definitive quantum advantage* is not supported by the empirical results of our work. However, this is not something we want to claim, as we see our paper also as critique on the large collection of literature that makes such statements without rigorous statistical analysis. We also explicitly discuss this in Section 7.\n\t2. Regarding the speculative extrapolation to larger, real-world environments: We acknowledge that the statements on scaling with larger quantum models in Section 6.2 are formulated too strongly. We will make clear that the extrapolation is speculative, and further empirical investigation is required to make it more definitive (for which we unfortunately currently lack the compute resources, as training models with over 14 qubits requires upwards of 1 day per run)\n\t3. Regarding tuning of the optimal baseline: Of course we can not guarantee that the classical baseline is optimal wrt all hyperparameters. However, we conducted an extensive hyperparameter search summarized in Appendix C.1 (2160 hyperparameter configurations for DDQN, 1800 for PPO), which should give reasonable backing for our claims.\n\n- Regarding **Experimental Design Or Analysis**: Overall we consider three *standard* RL algorithms, i.e. VPG, DDQN, PPO; while of course there exist many more, our paper does not focus on benchmarking a broad range of algorithms, but rather provides the overall framework; extension to other algorithms and environments should be straightforward also for third-party researchers with the provided open-source code. An extension to the standard Cartpole environment is discussed in the rebuttal for Reviewer **Vo3S**.\n\nOverall, we emphasize that our paper should not be seen as yet another one claiming quantum advantage on some task with some model. We again thank the author for pointing out that currently there are formulations in the paper that suggest otherwise. Rather, our work provides the tools to establish robust statistical backing for such comparisons between classical and quantum models.  \n\n---\n\n## On the use of the BeamManagement6G environment:\n\nOur motivation for considering only a single (new) benchmark environment is the focus of our work on establishing a benchmarking methodology rather than on solving specific problems.\n- We think that the BeamManagement6G environment has its justification for our work. First of all, environments like Cartpole or Mountain Car are certainly a good benchmark, but are so simple that they can be *solved* by rather small classical and quantum models (small meaning under 100 parameters), which makes scaling analysis difficult. Other more scalable and complex environments, like Frozen Lake and Atari have the problem that the state spaces are too large for meaningful encoding into a quantum circuit. Some thoughts on properties we in general need for a environment suitable for QRL:\n    - The state and action spaces must be sufficiently small for an efficient encoding into a quantum state.\n    - The dynamics must be sufficiently complex for justifying the use of QRL in the first place.\n    - The state elements should be continuous to facilitate the continuous nature of quantum states, which is not the case for e.g. Gridworld environments. For the BeamManagement6G environment, the energy is conitnuous and the codebook element intrinsically corresponds to continuous beam angles.\n\n- Why the BeamManagement6G environment is a particularly suited real-world task with relevance of our sample complexity estimator:  As we discuss in our paper, sample complexity as a metric is well defined only with respect to a given performance threshold. In the BeamManagement6G environment, different UEs may have different service quality requirements, which translates to a different error threshold epsilon. Different policies for different levels of QoS can be trained, which can be naturally translated to different performance threshold levels, as highlighted in Figure 4 in the paper.", "claims_and_evidence": "The paper provides robust statistical evidence—through extensive experiments and resampling techniques—for its benchmarking claims, particularly that hybrid quantum–classical models can achieve lower sample complexity than similarly sized classical models. However, the broader claim of a definitive quantum advantage is less convincing. Specifically, the comparison may not fully account for optimal tuning of classical baselines, and the experiments are limited to small-scale, simulated environments. This makes extrapolating the results to real-world, large-scale problems somewhat speculative.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria—such as the statistical sample complexity estimator and the BeamManagement6G benchmark—seem well-suited for comparing classical and quantum reinforcement learning, however the reviewer is not familiar with the relevant literature.", "theoretical_claims": "Did not check.", "experimental_designs_or_analyses": "Experiments seem sound, though comparison to a broader range of RL updates should be considered.", "relation_to_broader_scientific_literature": "Reviewer is not familiar with quantum literature.\n\nAuthors should perform more experiments with diverse RL updates to determine how they vary in the quantum setting. Additionally, broader discussion of baseline tuning is required.", "essential_references_not_discussed": "Reviewer not sufficiently familiar with quantum literature to comment.", "other_strengths_and_weaknesses": "The paper is commendable for its creative integration of quantum and classical reinforcement learning through a robust, statistically grounded evaluation framework, and its application to a real-world inspired benchmark enhances its relevance and clarity. However, the study's limited experimental scale and potential under-optimization of classical baselines may restrict the generalizability of its conclusions.", "other_comments_or_suggestions": "None for now.", "questions_for_authors": "None for now.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GEubUZuMTD", "reviewer": "Reviewer_3L1D", "summary": "This paper analyzes quantum reinforcement learning to provide a more nuanced evaluation on the potential for quantum advantage. They introduce a sample complexity metric that aims to tackle the evaluation issues, and perform a number of empirical simulations on a new RL environment to evaluate the potential for quantum RL in different regimes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the assessment of our paper and are pleased to read that the reviewer thinks that `This paper does a lot of things right, and I think it serves an important role in the QRL literature.` We appreciate the constructive comments and are confident that addressing them has further strengthened our paper.\n\n - Regarding **Methods And Evaluation Criteria**: We agree with the reviewer that it can be difficult for the reader to get intuition on e.g. the hardness of the BeamManagement6G environment. Nevertheless, we believe that the environment is particularly meaningful for benchmarking RL vs. QRL as we argue in the rebuttal for Reviewer **jcrL**. Following the reviewer's suggestion we included the application of our evaluation methodology to the CartPole environment, see the rebuttal for Reviewer **Vo3S**.\n\t\n- Regarding **Relation To Broader Scientific Literature**: \nIn the revised version of the manuscript we added a paragraph were we clearly state suggestions on how to achieve robust and reproducible benchmarking results based on the methodology we introduced and stress its importance for standardization and to avoide false claims.\n\n- Regarding **Essential References Not Discussed**: We have extended our discussion in Section 2 with the suggested paper and are incorporating additional references on benchmarking classical RL (e.g. arXiv:2402.03046, arXiv:2209.12016) \n\n- Regarding **Other Weaknesses**:\n\t1. We agree, that Figure 8 does not add much value. We think it makes sense to move it to the appendix--depending on space--and instead incorporate the above mentioned results on Cartpole into the main part. \n\t2. We agree with the reviewer that focusing on parameter count isn't always adequate; e.g. the difference between models with 387 quantum and 4,611 classical parameters is less relevant due to simulation overhead (since training the quantum model takes more time than the classical model).  \n    Indeed the parameter count by itself is of limited value, only metrics like e.g. energy, time and sample complexity matter. The key point we want to convey is that parameter count can be used to define a sequence of models in model space (assuming additional hyperparamter search for given parameter count). This sequence can then be used to identify potential trends for example on the sample complexity axis as we scale to more powerful quantum models. We made this point more precise in the modified version of the manuscript. Comparing parameter counts can still have merit also for ther axis like time and energy, because:\n\t   - Using parameter count as a proxy (e.g., in Figure 7), we speculate that scaling quantum models may provide a net gain in sample complexity. Validating this requires larger-scale empirical analysis beyond our computational resources. This limitation explains why most QRL work focuses on fewer than 10 qubits, and our paper questions advantage claims in this regime. \n\t   - For complex problems, large classical models become expensive and time-consuming to train. If similar performance would be achievable with significantly fewer quantum parameters, executing quantum circuits could potentially be more efficient in terms of time and cost as quantum hardware matures.\n\t\t\n- Regarding **Other Comments Or Suggestions**: We agree that the information content of Figure 1 and 6 is limited, especially as both describe the same setup. With the new empirical results on the Cartpole environment, we can replace one of the figures with a new one providing more intuition.\n\n- Regarding the **Questions**:\n    1. We acknowledge this valid concern. Saturation is expected but might occur at lower sample complexity for quantum models than for classical models. Though speculative, the performance gap to the best classical model at higher qubit counts is small.  \n    The 14-qubit model pushed our simulation capabilities to the limit due to the 100 runs needed per model for robust analysis; simulating the circuit is feasible, but gradient computation and training are very expensive. Nearly a quarter of the 40,000 compute hours were devoted to these systems. However, our work provides benchmarking techniques, once more efficient simulations or suitable quantum hardware is available. \n\n    2. We focus on scaling qubit number since performance saturates quickly with increased depth, as you pointed out. While quantum parameters certainly require different interpretation than classical ones, we do not think that this can be stated based on Figure 18. Indeed, also with scaling of classical depth (hidden layers) the performance saturates quickly, i.e. we see the same behaviour as for the quantum models.\n\nWe thank the Reviewer again for the detailed suggestions. We think that currently there are a lot of issues in the QRL literature, of which we want to address at least a small fraction. We hope the discussed revisions and extensions improve the relevance and quality of our work from your perspective.", "claims_and_evidence": "The core claims of this paper (regarding the nuance required when analyzing quantum RL) are generally clear and convincing. Potential improvements will be outlined below.", "methods_and_evaluation_criteria": "I’m not sure I’m sold on the new environment. I agree it is valuable to have an environment that is small that you can also tune the complexity and scale of readily (although I would say many of the minigrid/gridworld environments fit that niche). The industrial relevance of the environment doesn’t matter to this paper as well. The real issue is that the environment decreases reader comprehension of the main points of the paper. There is no intuition on how hard or meaningful results are in this environment, so it can be hard to really build an understanding of the impact of the comparisons. If the authors are attached to this environment, then at the very least, they should include other environments that the paper they cite use (e.g. the usual CartPole, BlackJack, minigrid, mountain car etc.). This would both help reader understanding by providing the analysis in context of something they already know, and also help verify the generality of the results (if things are only checked on one environment, maybe it’s just a phenomenon of that environment and doesn’t apply to others?).", "theoretical_claims": "The focus was on empirical work, any theoretical claims are readily checked.", "experimental_designs_or_analyses": "In general, the experiments are sufficient.", "relation_to_broader_scientific_literature": "This paper builds upon the broad literature of QRL which has recently been focused on hybrid approximations using traditional RL algorithms. It suggests a number of improvements in the analysis of this work, but could be more forceful in its suggestions and analysis (as much of the existing literature is a number of problems well beyond what many RL papers have).", "essential_references_not_discussed": "The analysis of RL algorithms is the subject of quite some interest, and it would benefit the authors to contextualize their proposed evaluation more against this backdrop. Additionally, some literature like https://arxiv.org/abs/2006.16958 is missing (especially given this CDF is quite reminiscent of their sample efficiency.", "other_strengths_and_weaknesses": "This paper does a lot of things right, and I think it serves an important role in the QRL literature. However, I think there are some things that are holding it back, but could be readily improved upon. Some of which has already been discussed.\n\nI don’t think figure 8 adds much. It just says “classical and quantum both solved it and random doesn’t”, but that was already conveyed in the text (and is generally true of RL environments). I think there is a place here for a connection to a more traditional RL figure. I know there are issues, as pointed out by this paper, on the “iteration vs reward” plots, but it could be good to connect what people know from RL literature to this new framework (for example, if I see sample complexity of [X], how would that reflect itself in reward graphs). This figure could also just be cut entirely.\n\n\nThis is not necessarily unique to this paper, but the focus on quantum parameter count could use more justification. What I mean is that, classical parameter count only matters because it is a semi-useful proxy for speed (or really energy, since there might be cases where we can use more processors in parallel and gain more speed at the cost of energy) which is why there’s been a big shift away from parameter counts towards things like FLOPS (also with the advent of modern ML methods, the correlation between param count and speed was beginning to loosen). Now perhaps the same is somewhat true for quantum, more layers does take more time, and more qubits do take more energy to control, but I’m not sure the comparison between parameter counts means much. This is not an argument with model complexity, I agree the plots regarding parameters vs. model complexity are fair. But the general comparison of “small quantum” vs. “small/large classical”. The number of parameters don’t really matter, it’s only the advantage that matters (either via time, or energy, or sample complexity), so the idea of comparing 400 quantum vs 400 classical parameters doesn’t really seem meaningful (because those parameters are doing different things and we are operating in very different regimes of computation). Comparing quantum vs. classical in general is important, but the focus of papers on saying “small Q is better than small C” as if that’s meaningful (like we don’t care about the number of parameters, just about how it performs on those three axes) should be more addressed, specifically focusing the parameter discussion around model complexity and comparisons around sample/time.", "other_comments_or_suggestions": "From a purely constructive criticism perspective (this has no impact on my evaluation of the paper), I’m not sold on Figure 1. I get what the authors are going for, but it takes up a lot of space and doesn’t actually say much beyond (small C < small Q < big C). I don’t really have an idea for a better figure, but I believe there is one that could have higher information density.\n\nI also feel the same about figure 6. It seems like this should just be a table, the visual information of this is conveying only a small collection of discrete points (and the colors/labels don’t offer anything beyond what text would in a table).", "questions_for_authors": "1. Presumably in figure 7 quantum will saturate too? Also why not do depth rather than width? I see that analysis in Figure 18, but it looks flat, which seems to indicate to my point that quantum parameters aren’t interpretable in the same way.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNTUaLzOnv", "reviewer": "Reviewer_Vo3S", "summary": "The paper introduces a standard benchmarking methodology for quantum reinforcement learning (QRL) algorithms, executed with high statistical rigor. The proposed methodology emphasizes sample complexity as a key metric and introduces a statistical estimator for empirical sample complexity. Additionally, the authors define a notion of statistical outperformance to facilitate fair comparisons between algorithms. The benchmarking framework is applied to the BeamManagement6G environment, and the results are well-visualized, providing clear insights into the comparative performance of different QRL methods. The authors also provide code, ensuring reproducibility.\n\n### Update after rebuttal:\nThe authors have addressed my main concerns effectively. Their clarification on the estimator’s properties, expansion on related RL benchmarking literature, and initial results on an additional environment (Cartpole) strengthen the paper. The open-source implementation and careful framing of quantum advantage claims further support their contributions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the positive assessment of our paper. We are sure that we can address the comments sufficiently in a camera-ready version of the paper:\n\n- Regarding **Theoretical Claims**: You advocated for a more formal proof of the estimator's properties. We have derived key properties like consistency and bias in appendix B, and also empirically analyzed the approximation quality wrt the central limit theorem.\n\t\n- Regarding **Relation To Broader Scientific Literature**: We are working on extending our discussion on related benchmarking efforts of classical RL in Section 2 of our paper. Among others, we now cover works like e.g. https://proceedings.mlr.press/v119/jordan20a.html.\n\nFurthermore, we want to address the stated questions:\n\n- Currently, there is unfortunately no standardization in QRL. Our work may establish a standard for measuring sample complexity, but more work is needed for other metrics.\n- Indeed the adaption to other QRL/RL environments is straightforward, as our implementation (available and open-source) makes use of the gymnasium environment structure; as discussed below, we are currently in the process of generating results for the readily available Cartpole environment, for details see below.\n\nIn general, we want to thank the Reviewer again for the detailed suggestions. We hope that the changes and extensions we discuss above improve the quality of our work from your point of view.\n\n---\n\n## On the extension to other environments:\n\nIn our initial research we performed experiments on the standard Cartpole environment with  vanilla policy gradient. We observed some superiority of quantum models over classical ones, consistent with claims in the QRL community. However, we emphasize that we do not claim this as quantum advantage, as elaborated below:\n- We appreciate the reviewers' suggestion that including results on more well-known environments would strengthen our paper's clarity and generalization. We have uploaded two figures addressing the Cartpole environment to  https://zenodo.org/records/15097065?token=eyJhbGciOiJIUzUxMiJ9.eyJpZCI6ImU0MzE3YTcwLTY3OTMtNDRmOC1iYjJiLWQ0N2EwNTExNTBjNyIsImRhdGEiOnt9LCJyYW5kb20iOiIwNjliOTVjZWM4MTM5NTA1ZTQ4NzhkMjJmZGVlMzU5YSJ9.iIKBA2-L10uYnzHsE4ibgLj3A4OaD73cVatmN-Unu_r78aOP3w8ZEArx1qvjW9C502Wlvo-10E5dYNkfMG52sQ (anonymized) and will incorporate them into the camera-ready version. To explain:\n\t- **Figure 1** compares classical and quantum models for the CartPole-v1 environment. Interestingly, the quantum model achieves lower sample complexity across all epsilon-delta configurations. We performed extensive hyperparameter tuning and tested classical models ranging from 30 to ~17K parameters, reporting the best one we identified. While this indicates some superiority, we explain below why this shouldn't be considered quantum advantage.\n\t- **Figure 2** provides a cross-section of Figure 1 at a reward of 475—the threshold typically considered as solving the Cartpolev1 environment. It illustrates that the best model depends on the desired success probability and that results may not always be significant.\n- We want to highlight that, although these plots suggest some superiority of quantum methods, it's crucial to phrase conclusions carefully. We attribute the observed behavior to the inductive bias of quantum models for this problem. In contrast, many current QRL works claim quantum advantage without sufficient statistics and with around 10 runs (the above plots average over 100). Our paper advocates for more rigorous testing of such claims and provides tools to facilitate this.\n- In conclusion, we believe the Cartpole environment (and most standard RL benchmarks) is insufficient for rigorous testing of QRL. The performance depicted above was achieved with a quantum circuits on only 4 qubits, which are easy to simulate. Moreover, the environment isn't adaptable for meaningful scaling analysis, unlike the BeamManagement6G environment we studied. As summarized in Sections 7 and 8, we cannot definitively answer whether QRL provides actual quantum advantage—this requires further analysis on larger systems. However, we are confident that our work offers valuable tools for such benchmarks, once more efficient quantum circuit simulation methods are available or when execution on quantum hardware becomes possible.", "claims_and_evidence": "The authors claim that their benchmarking methodology enhances reproducibility and standardization in the fragmented field of QRL. This claim is well-supported by the detailed description of their statistical approach, including the empirical sample complexity estimator and statistical outperformance metric. The inclusion of publicly available code further strengthens the credibility of their claims by allowing independent verification.", "methods_and_evaluation_criteria": "The paper employs sound statistical analysis and machine learning techniques to develop a robust benchmarking methodology. The choice of sample complexity as a core metric aligns well with existing evaluation methods in reinforcement learning. The proposed statistical estimator and outperformance metric offer a meaningful and transparent way to compare QRL algorithms. The evaluation is well-documented, and the methodology is appropriate for the problem domain.", "theoretical_claims": "The paper does not focus heavily on theoretical proofs but rather on empirical validation. The statistical framework introduced for sample complexity estimation appears well-founded. However, a more formal proof or derivation of the estimator’s properties could strengthen the theoretical contributions.", "experimental_designs_or_analyses": "The experimental setup is well-structured, with clear definitions of benchmarking criteria and performance metrics. The use of BeamManagement6G as a test environment is justified, and the results are well-visualized and easy to interpret. However, further validation on additional QRL environments would enhance the generalizability of the proposed methodology.", "relation_to_broader_scientific_literature": "The paper builds upon prior work on sample complexity in QRL but offers a novel benchmarking approach with enhanced statistical rigor. While it acknowledges previous efforts in QRL evaluation, a more detailed discussion on how this methodology compares with other benchmarking efforts in classical reinforcement learning would be valuable.", "essential_references_not_discussed": "The paper appears to cite relevant prior work on sample complexity and benchmarking in QRL.", "other_strengths_and_weaknesses": "Strengths:\n- The introduction of a standardized benchmarking methodology is a significant contribution to QRL research.\n- The statistical rigor in defining sample complexity and outperformance metrics is commendable.\n- The clear presentation and well-visualized results make the findings accessible and interpretable.\n- The inclusion of code ensures reproducibility.\n\nWeaknesses:\n- The paper primarily focuses on BeamManagement6G; validation on additional environments would enhance its impact.\n- While the empirical approach is strong, further theoretical justification of the proposed statistical estimator would be beneficial.\n- A broader discussion comparing QRL benchmarking with classical reinforcement learning benchmarks could improve contextualization.", "other_comments_or_suggestions": "Clarify whether the benchmarking methodology can be easily adapted to other QRL environments beyond BeamManagement6G.", "questions_for_authors": "How does the proposed benchmarking methodology compare to classical reinforcement learning benchmarking approaches in terms of adaptability and standardization?\n\nDo you plan to validate this methodology on additional QRL environments, and if so, what are the main challenges in doing so?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C4U2uwftw9", "reviewer": "Reviewer_nWH6", "summary": "This paper addresses the issue of valid performance comparisons between classic RL and QRL.  The authors propose a benchmarking methodology, which is based on a statistical estimator for sample complexity and a definition of statistical outperformance. In addition, a novel RL benchmark is established on which they compare the performance of DDQN and PPO using classical neural networks and VQCs. The results of the experiments show that the quantum variants consistently outperforms the classical version when the number of trainable parameters is similar.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the assessment of our paper. We appreciate the constructive comments of the reviewer and are confident that addressing them has further strengthened the quality of our work.\n\n- Regarding **Methods And Evaluation Criteria**:\n\t1. An argument for why the BeamManagement6G environment is particularly meaningful for QRL models is outlined in the rebuttal for Reviewer **jcrL**. To show the validity of our methodology for more diverse environments, we repeat the valuation for the CartPole environment in the revised version of the manuscript, as discussed in the rebuttal for Reviewer **Vo3S**\n\t2. On the formulation of the beam management task as MDP / RL environment please see the argument at the bottom.\n\t3. As the Reviewer correctly points out, including several previous observations in the policy input can provide some (implicit) indication on the UEs direction and speed of movement, thus making action selection easier. We emphasize that treating the information from several previous time steps as observation is exactly what we do. This allows the agent to infer the current direction of its motion. We will clarify this detail in the updated version of the manuscript.\n\n- Regarding **Essential References Not Discussed**: Thanks for drawing our attention to these references, we already incorporated them into Section 2 for a updated version of our paper. One comment: While from our perception the first one is targeted for fault-tolerant quantum computing, i.e. a somewhat different setup than we consider, the guarantee of advantages related to sample complexity is quite interesting.\n\n- Regarding **Other Comments Or Suggestions**: We adapted the use of referencing to be consistent. We acknowledge that the first referencing of Figure 1 being delayed to page 6 is not optimal and will adapt that. However, we would opt for keeping it on the front page, as we think the plot conveys the core concept regarding our sample complexity estimator.\n\nIn general, we want to thank the Reviewer again for the detailed suggestions, and are especially happy of the assessment ``I think the premise of this paper is very interesting and important: find a good common basis to compare RL and QRL methods to yield solid results from which statements about advantages and disadvantages can be derived.`` We think that this perfectly summarizes the intention behind our work and hope this or a similar rigorous benchmarking procedure will be established in the realm of quantum reinforcement learning. Finally, we hope that the changes and extensions we discuss above improve the quality of our work from your point of view.\n\n---\n\n## On the Formulation of the Beam Management Task as an MDP/RL Environment:\n\nWe acknowledge that we did not fully clarify the subtleties of the BeamManagement6G environment and why it constitutes a valid RL benchmark. Our environment models handover management problem (selecting the most suitable base station) not the beam selection problem per base station. As described in Appendix A, the trained policy selects the optimal base station, while a low-level algorithm (outside the policy) selects the best beam from the available codebook at that base station.\n\nAt each time step, the agent observes the selected base station, the beam ID, and the beam intensity received. The choice of the base station (the chosen action) results in different next states (different base stations lead to different next beam ID and received intensity). Therefore, this setting is not a contextual bandit problem since the state reached after taking an action is dependent on the action taken and by that determines the future state-action sequence.\n\nWhether to RL or not to RL (https://arxiv.org/abs/2405.19045, https://arxiv.org/abs/2401.14823) has been discussed extensively for the problem of handover management:\nAlthough the policy lacks an explicit planning component (without knowledge of the UE's future movements or the low-level beam selection algorithm at each base station), it must consider the future effects of its actions (i.e., the discount factor $\\gamma$ cannot be zero) for several reasons:\n- Ping-Pong Effects: Incorrect base station selection can lead to lower beam intensities and may require reconnection to the previous base station. Such ping-pong effects hinder network and UE efficiency. \n-  Timing of Handovers: The policy must implicitly determine the optimal timing to connect to a new base station and disconnect from the current one to maintain high beam intensity values, ensuring consistent quality of service during handovers. \n-  Intermediate Base Stations: Although not penalized in this work, connecting to an intermediate base station between two others may provide minor benefits for a few time steps. \n\nWe also think that the exact definition of the task should not be too important for the main contributions of our paper: We establish the robust benchmarking procedure in an environment-independent way.", "claims_and_evidence": "The claims seem to be supported for the setup of the experiments.", "methods_and_evaluation_criteria": "I don’t understand why the only benchmark in this paper is a completely new benchmark. Why not using at least one or two well established RL benchmarks like inverted pendulum or cart-pole?\nI’m not convinced that the new BeamManagement6G benchmark is particularly useful RL benchmark.\n1.\tIs it even an RL problem? As I understand, the agent can select any of the antennas at any time, and there is no planning ahead of the very next step needed. Such problems are called contextual bandits and can be seen as RL tasks with gamma=0. The agent’s only task is to select the best next antenna given the current context (state).  Can you explain why discount values of 0.9, 0.95, or 0.99 are needed?\n2.\tThe true Markov state is only partially observable. The agent is limited to use only the current values of Antenna, Codebook, and Intensity. But why? If the agent would be allowed to look further into the past it could estimate the direction and velocity of the mobile phone way better.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "The hyperparameters used in the experimental design for the RL algorithms are good choices. As mentioned above, I’m not convinced that this benchmark is a good choice for their experiments.", "relation_to_broader_scientific_literature": "I think the premise of this paper is very interesting and important: find a good common basis to compare RL and QRL methods to yield solid results from which statements about advantages and disadvantages can be derived.", "essential_references_not_discussed": "There are two recent publications in the area of QRL that could be added to the literature review:\n\nQRL algorithm with provable advantage:\nWiedemann, Simon, et al. \"Quantum Policy Iteration via Amplitude Estimation and Grover Search–Towards Quantum Advantage for Reinforcement Learning.\" Transactions on Machine Learning Research. 2023.\n\nModel-based Offline QRL:\nEisenmann, Simon, et al. \"Model-based Offline Quantum Reinforcement Learning.\" 2024 IEEE International Conference on Quantum Computing and Engineering (QCE). Vol. 1. IEEE, 2024.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Sometimes “Fig. X” is used and sometimes “Figure X”\nFigure 1 is placed on the front page but mentioned for the first time on page 6.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was discussed controversially and even after rebuttal and discussion the assessments ranged from Weak Reject to Accept. The claims are essentially seen as sufficiently substantiated. The rebuttal was able to address most of the points of criticism. 

The selection of benchmarks continues to be criticized, particularly the use of a previously unknown benchmark.
The criticism remains that the initial submission did not reach this level of quality and that extensive changes were necessary and have now been made, but can no longer be assessed by the reviewers.
Decision: Accept (poster)