Curriculum reinforcement learning for quantum architecture search under hardware errors
Authors: Yash J. Patel
Akash Kundu
Mateusz Ostaszewski
Xavier Bonet-Monroig
Vedran Dunjko
Onur Danaci
Venue: ICLR 2024 poster (2024)
Keywords: Quantum Computing
Reinforcement Learning
Quantum Chemistry
Quantum Architecture Search
Optimization
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: The key challenge in the noisy intermediate-scale quantum era is finding useful circuits compatible with current device limitations.
Variational quantum algorithms (VQAs) offer a potential solution by fixing the circuit architecture and optimizing individual gate parameters in an external loop. However, parameter optimization can become intractable, and the overall performance of the algorithm depends heavily on the initially chosen circuit architecture. Several quantum architecture search (QAS) algorithms have been developed to design useful circuit architectures automatically. In the case of parameter optimization alone, noise effects have been observed to dramatically influence the performance of the optimizer and final outcomes, which is a key line of study. However, the effects of noise on the architecture search, which could be just as critical, are poorly understood. This work addresses this gap by introducing a curriculum-based reinforcement learning QAS (CRLQAS) algorithm designed to tackle challenges in realistic VQA deployment. The algorithm incorporates (i) a 3D architecture encoding and restrictions on environment dynamics to explore the search space of possible circuits efficiently, (ii) an episode halting scheme to steer the agent to find shorter circuits, and (iii) a novel variant of simultaneous perturbation stochastic approximation as an optimizer for faster convergence. To facilitate studies, we developed an optimized simulator for our algorithm, significantly improving computational efficiency in simulating noisy quantum circuits by employing the Pauli-transfer matrix formalism in the Pauli-Liouville basis. Numerical experiments focusing on quantum chemistry tasks demonstrate that CRLQAS outperforms existing QAS algorithms across several metrics in both noiseless and noisy environments.

CURRICULUM REINFORCEMENT LEARNING FOR QUANTUM ARCHITECTURE SEARCH UNDER HARD-
WARE ERRORS
Yash J. Patell.t,, Akash Kundu Mateusz Ostaszewski', Xavier Bonet-Monroig"" 1 Vedran Dunjko's 16 and Onur Danaci
'(aQat) Applied Quantum Algorithms, Leiden University
IInstittte of Theoretical and Applied Informatics, Polish Academy of Sciences
'Joint Doctoral School, Silesian University of Technology
Warsaw University Technology, Institute Computer Science
Lorentzz Institute, Leiden University
LIACS, Leiden University
Equaa contribution
ABSTRACT
The key challenge in the noisy intermediate-scale quantum finding useful circuits compatible with current device limitations. Variational quantum algo- rithms (VQAs) offer potential solution by fixing the circuit architecture and optimizing individual gate parameters in an external loop. However, parameter optimization can become intractable, and the overall performance of the algorithm depends heavily the initially chosen circuit architecture. Several quantum ar- chitecture search (QAS) algorithms have been developed to design useful circuit architectures automatically. In the case of parameter optimization alone, noise effects have been observed to dramatically influence the performance of the op- timizer and final outcomes, which is key line of study. However, the effects of noise on the architecture search, which could be just as critical, are poorly under- stood. This work addresses this gap by introducing curriculum-basee reinforce- ment learning QAS (CRLQAS) algorithm designed to tackle challenges in realis- tic VQA deployment. The algorithm incorporates (i) 3D architecture encoding and restrictions on environment dynamics to explore the search space of possible circuits efficiently, (ii) an episode halting scheme to steer the agent to find shorter circuits, and (iii) a novel variant of simultaneous, perturbation stochastic approxi- mation as an optimizer for faster convergence. To facilitate studies, we developed optimized simulator for our algorithm, significantly improving computational efficiency in simulating noisy quantum circuits by employing the Pauli-transie matrix formalism in the Pauli-Liouvill basis. Numerical experiments focusing on quantum chemistry tasks demonstrate that CRLQAS outperforms existing QAS algorithms across several metrics in both noiseless and noisy environments.
INTRODUCTION
The past decade has witnessed dramatic progress in the study and development of quantum process- ing units, prompting extensive exploration of the capabilities of Noisy Intermediate-Scall Quantum (NISQ) hardware (Preskill, 2018). To account for the stringent limitations of NISQ devices, varia- tional quantum algorithms (VQAs) (Peruzzo et al., 2014; Farhi et al., 2014; McClean et al., 2016; Cerezo al., 2021) were developed as suitable way exploit them.
In essence, VQAs consist of three building blocks: a parameterized quantum circuit (PQC) or ansatz, quantum observable allowing the definition of cost function, and a classical optimization routine that tunes the parameters of the PQC to minimize the cost function. Each of the building blocks corresponds to active area of research understand the capabilities of VQAs.

---

Amaer!
Adloe
Clessist
Palcy n(4in)
State representation of circuits
Agent Double deep Q-network
Hybrid Quantum-Clessical Environment
Illustration of the architecture of the double deep-Q network utilized by the reinforcement learning (RL) agent. The RL state s here describes the quantum circuit encoded as censor-based 3D grid whose axes correspond to the qubit index, depth (moment) and gate type. This information is processed through a multi- layer perceptron. For the output, the agent computes the policy, according to which the actions (as gates) in state are prohahilistically chosen. A classical optimizer optimizes the circuit and, upon completion, provides a reward that guides the agent to select subsequent actions.
Figure 1:
One such promising VQA with an application in quantum chemistry variational quantum eigen- solver (VQE). In VQE, the objective is to find the ground state energy of a chemical Hamiltonian H by minimizing the energy
E(0) min (101))
(1)
6
The trial state (b( is prepared by applying PQC U(6) to the initial state |imitaa) where 0 specify the rotation angles of the local unitary operators in the circuit. The structure of this circuit significantly impacts the success of the algorithm. Traditionally, in VQAs, the structure of the PQC fixed before initiating the algorithm and is often motivated by physical (Peruzzo et al., 2014) or hardware (Kandala al., 2017) considerations. However, fixing the structure of the PQC within VQA may impose a substantial limitation on exploring the relevant parts of the Hilbert space. To circumvent such limitations, recent attention has turned towards automatically constructing PQC through quantum architecture search (QAS) (Grimsley et al., 2019; Tang et al., 2021; Anastasiou al., 2022: Zhang et al., 2022). This approach removes the necessity for domain and often yields superior PQC's tailored for specific VQAs. Given a finite pool of quantum gates, the objective of QAS is find an optimal arrangement of quantum gates to construct a PQC (and its corresponding unitary (00pt)). where Oops are optimal parameters) which minimizes the cost function (see Eq. 1).
knowledge
One such proposal to tackle the QAS problem is to employ reinforcement learning (RL) (Os- taszewski et al., 2021b; Kuo al., 2021), where the PQCs are defined sequence of actions generated by a trainable policy. The value of the quantum cost function (optimized independently via a classical optimizer) serves intermittent signal for the final reward function. This reward function then guides policy updates to maximize expected returns and select optimal actions for subsequent steps.
Up to the completion of this work, most algorithms for QAS have been formulated under the as- sumption of idealized quantum computer, free from physical noise and endowed with all-to-all qubit connectivity. However, it is essential to recognize that the QAS problem becomes even more daunting in light of the constraints imposed by current NISQ devices (Du et al., 2022). According to (Wang et al., 2021), the impact of noise on the trainability of VQAs in the context of QAS is multifaceted.
First and foremost, noise induces the phenomenon of barren plateaus, causing gradients within the cost function landscapes to vanish, thereby impeding optimization. Moreover, quantum noise in- troduces additional complexity by transforming certain instances of exponentially occurring global minima into local optima, posing a substantial challenge to trainability (Fontana et al., 2022). Addi- tionally, noise alters the cost landscape to an extent that the optimal parameters for the noisy land- scape may no longer align with the optimal parameters for the noiseless landscape (Sharma et al., 2020). In addition to the influence of physical noise the landscape, the finite sampling noise hinders the performance of classical optimizers working on that landscape, raising the necessity of optimizers robust to such conditions (Bonet-Monrois et al., 2023). Overall, the adverse effects in-
2

---

duced by the quantum noise demand swift simulation of numerous queries from noisy landscapes and robust classical optimization strategies operating under these conditions.
Performing experiments that assess the performance of QAS algorithms in the presence of noise, though computationally demanding. is a critical step toward understanding and ultimately overcom- ing the challenges presented by real-world quantum hardware.
The contribution of our work is two-fold. Firstly, we introduce the curriculum-bast reinforcement learning QAS (CRLQAS) method depicted in Fig. 1. Secondly, we provide an optimized machinery for CRLQAS that effectively simulates realistic noise-a crucial element for testing and enhancing larger system sizes. The machinery uses offline
method for quantum chemistry problems computation of Pauli-transfer matrices (PTM) and GPU based JAX framework to accelerate compu- tations by up to six-fold. To further improve the learning process of CRLQAS, we introduce several key components:
A mechanism, namely, illegal actions preventing invalid sequences of gates.
(2) A random halting procedure to steer the agent to learn gate-efficient circuits. A tensor-based binary circuit encoding that captures the essence of the depth of the PQC and enables all-to-all qubit connectivity.
(4) Two variants of simultaneous perturbation stochastic approximation (SPSA) algorithm that use adaptive momentum (Adam) and a variable measurement sample budget for faster con- vergence rate and robustness.
By leveraging strategies (1)-(4, we achieve chemical accuracy for a class of chemical Hamiltonians with better accuracy by maintaining both gate and depth efficiency in noisy and noiseless scenar- ios. Our numerical demonstrations often establish CRLQAS as the superior approach compared to existing QAS methods within the context of VQE across a spectrum of metrics.
2 RELATED WORK
There is a growing body of research in the field of QAS aimed at enhancing the efficiency and performance of quantum algorithms. Several key themes emerge from related works, encompassing QAS, optimization strategies, and the application of RL to quantum computing.
Evolutionary and Genetic algorithms Existing literature has explored various strategies auto- mate the design of quantum circuits. Evolutionary algorithms (EAs), particularly genetic algorithms (GAs), have been utilized evolve quantum circuits (Williams & Gray, 1998). While GAs demon- strate the capability to evolve simple quantum circuits, such as quantum error correction codes or quantum adders (Bang & Yoo, 2014; Potoček al., 2018; Chivilikhin et al., 2020; Bilkis et al., 2021), they face limitations in handling parameterized rotation gates. They are also known to be sensitive to gene length and candidate gate set size.
Sampling-based based algorithms As step forward, sampling- based learning algorithms have been in- troduced to sample circuits from candidate sets, addressing the QAS problem for ground state energy estimation (Grimsley et al., 2019; Tang et al., 2021).In (Zhang et al., 2022), the authors utilize Monte Carlo sampling to search for PQCs on QFT and Max-Cut problems. In (Du et al., 2022), a QCAS method based supernet and weight sharing strategy was introduced to better estimate energy for quantum chemistry tasks. However, in the presence of hardware noise, this method fails to estimate the ground state energy within a precision make realistic chemistry predictions. Meanwhile, (Wu et al., 2023) uses the benchmark their algorithm on VQAs, including quantum chemistry tasks.
technique (Gumbel, 1948) to sample quantum circuits and
algorithms RL techniques have also been applied to tackle the QAS problem for VQAs. Typically, such approaches employ carefully designed reward function to train the agent to choose suitable gates. (Ostaszewski et al., 202 la) employed double deep Q-network (DDQN) to estimate the ground state of the 4- and 6-qubit LiH molecules. RL has also been used to address the qubit routing problem (Li et al., 2019; Sinha et al., 2022; Pozzi et al., 2022). These works aim to minimize circuit depth through qubit routing using RI. techniques. Additionally, (Kuo et al., 2021) have employed RI. to test their algorithms on 2-qubit Bell state and 3-qubit GHZ, state in the presence of hardware noise. We summarize the relevant QAS methods for the ground state energy estimation task in Appendix K.
Slinforremeen--eaannnn---esess
3

---

3 CURRICULUM REINFORCEMENT LEARNING ALGORITHM
We give an overview of the CRLQAS algorithm construct PQCs for a VQA task, wherein we present state and action representations and the reward function used in this work. Later, we also describe the features of CRLQAS, which yields better performance across several metries.
In the CRLQAS environment, the agent starts every episode with an empty circuit. It then sequen- tially appends the gates to the intermediate eircuit until the maximum number of actions has been reached. At every time step of the episode, the state corresponds to circuit and an action to append a gate to that circuit. As we employ deep RL methods, we encode states and actions in a way that is amenable to neural networks. Thus, each state is represented by a newly introduced tensor-based binary encoding of the gate structure the PQC, which we describe Sec. 3.3. For our simulations to construct the circuits, we consider the gate set consisting of Controlled-Ne parameterized rotation gates (RX, RY, R2). We consciously omit the use of the continuous parame- ters of rotation gates and only utilize the estimated energy of the circuit in the state representation of the RI. agent.
(CNOT) and I-qubit
To encode the actions, we one-hot encoding scheme as in (Ostaszewski et al., 2021b) with CNCT and rotation gates represented by two integers. For CNOT, these values specify the positions of control and target qubits, respectively. In contrast, for the rotation gates, the first value specifies the qubit register, and the second indicates the rotation axis (the enumeration of the position starts from 0). For an N-qubit quantum state, the total number of actions is 3N + 2(²). where the first term comes from the choice of selecting the rotation gates and the latter from choosing CNOTs.
To steer the agent towards the target, we use the same reward function It at every time step & of an episode, as in (Ostaszewski et al., 2021b). The reward function R defined as,
if C < E, if 2 T. and C. 2 E, otherwise
R
5
C1-1-C
max
Ct-1 -C-in
where C, refers to the value of the cost function C' at each step, & is a user-defined threshold value and TH denotes the total number of steps s allowed for an episode e. T can also be understood the maximum number of actions allowed per episode. Note that the extreme reward values (1:5) signal the end of an episode, leading two stopping conditions: exceeding the threshold & or reaching the maximum number of actions. For quantum chemistry tasks, the threshold & is typically set to 1.6 10-3 as it defines the precision such that realistic chemical predictions can be made. The goal of the agent is to obtain an estimated value of Cmin within such precision.
The continuous parameters 0 of the quantum circuit that describes the cost function C are optimized separately using a classical optimizer to obtain the reward R. The choice of the classical optimizer is critical for the RL agent's success in the environment. The nature of the CRLQAS reward func- tion depends on the cost function evaluation. The cost function evaluation can be deterministic stochastic based both the classical optimizer and quantum noise. Stochastic optimizers, like SPSA, tend to converge toward different parameters, resulting in different function values across multiple trials, even when initialized identically. Moreover, the quantum noise may also lead to different function values even with the same parameters.
We consider both environment types (details discussed in Sec. 4) and successfully validate the ef- fectiveness of novel features introduced in this work for the CRLQAS method. The results of our ablation study to identify the features that standout in CRLQAS can be found in Appendix E. In the next section, we describe the features of this method. We adopt the "feedback curriculum learning" approach from the (Ostaszewski et al., 2021b) which is elaborated the Appendix B.I.
3.1 ILLEGAL ACTIONS FOR THE RL AGENT
As QAS is a hard combinatorial problem with a large search space, pruning the search space is beneficial for the agent to find circuits with different structures. Hence, we introduce a simple mechanism, namely, illegal actions to narrow down the search space significantly. The mechanism uses the property that quantum gates are unitary, and hence, when two similar gates act the same qubit(s), they cancel out. An elaborate discussion on the implementation of the illegal actions mechanism is provided in Appendix B.2.
4

---

3.2 RANDOM HALTING OF THE RL ENVIRONMENT
In the CRLQAS algorithm, the total number of actions executed by the agent within an episode, denoted as T5. is set using multiple hyperparameters. The hyperparameter, Ract, determines an upper limit on the total actions in an episode, meaning T S Ract- If the agent is not interrupted by any of the schemes mentioned in the paper. it selects maximum of T = Ract actions (gates).
If the RL agent finds a circuit that estimates an energy value (upon optimizing parameters) lower than this threshold, the episode is halted abruptly leading to T < Raat. When employing the random halting (RH) scheme, both the curricula and a stochastic sampling procedure then influence the episode length. We use samples from the negative binomial distribution to determine the cap on the total number actions per episode, T., at the beginning of each episode (Dougherty, 1990). The probability mass function of this distribution is
Roat - pm
Pr (X mpa)
p) Tan Mlst
Aaaiil
where Paat represents the hyperparameter for the total number of allowed actions per episode, and in this context, it is the total number of Bernoulli trials as well. Also, 7'fail denotes the number of failed Bernoulli trials, and the p denotes the probability of a Bernoulli trial to fail, which we provide as another hyperparameter. The probability mass function given above yields the probability of having Apa failed Bernoulli trials given aaat total trials and failure probability. In practice, we sample TÝ Aaai random number based on the failure probability, and the total number of experiments is determined via inverse transform sampling implemented in NumPy (Harris et al., 2020). This inverse sampling generates a number within the range T5 [0, naa), and we obtain this number the outset of each episode.
The primary motivation for integrating RH into CRLQAS is to enable the agent to adapt to shorter- length episodes, thereby facilitating the agent's i ability to discover more compact circuits in early successful episodes, even if occasionally leads to delay in achieving the first successful episode.
3.3 TENSOR-BASHE BINARY CIRCUIT ENCODING
Several QAS algorithms often require the compact representation of the circuit, also commonly known as encoding, as it allows for modification, comparison, and exploration of quantum circuits. Hence, the choice of encoding plays a vital role in efficiently maneuvering the search space and uncovering efficient and novel quantum circuits.
We provide the agent with a complete description of the circuit by employing binary encoding scheme that captures the structural aspects of the PQC, specifically, the ordering of the gates. To keep the dimension of the input tensor constant throughout an episode, the tensor must be prepared for the deepest quantum circuit that can be generated.
For constructing a tensor representation of the circuit, we initially specify the hyperparameter Racts which determines the maximum number of allowed actions (i.e., gates) in all episodes. We now define the moment of a PQC, which is crucial for understanding the tensor representation. The mo- ment or layer of quantum circuit represents all gates that can execute simultaneously; essentially, it is a set of operations acting on different qubits in parallel. The number of these moments determines the depth of the circuit. We represent PQCs with 3D tensors such that each matrix (2D "slice") rep- resents a different moment of the circuit, and the other dimension represents the depth (see Fig. 1). We use the maximum number of actions parameter, nact at given episode as an upper bound on the depth of the circuit. We give this upper bound for the extreme case where all the actions are implemented as I-qubit gates appended to the same qubit. As a result, at each episode, we initialize an empty circuit of depth Naa by defining a [Ma x ((N + 3) x N)] dimensional tensor of all zeros. Here N is the number of qubits. Each moment or layer in the circuit is depicted through a matrix of dimensions ((N + 3) x N). In this matrix, the initial N rows showcase the locations of the control and target qubits for the CNOT gates applied during that specific moment. Following these, the sub- sequent three rows indicate the positions of 1-qubit rotation gates RX, RY, and RZ, respectively. The visualization of such an encoding for a toy circuit is depicted in Appendix B.3.
3.4 ADAM-SPSA ALGORITHM WITH VARYING SAMPLES
In the realm of VQE within a limited measurement sample budget, several variants of simultane- ous perturbation stochastic approximation (SPSA) have displayed robustness towards finite sample
5

---

(shot) noise (Cade et al., 2020; Bonet-Monroig et al., 2023). One such family of variants, multi- stage SPSA, reset the decaying parameters while tuning the allowed measurement sample (shot) budget between stages. Implementing a moment adaptation subroutine in classical ML, such as Adam (Kingma& & Ba, 2014), alongside standard gradient descent, increases robustness and conver- gence rates. This combination has also shown promise in the domain of quantum optimal control for pulse optimization (Leng et al., 2019).
We investigate multi-stage variants of such an Adam-SPSA algorithm, exploring different shot bud- gets and continuous versus reset of decay parameters (after every stage). In doing so, we empirically observed increased robustness and faster convergence rates of a 3-stage Adam-SPSA with continu- ously decaying parameters. We note that this particular observation is novel and was not discovered before to the best of our knowledge. In Appendix F, we present empirical results demonstrating the convergence of this Adam-SPSA variant across VQE tasks involving various qubit numbers and shot budgets. Leveraging these enhancements, we managed to halve the number of function evaluations, thereby significantly reducing the evaluation time for RL training under physical noise.
3.5 FAST GPU SIMULATION OF NOISY ENVIRONMENTS
Most QAS algorithms require a considerable amount of noisy function evaluations unless a training- free algorithm is used. However, computing these evaluations becomes challenging within modern- day simulation frameworks. particularly due to the method of noise simulation known the Kraus operator sum formalism. This formalism hinders the use of fast simulation methods tailored for noiseless scenarios. Consequently, with increase in qubits and gates, simulations are hindered not only by the curse of dimensionality associated with dense matrix multiplications but also by the exponential increase in the number of noise channels and their corresponding Kraus operators.
Most importantly, this computational overhead needs to be paid online (during training and parame- ter optimization within episodes) at each step (see Appendix H). To mitigate this, the Pauli-transfer matrix (PTM) formalism is applied, allowing the fusion of noise channels with their respective gates to be precomputed offline. eliminating the need for recompilation at each step. In conjunction with PTMs, we employ GPU computation coupled with just-in-time (JIT) compiled functions in JAX (Bradbury et al., 2018), yielding up to six-fold improvement in RL training while simulating noisy quantum circuits (see Appendix l).
EXPERIMENTS
Our objective is to assess the performance of CRLQAS described in Sec. 3 to find circuits that are accurate and can overcome hardware noise. We benchmark CRLQAS for the task of finding ground-state energies via variational quantum eigensolvers (VQE) of three molecules, Hydrogen (H2), Lithium Hydride (LiH) and Water (H₂O). For all three molecules, we use their representa- tion in the minimal STO-3G basis, mapped to qubits through Jordan- Jordan-Wigner and Parity transfor- mations (Ostaszewski et al., 2021b; Wu et al., 2023). To simplify the computational task, we use symmetries of the problem taper off qubits, thus calculating the ground-state energies of 2-qubit H2 (H2 2). 3-qubit H2 (H2- - 3), and 4-qubit H2 (H2 - 4), 4-qubit LiH (LiH 4) and 6-qubit LiH (LiH - 6) and 8-qubit H₂O (H₂O 8) (see Appendix L for description of molecules). Addition- ally. we use hardware noise profiles from publicly available resources IBM (Corporation, 2023) to implement noisy models of varying complexity and run experiments on the first three qubit systems. The relevant details about the implementation of the CRLQAS method and its hyperparameters are outlined in Appendix C.
In the subsequent subsection outlining noisy simulations, the RL agent consistently receives signals as noisy energies during training, guiding its action choices. However, while post-processing the data of the trained agent, we only assess energies in noiseless scenario to determine the success or failure of an episode. An ablation study to identify the features that standout within the CRLQAS method in both noiseless and noisy settings can be found in Table 3 of Appendix E. Our analysis reveals that incorporating illegal actions without random halting prompts the agent to achieve posi- tive signal (a successful episode) early on, albeit resulting in larger eircuits. Conversely, introducing random halting encourages the agent to discover shorter circuits, there is a orf as the agent receives the positive signal at later stage.

---

Parameters
Parameters
Depth
18M Mumbai Median
Depth
IBM Ourense QCAS
10 deporalizing
Noiseless
I0M Mumbai 10
Noiseless
10a
Number Paramaters, Depth, Gates
10a
10
Hz-2
3--3
H2
LIH-4
Figure 2: Achieving the chemical accuracy for H2 (with 2-, 3- and 4-qubits), and LiH (with 4-qubits) molecules via systematic study under realistic physical noise where the noise model mimies the IEM Quantum devices. the initial episodes, the probability of choosing random actions is very high, avoid this, we consider the statistics from 2000 episodes onward and plot median over 3 different seeds. The different colours denote the different levels of noise (increasing from bottom to top), and the patterns (from left to right) denote the number of parameters, the depth and the number of gates, respec- tively. We reach the chemical accuracy for H2 and Ha (except the 10 times device) molecule for levels noise. Meanwhile, H2 4 molecule reaches the chemical accuracy noise profile of IBM Ourense even with qubit- connecc-cnttttityy constraints. Finally, with LiH 4, we achieve the chemical accuracy with shot and I-qubit depolarizing noise. Note that, for H2, we decreased the threshold (usually set chemical accuracy) 2.2 x 10 because the problem straightforward to solve.
minimum
noise of IEM Mumbai
4.1 NOISY SIMULATION
To simulate molecules, we consider a realistic noisy setting without drift, employing the noise pro- file of the IBM Mumbai i (see Appendix H) and TBM Ourense (Du et al., 2022) quantum devices. Despite not considering drift time-scales of quantum computers, our experiments on a real quantum device (see Appendix J) corroborate hypothesis previously noted in (Rehm et al., 2023), that clas- sical simulations noisy quantum circuit evaluations closely resemble actual hardware behavior, particularly under low noise levels. When noise is present, estimated cost function values for given parameters differ from those in noiseless scenarios. This discrepancy challenges leveraging prior domain knowledge (like ground state energies) for configuring rewards and curriculum mechanisms in RL training. Notably, our CRLQAS algorithm does not rely on prior knowledge of the true ground state energy value. Instead, it employs curriculum-bicg based learning method that dynamically adapts to the problem difficulty based on the performance of the agent. This approach makes the agent self-sufficient and allows accumulate knowledge at pace determined by performance.
We first simulate all the molecules in the noiseless scenario and then in the presence of shot noise. Subsequently, we incorporate the noise profile from the IBM Mumbai device, setting the 1-, and 2-qubir depolarizing noise to (i) the median, (ii) the maximum value, and (iii) 10 times the maximum noise value. Our findings, illustrated in Fig. demonstrate the impact noise levels on quantum circuit statistics (like depth, number of gates, and parameter count) to solve the ground state energy estimation problem via VQE for H2 2, Hg - 3, Hg - 4 and LiH - 4. Our results empirically verify commonly expressed hypothesis: increase in noise levels corresponds to increase in the number of gates and parameters in the circuit (Sharma et al., 2020; Fontana et al., 2021).
Moreover, conduct comparative analysis between CRLQAS and the QCAS method (Du al., 2022). With equivalent noise settings and connectivity constraints of IBM Ourense, our findings indicate that CRLQAS efficiently determines the ground state energy of H 4, yielding - -1.136Ha

---

Energy Error
10-2
10-1
Noisy Error
Noiseless Error
Threshold
Threshold
2500 5000
7500 Episodes
10000
12500 15000
2500
5000
7500 Episodes
10000
12500 15000
Figure 3: Learning curve of the LiH (with 4-qubits) experiment. The agent is trained with noise model
with 1-qubit depolarizing noise of strength 0.1 10 and sampling of the expectation values of 10 repeti- tions. The left panel shows the training curve under noise, the right panel is the evaluation of the points the left panel but without noise. The red dashed line indicates the chemical accuracy.
in contrast to the reported minimum energy of -0.963Ha in (Du al., 2022). Detailed data in Table 2 outlines the minimum energy and the number of gates for H2-4. We also perform noiseless simulation of the circuit presented in (Du al., 2022), yielding an energy error of 1.9 x 10-2 (with 16 gates). In contrast, CRLQAS achieves significantly superior energy errors of 8 x 10-5 (with RH, 32 gates) and 1.5 10-5 (without RH, 40 gates), demonstrating improvements by three orders of magnitude. Upon closer inspection of all the successful episodes during post-processing of the trained agent's data (i.e., searching for an intermediate step where the energy error just below chemical accuracy) for H2 - 4, we observed that CRLQAS indeed achieves energy error of 2.8 10- (with RH) and 5.5 10-4 (without RH) respectively, while utilizing only gates.
In Fig. 3, we present the learning curves of the agent from our simulations for LiH 4 with 1 1-qubit depolarizing noise strength of 0.1 x 10-2 2 and 10 sampling noise. The illustration tracks two crucial values using the optimal parameters discovered in the noisy setting: the noisy and noiseless energies, obtained by evaluating their respective cost functions for those parameters. The left panel depicts that in the presence of noise, the energy error closely aligns with the threshold curve and decreases with it. This trend indicates the learning trajectory the agent, where it learns to construct circuits that minimize the noisy energy error. Conversely, in the right panel, the energy behaviour does not mimic the threshold curve for the noiseless scenario. Notably, it even passes chemical accuracy despite the threshold being well above it. This divergence suggests that minimizing the noisy energy does not necessarily result in minimizing the noiseless energy, and vice versa.
We also trained RL agents to solve the ground state energy estimation problem via VQE for LiH in two additional noisy settings. In the first setting, we employ 1-, and 2-qubit depolarizing channels of strength 0.1 1 and 0.5 10- respectively. In the second, we utilized the median noise profile of IBM Mumba i device. Unfortunately, the agent fails to achieve chemical accuracy in these noisy settings. In the first setting, the agent attained a minimum noiseless error of 3.4 10 (trained for only 5000 episodes). In the latter scenario, it reached 3.3 x 10 (trained for 15000 episodes).
4.2 NOISELESS SIMULATION
We now present the analyses for noiseless settings. This is due to the considerable challenge of scaling noisy QAS beyond four qubits within the scope this work. The exponential increase in computational cost with the number of qubits makes it exceptionally difficult to handle billions of queries to the noisy cost functions.
To analyze the scaling performance of CRLQAS method, assess its performance for H2 4. LiH - 4, LiH - 6 and H₂O 8 molecules in comparison to quantumDARTS (Wu et al., 2023) and modified variant of qubit-ADAPT-VOE (Tang et al., 2021). 1). The results are presented in Table 1. Our results demonstrate that for H2 4, LiH 4, and LiH 6, CRLQAS surpasses these other QAS methods, producing only more compact circuits but also achieving lower absolute energy
8

---

Table tabular representation of noiseless simulation -2 LiH- LiH-6, H₂O-8 molecules. We simulate them in the noiseless scenario for the modified variant of gubit-ADAPT-VOS (Tang et al., 2021). quantumDARTS (Wu et al., 2023) and compare it with our CRLQAS. (*) denotes that the simulation is done using 1-, 2- and 3-qubit parameterized gates, which can he further decomposed into {RX, RY, RV, CNO N.A. is the abbreviation for Not Applicable, implying that the algorithm failed improve over the Hartree Fock state energy.
Molecule
Modified qubtt-DAAPE-QE
CRLQAS
quantumDARTS
Energy Ervor
# Params
Depth # Gates
Energy Eroor
Params
Depth
# Gates
Energy Ertor
# Params
Depth
# Gates
H₂ 4 1.9 10
38
29
38
7.2 10
7
17
21
4.3 x 10
26
18
34
LiH 4.6 x 10-6
47
38
47
2.6 10-6
29
22
40
1.7 10
50
34
68
LiH 3.7 10
N.A.
N.A.
N.A.
6.7
10
29
40
67
2.9
10
54
132
H₂O 2.6 10-M
10
10
emrors. Furthermore, in (Ostaszewski et al., 2021b), for the LiH 6 molecule, their RL algorithm achieves chemical accuracy only in 7 out of 10 independent seeds. Conversely, utilizing CRLQAS with the same molecule, we achieve solutions across all seeds, showeasing the enhanced stability of CRLQAS in contrast to the RL. method of (Ostaszewski et al., 2021b).
should be noted that we utilized a modified qubit- VQE in our simulations for comparative analysis. Specifically, we replace the typically large fermionic pool of operators with a parameter- ized gate set consisting
RY, RZ, RZZ, RYXXY, RXXYY, Controlled RYXXY, Cont Condrolle-RR (4)
The energy errors reported in Table I were computed by simulating ADAP--ADAP in Hartree Fock state (Slater, 1951) for 100 ADAPT-VQE iterations with Adam optimizer (learning rate 0.1 10-2, and 500 optimization iterations). Notably, for LiH and H₂O 8, fails to improve over the Hartree Fock state, repetitively applying the same gate in all iterations, thus resulting in zero values for parameters, depth, and gates. Finally, we emphasize that we exempt ourselves from doing fine hyperparameter tuning of qubit-ADAPT-VOEE which might improve its performance.
5 CONCLUSION
We developed auriculuum-bsse reinforcement learning QAS (CRLQAS) algorithm, specifically optimized to tackle the unique challenges of deploying VQE in realistic noisy quantum environ- ments. Our main contribution lies in introducing CRLQAS and analyzing its performance across different quantum noise profiles sourced from real IBM quantum devices like IBM Curense and IBM Mumbai. Our method achieved the ground-state energy within the precision defined by chem- ical accuracy while suggesting circuits characterized by minimal gate counts and depths, thereby establishing state-of-the-art performance in the sub-field of quantum architecture search (QAS).
We introduced depth-aware tensor-based 3D encoding for the agent's 's state description of the quan- tum circuit, illegal actions to reduce the agent's 's search space and to avoid consecutive application of similar quantum gate, a random halting mechanism steering the agent to find shorter circuits, and novel variant of the simultaneous perturbation stochastic approximation (SPSA) algorithm to reduce the energy function evaluations in the presence of noise.
Our numerical experiments focused on quantum chemistry tasks and demonstrated that CRLQAS outperforms existing QAS algorithms across noiseless and noisy environments for H2 (2-, 3-, and 4-qubit), LiH (4-, and 6-qubit) and H₂O (8-qubit) molecule. In pursuit of these experiments, we significantly enhanced the efficiency of simulating realistic noisy quantum circuits by employing the PTM formalism in the Pauli-Liouville basis, thereby fusing gates with their respective noise models and values.
In essence, owing to the notable adaptability of our approach and a significant six-fold speed-up due to PTM formalism, approach holds potential for applications in QAS for combinatorial opti- mization, quantum machine learning, reinforcement learning for quantum computing and quantum reinforcement learning. We have outlined limitations and future work in Appendix A.

---

ACKNOWLEDGEMENTS
YJP and OD would like to thank Stefano Polla and Hao Wang for the helpful discussions. YJP and VD acknowledge support from Totalene AK would like to acknowledge support from the Polish National Science Center under the grant agreement 2019/33/B/ST6/00011 and MO would like to acknowledge support from the Polish National Science Center under the grant agreement 2020/39/B/ST6/0151 and from Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme. XBM acknowledges funding from the Quantum Soft- ware Consortium. VD and OD were supported by the Dutch Research Council (NWO/OCW), as part of the Quantum Software Consortium programmes (project number 024.003.037 and NGF. 156.2252..2.031 This work was also supported by the Dutch National Growth Fund (NGF), part of the Quantum Delta NL programme. The computational results presented here have been achieved in part using the ALICE HPC infrastructure of Leiden University and SURF Snellius HPC infrastructure (SURF Cooperative grant no. EINF-6793).
REPRODUCIBILITY
To ensure the reproducibility of work, provide detailed descriptions of the experimental configurations and hyperparameters for the CRLQAS method and Adam-SPSA optimizer in Ap- pendix C and Appendix F, respectively. Additionally, information about the noise profiles of the IBMQ device and molecular systems is available in Appendix G and Appendix L, respectively. A comprehensive discussion of the noise models examined in our study and practical aspects of their software implementation can be found in Appendix H and Appendix I. The source code for all experiments conducted in this manuscript accessible here: https: ///////nnnnnuu//////////////aas/ 4oper. science, / CRLQAS/.
REFERENCES
Gabriele Agliardi and Enrico Prati. Optimized quantum generative adversarial networks for distri- bution loading. In 2022 IEEE International Conference on Quantum Computing and Engineering (QCE), pp. 824-827. IEEE, 2022.
Panagiotis G Anastasiou, Yanzhu Chen, Nicholas Mayhall, Edwin Barnss, and Sophia E Economou. Tetris-adapt-vge: An adaptive algorithm that yields shallower, denser circuit ansätze. arXiv preprini arXiv:2209, 10562, 2022.
Yazan Arouri and Mohammad Sayyafzadeh. An accelerated gradient algorithm for well control optimization. Journal of Petroleum Science and Engineering, 190:106872, 2020.
Jeongho Bang and Seokwon Yoo. A senetic-algorihhm-baeee method to find unitary transforma- tions for any desired quantum computation and application to a one-bit oracle decision problem. Journal of the Korean Physical Society, 65:2001-2000 2014.
Ville Bergholm, Josh Izaac, Maria Schuld, Christian Gogolin, Shahnawaz Ahmed, Vishnu Ajith, M Sohaib Alam, Guillermo Alonso-Linaje, B AkashNarayanan. Ali Asadi, et al. Penny- lane: Automatic differentiation of hybrid quantum-classical computations. arXiv preprint arXiv:/8/1.00 10.4968, 2018.
M Bilkis, María Cerezo, Guillaume Verdon, Patrick J Coles, and Lukasz Cincio. A semi-agnostic ansatz with variable structure for quantum machine learning. arXiv preprint arXiv:2103.06712, 2021.
Patrick Billingsley, Probability and measure. John Wiley Sons, 2017.
Carsten Blank, Daniel K Park, June-Koo Kevin Rhee, and Francesco Petruccione. Quantum classi- fier with tailored quantum kernel. npj Quantum Information, 6(1):41, 2020.
Xavier Bonet-Monroig, Hao Wang, Diederick Vermetten, Bruno Senjean, Charles Moussa, Thomas Bäck, Vedran Dunjko, and Thomas E OrBrien. Performance comparison optimization methods variational quantum algorithms. Physical Review A, 107(3):032407, 2023.
10

---

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Millm and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //gituubb com/ google e/ jax.
Heinz-Peter Breuer and Francesco Petruccione. The theory of open quantum systems. Oxford Uni- versity Press, USA, 2002.
Chris Cade, Lana Mineh, Ashley Montanaro, and Stasja Stanisic. Strategies for solving the fermi- hubbard model on near-term quantum computers. Physical Review B, 102((3):233122, 2020.
M. Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C. Benjamin, Suguru Endo, Keisuke Fujüi, Jarrod R. McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, and et al. Variational quantum algorithms. Nature Reviews Physics, 3(9):625-64,, Aug 2021. ISSN 2522-5820. doi: 1031038/ $42254-001-00348-9 URL http: / /xxpdxdddoe doi. org/ 0 1 1038/ 22218882244222-2-03333-33
D Chivilikhin, A Samarin, V Ulyantsev, lorsh, AR Oganov, and O Kyriienko. Mog-vqe: Multiob- jective genetic variational quantum eigensolver. arXiv preprint arXiv:2007.04444. 2020.
Jerry M Chow, Jay M Gambetta, Antonio Corcoles, Seth T Merkel, John Smolin, Chad Rigetti, Poletto, George A Keefe, Mary B Rothwell, John R Rozen, et al. Universal quantum gate approaching fault-tolerant thresholds with superconducting qubits. Physical review letters, 109 (6):060501, 2012.
IBM Corporation. IBM Quantum. https: /ut.nuuuuuuuummmmm computing ibm. wtt.://oo/eeeeeeeeee resources, 2023. Accessed on August 8, 2023.
Edward Dougherty. Probability and statistics for the engineering, computing, and physical sci- ences. Prentice-Hall, Inc., 1990.
Yuxuan Du, Tao Huang, Shan You, Min-Hsiu Hsieh, and Dacheng Tao. Quantum circuit architecture search for variational quantum algorithms. npj Quantum Information, 8(1) 8(1):62, 2022.
Edward Farhi, Jeffrey Goldstone, and Sam Gutmann. A quantum approximate optimization algo- rithm. arXiv preprint arXiv: Ttfl::4441.020 1.0028, 2014.
Enrico Fontana, Nathan Fitzpatrick, David Muñoz Ramo, Ross Duncan, and Ivan Rungger. Evalu- ating the noise resilience of variational quantum algorithms. Physical Review A, 104(2):022403, 2021.
Enrico Fontana, M Cerezo, Andrew Arrasmith, Ivan Rungger, and Patrick Coles. Non-trivial symmetries quantum landscapes and their resilience to quantum noise. Quantum, 6:804, 2022.
Enrico Fontana, Manuel S Rudolph, Ross Duncan, Ivan Rungger, and Cristina Cirstoiu. Classical simulations of noisy variational quantum circuits. arXiv preprint arXiv:2306.05400 2023.
Daniel Greenbaum. Introduction quantum gate set tomography. arXiv preprint arXiv: 15000: 2015.
Harper R Grimsley, Sophia E Economou, Edwin Barnes, and Nicholas Mayhall. An adaptive variational algorithm for exact molecular simulations quantum computer. Nature comm.,, 10 (1):1-9, 2019. URL https: /ooi wwwg/.../........./..oo/ 1038/ 3 447-4467-000 9- 1-10988-2.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1948.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. Nature, 585(7825):357-362. 2020.
Pascual Jordan and Eugene Paul Wigner. Über das paulische äiuivaaeezverrbot. Springer, 1993.
Abhinav Kandala, Antonio Mezzacapo, Kristan Temme, Maika Takita, Markus Brink, Jerry M Chow, and Jay M Gambetta. sordwarre-effcicee variational quantum eigensolver for small molecules and quantum magnets. nature, 549(7671):242-22) 2017.

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Philip Krantz, Morten Kjaergaard, Fei Yan, Terry P Orlando, Simon Gustavsson, and William D Oliver. quantum engineer's guide to supereonducting qubits. Applied physics reviews, 6(2), 2019.
En-Jui Kuo, Yao-Lung L Fang, and Samuel Yen-Chi Chen. Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.077 2021.
Zhaoqi Leng, Pranav Mundada, Saeed Ghadimi, and Andrew Houck. Robust and efficient algo- rithms for nigh-dimensional black-box quantum optimization. arXiv preprint arXiv: 1rX0v31910.03511, 2019.
Gushu Li, Yufei Ding, and Yuan Xie. Tackling the qubit mapping problem for nisq-era quantum devices. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1001-1014, 2019.
Junyu Liu, Frederik Wilde, Antonio Anna Mele, Liang Jiang, and Jens Eisert. Noise can be helpful for variational quantum algorithms. arXiv preprint (rXxi:2210000723. 2022.
Manuel López-lbáñez, Jérémie Dubois-Lacoste. Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016.
Jarrod R McClean, Jonathan Romero, Ryan Babbush, and Alán Aspuru-Guzik. The theory of variational hybrid quantum-classical algorithms. New J. Phys., 18(2):023033, 2016. URL
2////////22022333 https: /ttps://ddoi.oo/ org/10. 2111////116-223011/2222
Jarrod R McClean, Nicholas C Rubin, Kevin J Sung, Ian Kivlichan, Xavier Bonet-Mouroig, Yudong Cao, Chengyu Dai, E Schuyler Fried, Craig Gidney, Brendan Gimby, et al. Openfermion: the electronic structure package for quantum computers. Quantum Science and Technology, 5(3): 034014, 2020.
Michael A Nielsen and Isaac Chuang. Quantum computation and quantum information. Cam- bridge university press, 2010.
Román Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Annals of physics,
2014.
Mateusz Ostaszewski, Edward Grant, and Marcello Benedetti. Structure optimization for parame- terized quantum circuits. Quantum, 5:391, 2021a.
Mateusz Ostaszewski, Lea M Trenkwalder, Wojeiech Masarczyk, Eleanor Scerri, and Vedran Dun- jko. Reinforcement learning for optimization of variational quantum eircuit architectures. Ad- vances in Neural Information Processing Systems, 34:18182-188888 2021b.
Thomas E O' O'Brien, B Tarasinski, and Leo DiCarlo. Density-matrix simulation of small surface codes under current and projected experimental noise. npj Quantum Information, 3(1):39, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter Love, Alán Aspuru-Guziik, and Jeremy I. O' O'brien. A variational eigenvalue solver on photonic quan- tum processor. Nature communications, 5(1):4213, 2014.
Václav Potošek, Alan P Reynolds, Alessandro Fedrizzi, and David W Come. Multi-objective evo- lutionary algorithms for quantum circuit discovery. arXiv preprint arXiv:1812.04445. 2018.
Michael JD Powell. A direct search optimization method that models the objective and constraint functions by linear interpolation. Springer, 1994.
12

---

Matteo G Pozzi, Steven J Herbert, Akash Sengupta, and Robert D Mullins. Using reinforcement learning to perform qubit routing in quantum compilers. ACM Transactions on Quantum Com- puting, 3(2):1-25, 2022.
John Preskill. Quantum computing the nisq era and beyond. Quantum, 2:79, 2018.
Patrick Rall, Daniel Liang, Jeremy Cook, and William Kretschmer. Simulation of qubit quantum circuits via pauli propagation. Physical Review A, 99(6):062337, 2019.
Arthur G Rattew, Shaohan Hu, Marco Pistoia, Richard Chen, and Steve Wood. A domain-agnostic, noise-resistant, hardware-efficient evolutionary variational quantum eigensolver. arXiv preprint 1rXiv:1910.09694, 2019.
Florian Rehm, Sofia Vallecorsa, Kerstin Borras, Dirk Krücker, Michele Grossi, and Valle Varo. Precise image generation on current noisy quantum computing devices. Quantum Science and Technology, 9(1:005509,, 2023.
Manuel S Rudolph, Enrico Fontana, Zoë Holmes, and Lukasz Cincio. Classical surrogate simulation of quantum systems with lowesa. arXiv preprint arXiv:2308.09109, 2023.
Amit Sabne. Xla: Compiling machine learning for peak performance. 2020.
Jacob T Secley, Martin J Richard, and Peter J Love. The bravyi-kitaev transformation for quantum computation of electronic structure. The Journal of chemical physics, 137(22), 2012.
Kunal Sharma, Sumeet Khatri, Marco Cerezo, and Patrick J Coles. Noise resilience variational quantum compiling. New Journal of Physics, 22(4):043006, 2020.
Animesh Sinha, Utkarsh Azad, and Harjinder Singh. Qubit routing using graph neural network aided monte carlo tree search. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 9935-9943, 2022.
John C Slater. A simplification of the hartree-fock method. Physical review, 81(3):385, 1951.
James C Spall. A stochastic approximation technique for generating maximum likelihood parameter estimates. In 1987 American control conference, pp. 1161-1167. IEEE, 1987.
Richard S Sutton and Andrew G Barto. Reinforcement learning An introduction. MIT press, 2018.
Yasunari Suzuki, Yoshiaki Kawase, Yuya Masumura, Yuria Hiraga, Masahiro Nakadai, Jiabao Chen, Ken M Nakanishi, Kosuke Mitarai, Ryosuke Imai, Shiro Tamiya, et al. Qulacs: fast and versatile quantum circuit simulator for research purpose. Quantum, 5:559, 2021.
Ho Lun Tang. VO Shkolnikov, George S Barron, Harper R Grimsley, Nicholas J Mayhall, Ed- win Barnes, and Sophia Economou. qubit-adapt-vqe: An adaptive algorithm for constructing hardware-efficient ansätze a quantum processor. PRX Quantum, 2(2):020310, 2021. URL
fttps::/doot . /tlgg/10.. 1.0 /twp///Ptttu.....//// 2. 020310.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: funda- mental algorithms for scientific computing in python. Nature methods, 17(3):261-22) -222, 2020.
Hanrui Wang, Yongshan Ding, Jiaqi Gu, Yujun Lin, David Z Pan, Frederic Chong, and Song Han. Quantumnas: adise-adaptive search for robust quantum circuits. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 692-708 IEEE, 2022.
Samson Wang, Enrico Fontana, Marco Cerezo, Kunal Sharma, Akira Sone, Lukasz Cincio, and Patrick J Coles. Noise-induced barren plateaus in variational quantum algorithms. Nature com- munications, 12(1):6961, 2021.
Colin P Williams and Alexander G Gray. Automated design of quantum circuits. In NASA In- ternational Conference on Quantum Computing and Quantum Communications, pp. 113-125. Springer, 1998.
13

---

Christopher J Wood, Jacob D Biamonte, and David G Cory. Tensor networks and graphical calculus for open quantum systems. arXiv preprint arXiv:1111.6950 2011.
Wenjie Wu, Ge Yan, Xudong Lu, Kaisen Pan, and Junchi Yan. Quantumdarts: Differentiable quan- tum architecture search for variational quantum algorithms. International Conference on Machine Learning, 40, 2023.
Shi-Xin Zhang, Chang- Yu Hsieh, Shengyu Zhang, and Hong Yao. Differentiable quantum architec- ture search. Quantum Science and Technology, 7(4))045023, 2022.
14

---

A LIMITATIONS AND FUTURE WORK
Computational Demands The training process for the agent is computationally demanding, pos- ing challenges both in terms of evaluating quantum circuits quantum computer and training the algorithm on classical devices. This limitation warrants further exploration for more efficient computational strategies.
Evolution of RL Methods Reinforcement learning (RL) methods continually evolving, and while promising, they face challenges related to sample efficiency, stability, and sensitivity. Rec- ognizing these evolving aspects crucial for refining the proposed algorithm and addressing its limitations.
Validation on Real Quantum Hardware A limitation of this work is the absence of validation on real quantum hardware due to current cost constraints. Future research should include experimenta- tion on practical quantum devices to assess the algorithm's performance in real-world scenarios.
Scalability Challenges The proposed algorithm's scalability to larger quantum circuits, more complex quantum chemistry problems, or different noise models is a potential limitation that re- quires thorough investigation. Current experiments train the agent from scratch, necessitating ploration for scalability improvements.
Transfer Learning Exploration Investigate the feasibility of transfer learning for the proposed algorithm, particularly in the context of different molecules and various noise scenarios. This ex- ploration aims to enhance the algorithm's adaptability and generalization across diverse quantum tasks.
Application Scenarios Enhancement Explore more applicable scenarios, such as pre-training the algorithm simulations and fine-tuning real quantum devices. This approach can potentially improve the algorithm's 's efficiency and performance in practical quantum computing applications.
B DESCRIPTION OF CRLQAS COMPONENTS
B. FEEDBACK-DRIVE CURRICULUM LEARNING
The moving threshold technique (see Fig. is a feedback-driven curriculum learning method in- troduced in (Ostaszewski et al., 2021b). During the learning process the agent pursues a parameter & that marks the lowest energy known by the agent so far and updates threshold parameter with respect to this parameter based on some rules. In the beginning, the &a parameter is set to hyper- parameter Ş If the agent finds an energy value lower than the current one, updates &2 to this new energy value. Another hyperparameter "fake minimum energy" μ., proxy to the lower bound of attainable ground state energy is set as a target for the agent'. We compute this proxy by taking the summation of absolute values of Pauli string coefficients stemming from the Hamiltonian. In the absence of amortization, the algorithm shifts the threshold to //4 ε2| for the new Ez. In the pres- ence of amortization, however, it adds parameter to that threshold as | 1 Eal + 5, where å the amortization hyperparameter. In the meantime, the agent continues its exploration with subsequent actions and episodes and records the number of successful actions. Here, there are two rules play. The first rule greedily shifts the threshold to | E2l after G episodes. Here G is hyperparameter as well. The second rule slowly decreases the threshold parameter each time there successful episode by subtracting factor of 8 Here k is the radius of shifts, also hyperparameter. Upon setting the threshold to (2 if the agent fails to improve the energy value in consecutive episodes the threshold is increased back to μμ - &2l + 8, as demonstrated in Fig. 4. This way the agent is given an opportunity to trace its steps back it was stuck a local minimum.
Notably, this method does not require any prior knowledge regarding the true value of the ground state energy and does not impose any specific constraints on the initial threshold value, unlike exist- ing QAS methods in the literature.
One can set the target of the agent to such value for VQE because from Rayleigh's 's variational principle, the agent theoretically can never attain an energy below the true ground state energy.
15

---

AEA
Episodes
Figure 4: Demonstration of the foedback-driven (green) process, depicting the impact of amortization occurrences (pink), denoted by d. The initial occurrence corresponds a non-zero adjustment in the threshold, transitioning from ςι to Êz. indicating the agent's success in enhancing the energy estimate during training. The subsequent amortization event illustrates the scenario where the agent falls short of improving upon the current threshold &2 or the improvement is marginal compared to the amortization value. Consequently, the threshold undergoes a sudden increase due to the reset of the amortization value. It's noteworthy that the ultimate threshold, subsequent to the second amortization reaching zero, may also less than {2.
B.2 ILLEGAL ACTIONS FOR RL AGENT
The illegal actions (IA) scheme is an adaptation we developed in this work to prevent the RL agent from choosing actions that either revert or add redundancy to the effect of the previous actions. In our context, each action involves appending gate to qubit wire within the circuit. This scheme relies on two heuristics.
The first heuristic centers on the nature of unitary matrices within quantum gates. When adding a unitary (gate) to qubit wire at a certain circuit moment (layer), appending the same unitary to the same wire in the subsequent moment effectively negates the former's effect. This multiplication results in an identity matrix or an idling operation. Our CRI -AAS algorithm is designed to progress forward, consistently increasing the total number of gates in the circuit by appending gates. It does not retract or prune gates dynamically. To restrict redundant idling operations and enhance the RL agent's is exploration efficiency, the first rule of IA prohibits adding CNOT gate specific wires if the same gate was appended to those wires in the previous layer.
The second heuristic focuses 1-qubit rotations. When optimizing the parameter of 1-qubit rotation gate to a certain value 0" at a given moment, appending the same rotation gate to the same wire in the next moment introduces redundancy from an optimization perspective. As CRLQAS continuously optimizes at each step, subsequent rotation gates with the same angle will yield redundant values. Thus, the second rule IA prevents adding a 1-qubit rotation gate (e.g., RX, RY, R2) if the same gate was appended to that wire in the previous layer. The RL agent must remain informed about the disallowed actions via subroutine when it is about to choose an action (gate). Below, we provide implementation details for this IA subroutine.
When the agent determining its next action, the subroutine scans the diree-dimensional tensor representing the circuit to identify the previously added gates. It then translates this information into action numbers based on the number of qubits N, presenting RL agent. An example of such list can be exemplified as the following.
format understandable by the
Alleaal [CNOT (i, (i,jN) N), RX (k, N)...
(5)
Here i and j denote the ctrl and targ qubit wires of CNOT gate for N qubits, and the & denotes the wire where RX gate was appended. For example, when N 4, 0, and A 2 the list takes the following form.
(6) Since the first N actions are reserved for three 1-qubit rotation directions acting on N qubits in our numbering convention, the action number for RX (2, 4) is 6 in the Eq. 6. Similarly, we use a ctrl major numbering convention, the action number for the CNOT (0, 1, 4) gate given as 12 as the first action after array of 1-qubit rotation gate actions. During the selection of the next action, the RL agent updates the Q-table of Q-values corresponding to these action numbers based on the current RL state (quantum circuit) by utilizing a DQN. Without the IA scheme, the agent would typically choose the action with the highest Q-value. However, with the IA scheme, the agent identifies illegal actions using the subroutine's 's provided list and updates their Q-values to po in the
Ankgall [CNOT (0, 1,4) RX (2,4) + [1..
16

---

Q-table. Consequently, when the agent selects actions based on the highest Q-values, those actions with -oo Q-values (i.e., illegal actions) are effectively discarded.
B.3 ILLUSTRATION OF TENSOR BASED ENCODING
0 0
0
0
CX
0
1
RX{
RY
0
1
RZ <
1
0
0
1
depth 1
depth 2
depth 3
10)
RZ((1)
(
RZ((7)
10)
(x2)
RY(0s)
10)
(Y,)
10)
(P,)
RZ((%)
Figure 5: Illustration of tensor-based encoding for a 4-qubit (Le.,, N = 4) toy circuit with naxt 3. We initialize tensor of zeros of dimension [Ta X ((N + 3) x N)), equating to [3 x ((4 3) x 4)] for this circuit. Each blue-colored matrix of size ((4 3) 4), represents different moment (depth). Within this matrix, the first (4 x 4) block is reserved qubits, respectively. The remaining (3 x 4) block of the blue-colored matrix then encodes rotation gates. The columns mark the position of the qubit wire (index) and the rows mark the rotation direction m. Here, 1, and yields the rotations RX, RY and RZ, respectively.
CNOT (CX), with rows and columns encoding target and control
C CRLQAS IMPLEMENTATION & HYPERPARAMETERS
In our experiments, we employed the Double deep-Q network (DDQN) algorithm with variable step sizes in the n-step trajectory roll-out update (Sutton & Barto, 2018). For noiseless experiments involving simple molecules with low qubit numbers, such as H2 2, H₂ - 3, H₂ and LiH 4, we used single step, = 1. For noisy simulations of simple molecules, such as H2 2, H2 with the IBM Mumbai noise model, and H2 with the IBM Ourense noise model, we used 5 steps. Conversely, for noiseless simulations of complex molecules LiH 6 and H₂O 8, and for the relatively challenging task of simulating LiH with the IBM Mumbai noise model, steps.
employed
In these configurations, we set the discount factor (1) to 0.88. We implemented an e-greedy policy for selecting random actions, with decaying by factor of 0.99995 per step from initial value of c = until it reached a minimum value of e = 0.05. The memory replay buffer size was fixed at 20000, and the target network in the DQN training process² was updated after every 500 actions. In the curriculum learning strategy, we implemented a testing phase after every 100 training episodes. In this testing phase we set the randomness factor ensure set of deterministic actions. We exclude the experiences acquired in this phase from the memory replay buffer. We greedily adjusted the threshold after G 500 episodes for both noiseless and noisy 2-, 3-, and 4-qubit problems. Conversely. for 6-, and 8-qubit problems, the threshold underwent adjustments every G 2000 episodes, with an amortization radius set at = 0.0001. This amortization radius decreased by å/K 0.00001 after every 50 successfully solved episodes, beginning from an initial threshold value of & = 0.005.
halt the random exploration, and
We conducted simulations of noiseless quantum circuits using the Qulacs library CPU (Suzuki et al., 2021). . For the simulations of noisy quantum circuits, we utilized the JAX library on two com- puting clusters equipped with NVIDIA-A100 GPUs (Bradbury et al., 2018). In experiments without
Thhe neural network trained using Adam optimizer (Kingma & Ba, 2014).
17

---

finite sampling (shot) noise, employed the gradient-free COBYLA optimizer (Powell, 1994) with default hyperparameter settings from Scipy (Virtanen et al., 2020) and 1000 iterations to optimize circuit parameters at each step of the RL episode. In the presence of shot noise, we employed the m-stage Adam-SPSA (where m is an integer) developed this work. Specifically, we used m = stage Adam-SPSA for two and three qubit problems, and Unlike the local optimization approach for circuit parameters, where only the angles of the latest ap- pended parametrized gate are optimized, we adopted global optimization approach (Ostaszewski et al., 2021b). In each step, we used the circuit parameters from the previous step as initial values, but we optimized all the parameters.
3 stages for four qubit problems.
The hyperparameters of the Double deep-Q network algorithm were selected through coarse grain search. The employed network architecture consisted of fully connected network with hidden layers, each with 1000 neurons for the noiseless 4-qubit case, 2000 neurons for the 6-qubit case, and 5000 neurons for the 8-qubit case. In the noisy case, however, we employed 1000 neurons at each layer for the 2-, and 3-qubit problems, while the number of neurons per layer ranged upto 2000 neurons for the 4-qubit problem depending on the complexity of the problem. Simulating l-qubit depolarizing noise required only 1000 neurons per layer, but the noise model of IBM-Mumbai devices required up to 2000 neurons for the 4-qubit problem. Similarly, in the noiseless case, we capped the maximum number of gates at 40 for 4-qubit problems, 70 for 6-qubit problem, and 250 for 8-qubit problem. the noisy case, we capped the maximum number gates at 40 for 2-, and 3-qubit problems, but ranged up to 60 gates for the 4-qubit problems depending on the complexity of the noise model.
D COMPARISON WITH QCAS (DU ET AL., 2022)
Table 2: Comparison summary between CRLQAS and QCAS for the 4-qub H2 molecule. The average taken over 3 seeds. The bold numbers highlight the optimal performance of the CRLQAS algorithm. For both CRIQAS (RH) and CRLQAS (wo-RH), the first row represents the minimum achievable error and corresponding number of gates required for these settings. Additionally, the second row demonstrates the minimum number of gates needed to achieve errors just below the chemical accuracy threshold. Notably, using only 10 gates in both settings allows for better accuracy the target. Each approach, RH and wo-RH, presents its unique advantages and disadvantages. The wo-RH setting achieves energy estimates significantly below chemical accuracy, Conversely, using RH yields slightly less accurate energy estimates but with shallower circuits.
Method
Minimum Noiseless Err. Avg. Noiseless Minimum Err. Gates
CRLQAS (RH)
x 10
1.7 x 10-4
32
2.8 x 10
10
CRLQAS (wo-RH)
1.5 x 10-5
8.7 10
40
5.5 10-5
10
QCAS (Du et al., 2022)
1.9 x 10
18

---

E ABLATION STUDY OF DIFFERENT COMPONENTS OF CRLQAS
Table 3: Results of ablation study for CRIQAS method. We conducted thorough investigation identify the features that standout within the CRIQAS method in both noisy (for 4-qubit LiH and H2) and noiseless (for 6-qubit LiH) settings. Initially, we find episode(s) with the best noiseless error(s), gathering circuit statistics (depth, gate count, etc.) and then acquire the noisy error(s) for these episode(s). The last two columns provide insights into the learning performance of the CRLQAS method. We present median statistics over three random seeds for noisy experiments and five seeds for noiseless settings. Notably, we emphasize noiseless errors (in both noisy and noiseless settings) achieving chemical accuracy and denote in bold. wo-X denotes the dcactivation of feature X, where X E {IA, RH}. Additionally, N.A. denotes Not Applicable, implying that none of the agents over three seeds achieved chemical accuracy.
Molecule
Environment
Noisy Err.
Naiselesx Err. Depth
Gates # Params
Act. to 1st Succ. Ep. Suce. Ep.
Lift 4
(wo wo REI, Numboi Median)
0.0581
0.0026
24
40
33
N.A.
N.A.
Lift- 4
(IA, wo-RII, Numboi Median)
0.0818
0.0015
23
40
30
66221
I
LiH 4
(wo-IA, RH, Mumbai Median)
0.0535
0.0029
15
29
21
N.A.
N.A.
LIH 4
(IA, RH. Xumbai Median)
0.0670
0.0024
15
29
23
N.A.
N.A.
LIH 4
(IA. RH. Numbei Median & shot noise)
0.0735
0.0038
17
28
18
N.A.
N.A.
Ha-
(IA, wo-RH, Ourense)
0,2382
0.0001
26
14
59311
1209
H3-
(wo-IA, wo-RH, Ourense)
0.3118
0,0004
29
58157
10
Ha-
(IA, RH. Ourense)
0.1372
0,0002
27
42331
217
(wu-IA, RII, Ourense)
0.1522
0.0003
35593
6
LiH
([Ostaszewski et al., 202(b), noiseless)
0.0024
N.A.
N.A.
LiH
(IA, wo-RH, noiseless)
0.0008
356604
30
LiH
(IA, RH. noiseless)
0.0012
30
641593
3
LiHl- 6
(wo-IA, RH. nuiseless)
0.0037
23
45
N.A.
N.A.
Lill 6
(wo-IA, wo-RHI noiseless)
0.0016
31
62
N.A.
N.A.
F SPECIFICS FOR ADAM-SPSA
We implement a version of simultaneous perturbation stochastic approximation (SPSA) that was not implemented in the context of VQE before. It combines stochastic gradient estimates of SPSA (Spall, 1987) with adaptive moment estimation (Adam) (Kingma & Ba, 2014) leading to more stability and faster convergence while retaining the noise robustness of SPSA. To estimate gradient term at a given iteration, the SPSA algorithm randomly samples a number of angles amount of bi- nary directions from the Rademacher distribution, denoted by Ag. By adding and subtracting these shifts from the current set of angles 0= = 0 + Ak. we acquire two sets of angles such that we eval- uate the cost function twice (two function evaluations per iteration) there to acquire the stochastic
gradient approximation J - Then the algorithm proceeds similarly to standard gradient descent 0++1 0g a Jk, with the exception of both parameter shift scaling parameters eg and the learning rates a decay at each iteration l slowly with respect to hyperparameters a, c, Ysp-
Ar
Similar to incorporating adaptive moment terms (parameter adaptation and momentum) in gradient descent optimization, we integrate these terms with the stochastic gradient estimate from SPSA. In this integration, three additional hyperparameters, namely 31, 92, and X, are introduced to gov- ern the adaptation and momentum terms. In the quantum context, Adam enables the utilization of gradient information from previous iterations in classical post-processing without relying on addi- tional quantum queries, thereby enhancing robustness and convergence rates. After testing multiple variants of SPSA in a well-known problem of VQE through hardware efficient ansatz (Agliardi & Prati, 2022; Arouri & Sayyafzadeh, 2020; Kandala al.. 2017), we found the variant used in (Leng et al., 2019) to be the most stable. Throughout this work, we refer to this specific variant as Adam- SPSA (Leng al., 2019). In this version of Adam-SPSA, the momentum term 3n undergoes updates
19

---

Table 4: The hyperparameters of Adam-SPSA optimizer used during the noisy simulations. In the noisy simulation of 2-, and 3-qubit problems we used 1-stage version of the algorithm. therefore only single maximum function evaluation hyperparameters are given. The parameters within the curly brackets denote the maximum number of function evaluations in the 3-stage version of the algorithm. We provide Max fevals both for 1 I-stage and their 3-stage equivalents.
Molecule
B1
32
7sp
X
Max fevals
Shots
H2-2
1.2104
0.9531
0.9414
0.9983
0.1039
0.0984
0.9277
500
10a
H2-3
0.5188
0.9859
0.716
0.6265
0.0938
0.0974
0.6483
500
10a
LiH-4
1.2324
0.9709
0.6114
0.9326
0.2215
0.1485
0.9772
1600
3300
10c
(1191,
119] (2383, 715, 238)
LiH-6
1.7564
0.8365
0.6841
0.9048
0.1068 0.1549 0.1223
2000
10a
(1430, 429, 143)
at each iteration using the À scaling hyperparameter, ensuring numerical stability. Conversely, the other momentum term B2 remains constant. The pseudo-code for Adam-SPSA is outlined in Alg. 12, with the newly introduced Adam momentum components highlighted in green.
In the presence of finite sampling (or shot) noise, keeping the number of measurement samples, N shotss constant throughout the training is named 1-stage optimization (Cade et al., 2020; Bonet- Monroig et al., 2023). Similarly, 3-stage version of vanilla SPSA is proposed in (Cade et al., 2020; Bonet-Monroig et al., 2023) such that the number of measurement samples is increased at each phase. In the 3-stage optimizers, the first phase is implemented with shot budget of (-(())
shote
Nshots/ / 10 for a function evaluation budget of ni the second with a shot budget of (2
shots
Nahots for function evaluation budget of n and the third with shot budget of (v)
shots
10Nshots for function evaluation budget of n The 3-stage SPSA algorithm introduced in these
papers resets the decaying hyperparameters to their default values during transitions between stages.
In this work, we propose 3-stage Adam-SPSA where the decaying hyperparameters are continu- ously evolving (i.e.. not reset to defaults) while changing between stages such that the momentum of the iterations from higher measurement samples can be utilized in the later stages. Our proposed algorithm also reports the latest function evaluation as the best function evaluation, unlike the others. After examining various versions of SPSA, considering factors like the inclusion of Adam, and ex- perimenting with or without parameter reset, we observed that the SPSA variants without parameter reset are constrained to use the best function evaluation because they significantly diverge from the solution after a certain number of iterations. Our simulations empirically show that the continuity of hyperparameters between stages leads the algorithm to converge towards the optimum while Adam increases the rate of such convergence.
We conducted experiments involving various variants of 1-, and 3-stage SPSA, both with and with- out Adam, and with and without parameter reset. The evaluation was performed 2-, 3-, 4-, and 6-qubit systems utilizing the VQE approach with a hardware-efficient ansatz of depth 10. For the 2-qubit problem addressing H2 2, we employ 10x samples. Similarly, the 3-qubit problem targeting H2 - 3 utilizes 10a samples, and the 4-qubit problem for LiH-4 employs 10t samples. Lastly, the 6-qubit problem dealing with LiH-6 utilizes 10 samples in each function evaluation. We fine-tuned mune-t the hyperparameters of the SPSA variants for these problems using an evolution- ary algorithm-batt based stochastic hyperparameter optimizer library called IRACE in R (López- Ibáñez et al., 2016). The final hyperparameters used in the experiments are outlined in Table 4. Specifically, we tuned the hyperparameters within the following ranges for each hyperparameter: a E [0.01, 2). E [0, 1]. e E [0.01, 2]. Tep E [0, 1/6], À E [0.01, 0.999], and 31, 82 E [0.6, 0.999]. For running the IRACE algorithm, we allocated 2500 evaluations of the hyperparameters as the total budget. We implemented a single run of IRACE using this budget, and utilized the F-test to eliminate worse configurations.
After the hyperparameter optimization, we executed each combination (for each SPSA variant and each problem) in 100 independent runs for the qubit systems described above. Both Fig. 6 and Fig. illustrate the optimization traces of SPSA variants with 1-, and 3-stage sampling strategy, respec-
20

---

Algorithm 1: Simultaneous Perturbation Stochastic Approximation with Adam (Adam-SPSA)
Input
Initial parameter vector 0, Objective function f(0), Number of iterations K Optimal parameter vector 0*
Output Hyperparameters: a, Ot, C, Nssp. X, f1, A2
Initialize momentum parameters to zero: m, U. <
for k = 1 to K do
Compute the scaling parameters ak + (1+11
Ck f (1+1)10
Compute the hyperparameter 31.k (1++11
Randomly choose perturbation vector A with elements +1
Evaluate objective function gradients: 9t. + f(0 cx and 9h. f(t Ck Ai)
Compute gradient estimate: VJ&+ - 51 -k
20
Biased update of moment parameters and m t f1.m + (1 k1,k) V1 Jk.
4- 32 + (1 Jx))
Unbiased computation of moment parameters and in -- 1-91,6 6 +
Update gradient estimate: JJ f- vi
10
Ve+k
11
Update parameters: +- 0 - a₁ J.k.
12
return
tively. The colors show the results of the vanilla SPSA (brown and black) and Adam-SPSA (orange and grey). The thick lines (black and grey) on top of individual optimization traces indicates the median of 100 independent runs, The markers (brown and orange) refer the final candidates of
Final candidates of SPSA
SPSA median
Final candidates of Adam-SPSA
Adam-SPSA median trace
10a
Hjr2
Energy Error
10
10-1
10
10
50
100
150
200
250
50
100
150
200
250
109
10a
Energy Error
10
10
10
10"2_
200
400
600
800
200
400
600
800
1000
Number of iterations
Number of Iterations
Figure 6: Optimization traces of the one-stage sampling strategy SPSA (brown and black) and Adam- SPSA (orange and grey) on the 2-, 3-qubit H2 and 4-, 6-qubit LiH molecules (whitesmoke text-box), using the hyperparameters outlined in Table 4. The individual traces represented by thin lines, while the thick line on top indicates the median 100 independent runs.
21

---

Reset Parameters
Continuous Parameters
10a
10a
Hz-2
Hy-2
10
Energy Error
10T1
10
10-3
Anall candidates of SPSA
Anall candidates of Adam-SPSA
10
median trace Adam-SPSA median
10
50
100
150
200
250
sb
100
150
200
250
My-3
10a
109
Energy Error
10-1
10
200
250
100
200
250
10
10a
LI-4
LIH
Energy Error
10
10-2
2
10-3
200
400
600
8do
200
400
600
800
109
100
6
Energy Error
10-2
10-2
10-2
200
400
600
800
1000
200
400
600
800
1000
Number of Iterations
Number of Iterations
Figure 7: Optimization traces of the three-stage sampling strategy of SPSA (brown and black) and Adam-SPSA (orange and grey) the 2-, 3-qubit H2 and 4-, 6-qubit LiH molecules (whitesmoke text- box), using the hyperparameters outlined Table while the thick line on top indicates the median 100 independent runs. The left and right panels showcase the resetting and continuous evolution of SPSA (Adam-SPSA) hyperparameters, respectively.
individual traces are represented by thin lines,
SPSA (Adam-SPSA) after every optimization The error bars are one-sigma standard-erro of 100 independent runs. The y-axis in both the figures is given in loglscale. In Fig. 7, the left and right panels showcase the resetting and continuous evolution of 3-stage SPSA (ddm--PSS) hyperpa-
22

---

rameters, respectively. Note that the number of iterations (maximum function evaluations) outlined in Table are different than the hyperparameters generated for this systematic benchmarking study.
In both 1-, and 3-stage Adam-SPSA, unlike the vanilla SPSA without Adam momentum, the con- vergence towards the minima is qualitatively much faster. In 3-stage variants, as seen in Fig. 7, this convergence behaviour is also apparent quantitatively (especially in LiH 4 and LiH 6), though the differences are marginal and inconclusive for the 1-stage in Fig. (except for LiH - until 700 iterations). Moreover, for 3-stage variants, we observe that the variant with resetting of hyperparameters (left panel in Fig. 7) tend to diverge from the optimal cost function values after certain number of iterations only to converge back the optima after training for longer. We noted this convergence behavior is at odds with the fast convergence rates we are looking for in our RL training- In contrast, the variant with continuous evolution of hyperparameters (right panel in Fig. 7) do not suffer from this draw back as the cost function values consistently went lower with the number of iterations without making a sharp turn away from the optima. Utilizing these insights from our analysis of various SPSA variants, we employ 1- and 3-stage Adam-SPSA (with continu- ous evolution of hyperparameters) in our noisy experiments. This helps cut down the total number of function evaluations by half, thereby doubling the speed of our RL training. This improvement at the algorithm level helped us simulate noisy systems that suffer from computational complexity and large run times.
G IBM MUMBAT DEVICE NOISE CALIBRATION DATA
Table 5: Tabular representation of the median, max and 10xmax noise of IBM Mumbai device. Additionally, the qubit frequency and the anharmonicity are the same for median, and 10xmax settings and set 4.896 GHz and -0.33 GHz, respectively.
1-Qubit Dep. Noise
2-Qubit Dep. Readout Error Noise
Thermal Relaxation Noise (us)
1-Qubit Gate Time (s)
2-Qubit Gate Time (s)
Noise Profile
Ti = 122. 28,s.
Median
2.44 x 10-4 8.25 x 10-3 3 2.25 10-2
35 10-9
116 x 10
T3 167.2
Ti 122.28pss,
Max
1.45 x 10-3 2.30 x 10
x.7 10-2
35 10-9
739.55 10-6
T3 = 167 2us
Ti 122.28pus
10xMax
1.45 10-2 2.30 x 10
8.7 10-1
35 10"a
739.55 10-
T: = 167.2us
H DETAILS OF QUANTUM NOISE MODELS
We implemented multiple noise models of varying complexity to serve as a testbed for evaluat- ing the proposed CRLQAS method. Initially, we modeled sampling (or shot) noise as indepen- dently and identically distributed random variables sampled from Gaussian distribution with zero mean MBonet-Monroig et al., 2023; Liu et al., 2022). Secondly, we modeled gate infidelities of 1- and 2-qubit gates as 1-, and 2-qubit depolarizing channels (Nielsen & Chuang, 2010). Thirdly, we incorporated two physical models for state preparation and measurement (SPAM) noise. State preparation errors were modeled by considering initial states as thermal states due to the residual thermal population of transmons. Read-out errors were modeled bit-flip channels applied at the end of the circuit. Lastly, we modeled thermal relaxation and decoherence using the thermal relax- ation channel, especially when relaxation times Ti were smaller than the coherence time T2 (Blank et al., 2020).
We manually specified parameters for the first two noise models. Subsequently, we obtained noise parameters from the benchmarks of the IBM Mumbai device on August 8, 2023, to create more realistic noise models. By obtaining Kraus matrices for the gates and noise channels, we computed Pauli-transfer matrices (PTMs) offline, eliminating the need for online computations during exper- iments. We combined the PTMs of gate implementations with those of subsequent noise channels, effectively obtaining noisy gate PTTMs
H. SAMPLING NOISE
The VQE cost function, denoted as C(0) in Eq. 1, can be expressed in terms of random variables X,
P. In this scenario, with given parameters 0 and the observable (chemical Hamiltonian)
23

---

H = CiP, then the cost represents the mean of the sum of n such random variables:
((0) = - (X,) - ((P))
1=1
In real quantum devices, users have access to a finite sample estimator. The expectation value of each Pauli string is estimated P. through multiple state preparations, basis transformations, and measurements. For each instance, the initial state is reset, and the quantum state undergoes a basis transformation into the computational basis of the given Pauli string to compute bit-string sample. These bit-strings are sampled M times (shots) to estimate the eigenvalues P. with a variance of Var(P:) = 1/M. Using these estimates X; e P, and given parameters and shots per observable M, the estimator for the cost function is as follows:
12
C(O, () = Σ X,
X Σ c.P
(8)
Here, the difference between each Pauli-string estimate and the expectation value is denoted by random variable (MM = P; - (P:), drawn from the binomial distribution with variance Var(P; (Bonet-Monroig al., 2023). Assuming Pauli string observables are measured independently, they can be modeled as independent and identically distributed (i.i.d.) random variables (Bonet-Monroig et al., 2023). In that case, the variance of C(0, M) can be propagated directly as follows:
2 = Var [C] = Σ c
M
According to the Central Limit Theorem, in the limit n + oo, the difference between the true cost function and its finit-ssamplebbsee estimator converges normal distribution centered around zero mean (Billingsley, 2017):
lim Σ (P.
lim c.((MM ~ ~N(0 sn)
(10)
This expression illustrates that, in the limit where the chemical Hamiltonian has numerous Pauli- string terms (n >> 1), sampling Pauli-string estimation errors from the normal distribution e(M) (O0, v/r ) is reasonable approximation compared to the binomial distribution. This approxima- tion serves as computationally efficient model for sampling (shot) noise and is consistent with other literature (Liu et al., 2022).
H.2 PAULI-TRANSFER MATRICES
density matrix p, a complex-valle -valued object of dimension 2 x 2 for N qubits, can represent both the quantum statistics of a single quantum state (b) (Pl (known as pure state) and the statistics of classical ensemble of multiple quantum states = Σ. P: (ws) (wsll (known as mixed state). In the case of mixed states, each constituent quantum state (1%) occurs with probability p., which sums up to unity. This happens because coupling with external processes like measurement or thermal relaxation, applies (non)unitary process to the quantum state with some probability (Breuer & Petruccione, 2002). Both the unitary processes, normally acting in closed quantum systems, and the above-mentioned processes be represented by quantum channels A. These channels are completely positive trace-preservins
(CPTP) operators in the N dimensional Hilbert space, mapping density matrix to another, A C2 x2" + C22 x2* A quantum channel acting on p is conventionally represented using Kraus matrices K₃:
A ( _KKpk] Kipk!
(11)
The Kraus matrix usually has the form K; VPiA, In the case of unitary channel (e.g.. quantum gate in a closed system), only a single unitary matrix A, is applied with a unit probability (p, 1), keeping the quantum state pure. In open, noisy systems, (non)unitary operations A, are applied with probabilities p; < 1, resulting in a classical mixture of possible outcomes (i.e., a mixed state). In digital quantum computers, physical noise is typically represented by applying the unitary
24

---

channel of gate on qubit, followed by the application of various quantum channels representing gate noise on the same qubit. It is computationally advantageous to represent each of these channel applications matrix product, requiring vectorization of the density matrix through algebraic manipulation.
P Σ 1
Pa.
(0) G - lp) = = Σ Pis (2) 00 15)
(12)
Here, representing the computational basis states given in outer-product format (t) (j] instead of the tensor product (i) 69 [j), allows for vectorization in the column-major order of the density matrix. This unrolling operation of the matrix, denoted as "vec" (+)) results in the Choi representation of the density matrix. Conversely, the rolling operation back in the column-majon order is known as "unvec" (o()) (Wood et al., 2011). The isomorphism c-1 is mapping from C x2* to C in column- Bajumn-majjor ordering, and vice versa (Blank et al., 2020). In this isomorphism, the application of Kraus matrices can be expressed single matrix product:
E(p) Tri {A II}
(13)
Here, the Choi matrix (or super-operator) ε of a channel A is obtained by tracing out the sub- space of identity L. This matrix then can be used to evolve quantum states under the influence of quantum channel by matrix-vector multiplications, Ip') EΛ |p)cnoi: In the context of processes characterization (Chow et al., 2012) and classical simulation of variational quantum algorithm landscapes (Rall et al., 2019; Fontana et al., 2023; Rudolph et al., 2023) that involve measurements using the Pauli-Liouville bases instead of the computational bases for the vector- ization and super-operator representation (Pauli-transfer Matrix, PTM) have an extra computa- tional advantage. In this formalism, the state, the observable, and the channel super-operators are written either using the Pauli basis set Bpauti = v2 (1,X,Y, Y, or the Ozyl basis set
}
Boryl = {1 + X/ Y / (1 2)/2} (O'Brien et al., 2017; Greenbaum, 2015). A stan- dard quaternary notation of these bases is used construct multi-qubit Pauli strings, P. To acquire the expansion coefficients in this basis, the Hilbert-Schmidt norm, (A = Tr (A'B), is utilized.
ΣGP. + 4=1p) Tr (P.p)
Σ d.P + d= 10)0); = (P.O))
= Tr.A + A(P)) = ΣK,PK
(14)
Here, Ip) and (O) represent the vector representation of the state and a physical observer (e.g.. Hamiltonian) in Pauli-Liouville basis. The i-th element of these vectors is denoted by e and di, which are scalars. While RA [ represents the PTM element at the i-th row and j-th column. In this formalism, the channel PTMs can both propagate the initial state forward (Schrodinger propagation) or propagate the observable backward (Heisenberg propagation) to acquire the final expectation via matrix-vecte vector multiplications (Rall et al., 2019; O' O'Brien et al., 2017; Fontana al., 2022; 2023). These vectorization and super-operator schemes enable fusing gate channels with the subsequent noise channels offline (ahead of the simulations for RL training). The PTM of a noisy gate is given by fusing the PTMs of K channels acting on the qubit after the PTM of the gate G as RG = RAX-1 RAK-2 RAo RG- In our simulations, we use Ozy1 bases for N-qubit PTMs, due to their similarity with noiseless 2.N 2ub-t state- vector simulation, and we apply the noise models of the real devices with actual parameters we describe in the following sub-sections.
H.3 DEPOLARIZING CHANNEL
We model the gate infidelities as 1-qubit 2-qubit depolarizing channels (Nielsen & Chuang, 2010). The depolarizing channels can be represented using density matrix formalism, P. and Kraus matri- ces, K, Specifically, I-qubit depolarizing channel, Addr)) acting on qubit q with error proba- bility (or the noise strength, gate infidelity, etc.) p, is defined as follows:
A (dep) (p) (1 3/) p + (XapX, + YopY, + 7g)
(15)
25

---

In the expression above, A represents the channel's application on the density matrix, and Xa. Yg, Zn denote the Pauli operators acting on qubit The qubit is left unaffected with a probabil- ity of (1 3p) ), and with probability of 4. it undergoes an amplitude bit-flip (X xggXg) g). phase-flip ZZg or a combination thereof (Yg)K)))
Initially, we apply this noise model with the probability parameter Pone 0.1 10-2 upon im- plementation of each 1-qubit gate. In other words, a I-qubit depolarizing channel of the strength Pone 0.1 10-2 is initially utilized to model the rotation gate infidelities. Subsequently, we apply this model to both the control and target qubits, letel and Quarg, during the implementation of the 2-qubit control NOT gate CNOT(1etr1 [targ). In this case, 2-qubit depolarizing channel used to model the CNOT gate infidelity, applying this 2- 2-qubit noise model with a probability (or strength) parameter Pwwo 0.5 x 10-2.
To model gate infidelities in various TEM devices, we employed the following parameters. For TBM Mumbai device (see Table 5), the median 1-qubit gate infidelity was represented by Pone median 2.44 10-4, and the median 2-qubit gate infidelity was represented by prueeiia 8.25 x 10-3 Additionally. the maximum I-qubit gate infidelity was denoted as Pany maximum 2-qubit gate infidelity was denoted Piwo = 2.30 x 10 2 We also created a more noisy scenario by scaling these maximum infidelities by ten folds, resulting in Pone = 10 x prne and 100max 10 x Pawa These parameters, representing median, max, and 10 x max infidelity strengths, were applied as depolarizing channel strengths uniformly across all qubits and gates in their respective experiments, assuming full device connectivity. The data for the IBM Mumbai device was collected on August 8. 2023.
1.45 x 10 and the
In contrast, for the IBM Ourense device, we considered individual gate infidelity information while also factoring the qubit connectivity of the device. The qubits 0 0-1 2 - 3 with 1-qubit infidelities were set as follows: Pare 5.22 10-4, Pone = 4.14 x 10-4, Pone 1.84 10-4, and Pone 4.3 x 10m For 2-qubit gates involving qubit 1 and its neighbours, the corresponding 2-qubit depolarizing channel strengths were determined as phu = 9.55 x 10-3, Pi 9.44 10 10-3, and pl 1.25 x 10-2, The device data used for IBM Ourense was sourced from (Du et al., 2022).
H.4 READOUT ERRORS AS BIT-FLIP CHANNEL
We incorporate readout errors into our model, inspired by (Blank et al., 2020), treating them as bit- flip (amplitude flip) operations applied just before the measurement (Niclsen & Chuang, 2010). The bit-flip channel, representing the readout error with strength p on qubit q, is characterized by the following Kraus representation.
P. (o) = (1 p
(16)
In this scenario, the qubit g has probability of 1 1ppp retain its current state, and with a probability p. it flips its amplitude, akin to æ rotation around the x-axis in the Bloch sphere. Again, noise profiles of two different IBM devices were implemented.
Firstly, we uniformly apply readout noise levels obtained from IBM Mumbai devices, categorized as median and max, to all qubits in our experiments. The readout error strengths for IBM Mumbai device are denoted as mediinn = 2.25 x 10-2 and pm = 8.7 x 10-2, acquired on August 8, 2023, as listed in Table 5. For the 10 x max experiments, use 87% readout errors, ten times the max readout errors rendered VQE experiments unfeasible due to signal loss.
Secondly, we account for readout errors of qubits 0 1 2 3 from the IBM Curense de- vice (Blank et al., 2020). The corresponding error values are p% = 1.65 x 10-2, .pp = 2.38 x 10-2, Pi 1.57 x 10 and pi 3.95 10
1.4.1 STATE PREPARATION ERRORS AND THERMAL RELAXATION CHIANNEL
The state preparation errors in transmon qubits are due to residue thermal populations that can be modelled as thermal states (Krantz et al., 2019). A transmon qubit, a quantum anharmonic oscillator in the truncated Hilbert space, can be modeled as a k-level quantum system (a qudit) with unequal level spacing such that the subspace of the first two levels is reserved for computation. A widely used approximate model is qutrit (where the infinite-dimensinaa Hilbert space is truncated to k = 3) where the ground state energy is set to be zero, Eg = 0, the first excited state is determined by the qubit frequency (w) parameter, Eg hw, and the second excited state is determined by
26

---

the anharmonicity parameter (6, typically negative number), Eg h (2w + 6). Given the fridge temperature T, the initial quantum state of a qubit with residual thermal populations is the following.
exp (- )
E1
Eg (-ReT)
PaZ = (0) (0) + exp
11) [1]
12) (2)
Here Po denotes the initial thermal state, kn and h denote the Boltzmann and reduced Planck con- (-
stants, and Z denotes the equipartition function Z = 1 exp
(-ha) F5
kg
+ exp
kg E
F2
The prob-
ability of the quantum state being initially in the excited state is given as Pe exp our experiments, we used Wmmmnaii = 4.894GHz, 8mmmbai = -0.33GHz for the IBM Mumbai device and we used Wourresse 5GHz, Oaurense
/Z. In
K
O.33GHZ for the IBM Ourense device acquire these
parameters.
Each time qubits are reset, the decoherence, which is caused by coupling of the quantum system to external degrees of freedom such as stray fields, allows computations to run for a certain amount of time (coherence time) before the strictly quantum mechanical properties are lost. The thermal relaxation time. Ti. measures how long it takes for a qubit that is initially prepared in the excited state [1) (the North pole of the Bloch sphere) to decay back to the ground state (0) (the South pole of the Bloch sphere) (Krantz et al., 2019). The phase-coherence time, T2, measures how long it takes to lose the phase information. When qubit initially prepared on the equator of the Bloch sphere, such as +) state, it can not be distinguished from other states on the equator, such as a 1-> state, due to being end up in a classical mixture of these states after T2. These thermal relaxation and decoherence processes do not happen instantly after Ti and T2 amount of time, but the qubits experience them rather gradually as 1-, and 2-qubit gate implementations also take a finite amount of time, Atone and Atowo hindering the perfect implementation of these gates beside the gate infidelities. These gradual processes can be modeled as quantum channels that follow the application of 1-, and 2-qubit gates (Blank al., 2020). The effect of the thermal relaxation channels depends on how good the initial state is prepared or the excited state probability, Pe. decoherence times Ti and T2 of the quantum hardware in hand, and how fast the gates are implemented Atone and Atwoo (for values see Table 5). For these gate durations, we can define thermal relaxation and dephasing
Using these, we can define the qubit reset probability Prese E,,, and the following probabilities.
Ax gate error rates ET, = exp A. and eT, = exp - 75
Ti
Pia Ps Pro Pri
Pz = (1 Preset) (1 eT,
eT,,TTT) cT2 )2/) //2
Pro (1 - Po)) Preset
Pr. PePreset
(17)
Here Pro gives the probability that qubit resets the ground state 10), Pr, gives the probability qubit resets to the excited state [1), Ps gives the probability that qubit in the ground state is hit by phase- flip operation Z, and Pia gives the probability that qubit in the ground state is hit by an identity gate I. The regimes where T2 < Ti and T2 > Tj have different channel representations (Blank et al., 2020). The Kraus representation for the dierall-relaxttinn channel (TRC) acting on the qubit q when T2 S Ti follows the probabilities described above.
Ko Vpial
K1 VpzL
Pro
+ Z
Pro X - iY
K₂
K3
2
2
2
Pr. X + ii
Pr.
I Z
K4
K5
2
2
2 2
terms. Here K₂ (Kş) operator resets the qubit to (0) ((1) with probability Pro (pr,). Similarly K3 (K₂) operator relaxes (excites) the qubit into (0) (1)) with probability Pro (pr,). While K, and K, implement / and Z with their respective probabilities. In the T2 > Ti regime, the quantum channel is represented by the following Choi
In this regime, TRC has amplitude and
27

---

matrix (Blank et al., 2020).
Pry
0
C
0
Pri
0
0
A
(18)
0
0
Pro
0
CT2
0
Pro.
The eigendecompositiin and "unvec" operations yield Kraus matrices K3 = vxD(vx). For the T2 Ti regime, we use the analytically computed Kraus matrices given in Pennylane software library (Bergholm et al., 2018). To compute these, the Ti and T2 times of IBM Mumbai and IBM Ourense devices we use are the following. The ones for Mumbai devices are the median values of the data acquisition date Ti = 122.28us.. T2 = 167. 2ps. (see Table 5). And. the ones for Ourense devices are, 7h 75.75us Th 50.81 T 78.47us. T3 27.56,s T? 101.51ps, jss T2 = 107μs, and T = p9.54ps, T2 78.38ps (see (Du et al., 2022)).
SOFTWARE LEVEL SIMULATION OPTIMIZATION
Our RL agent training involves billions of queries to the classical simulator, incurring substantial time and computational costs. In each episode, unless terminated by the random halting protocol or by reaching threshold value before the final action, the RL agent executes a series of actions Pact (an integer). During each action i, where E [aatt] the agent appends new 1-, or 2-qubit gate to
quantum circuit. At the simulator level, this corresponds to compiling ng noog + 1lgg gates (4) applied by the agent until action i. Here, Nl represents the total number of gates implemented during cost function evaluation (subsequent matrix-vector multiplications) at action i, such that at (i)
(t) (i)
the final action i = Platss total of Rax ng (1,aa) gates are applied. Additionally, nogg and 1ggg denote the number of 1-, and 2-qubit gates compiled during action i. The cost function evaluation
(i)
takes (1) & lieall amount of time for each action i. The value of (4) can be approximated as an intermediate value of /eeaal even though the function evaluation time cumulatively increases with Furthermore, at each action i, classical optimizers require 7eveall cost function evaluations. Given total of ne episodes, training an RL agent roughly consumes ne. X Paat X Titeval X teeaal time, with an additional 20 seconds per episode for neural network training on an NVIDIA-A100 GPU.
As of the completion of this work, Qulaes remains the most efficient classical simulation framework for bython-based quantum circuit evaluations. This efficiency is derived from effective memory- access, swift linear algebra facilitated by C++ SIMD optimizations, and an underlying algorithm that handles horizontal and vertical gate fusions for named gates in noiseless scenarios (Suzuki et al., 2021). However, the advantage using Qulacs in our experiments diminishes when transitioning to noisy simulations using the Kraus operator sum formalism, the algorithmic benefits related to gate fusions are not as prominent in this context. The total number of dense Kraus matrix multiplications defined as 7km = 2 Roge 2mtggRtgeWees)
Here, noge represents the number of 1-qubit channels following 1 I-qubit gates, and Bok denotes the number of Kraus matrices in these 1 I-qubit channels. Similarly, ag indicates the number of 2-qubit channels following 2-qubit gates, and nek specifies the number of Kraus matrices in these 2-qubit channels. It is important to note that the term for 2-qubit channels is multiplied by two, different noise channels need to be applied at each qubit during an evolution. Additionally, the overall number is multiplied by two because the density matrix must be affected both from the left and the right by Kraus matrices. For mathematical implementations of these operations, please refer to Appendix H.
Crucially, these noise simulations are performed online (during simulation) each time. As the num- ber of qubits and gates increases, the cost function evaluation times (fivvt) grow exponentially. This results in RL training times escalating from the scale of days the noiseless case to months in the noisy case, rendering impractical to obtain prompt feedback from the training, even for basic unit tests and byperparameter optimization. To address these challenges, we precomputed the Pauli- transfer matrices (PTMs) of the noisy gates before simulations and developed GPU-base simula- tion framework in JAX (Python) (Bradbury et al., 2018). Due to time constraints, we were unable to implement the backward forward propagation algorithms for sparse Paulis as prescribed in (Rall
28

---

et al., 2019; Fontana et al., 2023). However, we successfully implemented fast, dense matrix-vector multiplication-bal based algorithms for both noiseless and noisy cases.
To guide the simulator the gates to be implemented, provide it with NumPy array (Harris ct al., 2020) of dimension ng x 2. The first column enumerates the subsequent gates or PTMs in ascending order, with the first element corresponding the first gate, and so forth. The second column contains the variational parameters to be implemented, with parameters for non-parametric gates set zero. We also provide the positions (row numbers) of this instructions array to facilitate updating parameters in the correct order outside closed-circuit GPU computations occurring in XLA (accelerated linear algebra library) (Sabne, 2020). A just-in in-time (JIT) compiled JAX function reads these instructions (in XLA) and, for each row (or instruction), generates a unitary matrix or PTM using the gate number and angle. Subsequently, another JIT-compiled function takes a vector from the previous iteration and the matrix constructed by the previous function, returning new vector after multiplication. One level higher, another JTT-compiled function iterates through these matrix- vector multiplications (forward propagation) row by row using JAX's native foriloop, carrying the vector from the previous iteration until the end of the circuit. Finally, another JIT-compiled function takes over this vector and either performs inner product operation ai H& for dense Hamiltonians in noiseless simulations or a dot product (H) p)) for Hamiltonian vectors in noisy cases. All these functions are merged into single JIT-compiled expectation value computation subroutine. When the user queries this subroutine with a new set of angles, it first updates the variational parameters of the instructions array, then feeds this arryy to the JIT-compiled subroutine (running on GPU using XLA instructions) to obtain real number output. Due to the asynchronous nature of GPU comput- ing and the current limitations of XLA, which require static tensor geometries as input and outputs, these computational subroutines need to run concurrently with static input-output shapes to leverage computational benefits. Our implementation takes these aspects into account. These subroutines could be further integrated with sparse Pauli algorithms (Rall et al., 2019; Fontana et al., 2023) or tensor network methods (Orús, 2014) to extend the scalability of these noiseless and noisy simula- tions in terms of qubit numbers and gate counts. However, we leave this avenue for exploration in future work.
Qulacs_0 -+-- Qulacs_
Qulacs_ -I- Quiacs_3
Qulacs_4
JAX_O
JAX_all
2
32-3
LIH-4
10000
1750-
3000
1500-
2500-
8000
Function time
1250
2000
6000
1000-
1500
750
4000
1000
500
2000
250
500-
3
11
13
11
13
Depth
Depth
Depth
Figure 8: Computation time per VQE function evaluation in hardwarre-fficient ansatz (HEA) using differing software and methods. The x-axis shows the HEA depth for the given problem. Different traces display the implementations in different software and different noise models.
In Fig. 8, we evaluate the performance of different simulators implementing various noise models as the hardware-efficient ansatz depths increase (i.e., with growing gate counts) for different qubit system sizes. Noise models are identified by cumulative numbers that increase with the addition of noise channels after their respective gates. Initially, noise model "0" represents the noiseless sim- ulator. Subsequently, noise model "1" introduces a 1-qubit depolarizing channel after each -uqbb gate. Building on this, noise model "2" adds depolarizing channel to each "ctrl" and "target" qubit of the CNOT gate after post-implementation. Further extensions include noise model "3" in- troducing I-qubit thermal relaxation channel after each I-qubit gate, and noise model "4" which, on top of noise model "3" adds a thermal relaxation channel to each "ctrl" and "target" qubit of the CNOT gate. While Qulacs exhibits optimal evaluation times in the noiseless case for all qubit numbers, the advantage diminishes with the inclusion of any noise channel, especially as gate count and the number of dense matrix multiplications increase due to Kraus operator sum formalism (see
29

---

Appendix H). This trend becomes more pronounced with the addition of noise channels following gate implementations (i.e., an increase in the noise model number), leading to an exponential in- crease in the number of Kraus operations. In terms of quantitative results, for depth 13, JAX-PTM yields an improvement of up to a 2.65-, 3.9-, and 9.93-fold over Qulacs noise model in 2-, 3-, and 4-qubit problems, respectively. We acquired these statistics from 20000 independent runs.
In Fig. 9, we show the RL episode times concerning different noise models and software, without implementing any random halting or early stopping of episodes, allowing each proceed up gate count of 40. We ran the RL training for each of these scenarios for 200 episodes once. The neural network training takes approximately 20 seconds per episode. Again, in terms of quantitative results, we observe an improvement of up to a 1.34-, 1.59-, and 3.85-fold over Qulacs 4 noise model in 2-, 3-, and 4-qubit problems, respectively. However, excluding the neural network training time, the improvement was more pronounced. We observed 2-, 2.9-, and 5.74-fold speed-up for the above problems, respectively.
JAX
--- Quiacs v6.0
H2-2
Hz-3
LiH-4
200
40.0
45-
Episode time (secs)
37.5
35.0-
40-
32.5-
35-
125
30.0
100
27.5
25.0
25-
22.5-
25-
20.0
20-
2
3
channels
Number noise channels
channels
Figure 9: Computation time per episode RL agent using differing software and methods. The x-axis shows the number of noise channels added after their respective gate, and x O corresponds to noiseless simulations.
In conclusion, the use of PTMs enabled to transform the N N-qubit noisy simulation problem, re- gardless of the number of noise channels, into the 2.N -ubit noiseless simulation problem involving dense matrix-vector multiplications. Utilizing JIT-compilation and GPU computation capabilities in JAX, we achieved up to six-fold improvement in RL training while simulating noisy circuits. This enhancement allowed to successfully train RL agents for QAS problem under the effect of quantum device noise.
J BEHAVIOURAL DIFFERENCE OF REAL DEVICE & CLASSICAL NOISY SIMULATION
Recent studies suggest that despite device drift, low-level hardware noise in quantum machine learning tasks yields outcomes where real hardware and simulations demonstrate comparable be- havior (Rehm al., 2023). Our comparative experiment simulation of its noise profile² aligns with these findings in our context as well. Using the circuit generated CRI QAS method for 4-qubit H2 molecule (see Fig. 10), validated
TBM Lagos device and the classical
hypoth- esis by comparing measured expectation values on the device to the classical noise model used in simulations CRLQAS. Fig. 1l showcases that the expectation values of Pauli strings align within one-sigma standard deviation етror bars. The expectation values were computed individually for each Pauli string which allowed for fine-grained analysis. is noteworthy that the full Hamiltonian expectation values also exhibit similar trends, with a measurement of -0.0615 + 0.5295 on IBM Lagos and -0.0335 0.5294 on the noisy simulator. Our experiments were executed using the Qiskit Runtime IBM Client API.
We performed the classical simulation of the 1BM Lagos noise profile based on the calibration data of November 16, 2023.
30

---

40
Ry
Ry
Rz -0.89
Rg 1.188
R2 0.32
Rg
Ry 2.000668
Rz 8.00055
3:44
-0.00
41
42
93
Figure 10: The parametrized circuit generated by the CRLQAS (RH, wo-IA) method for the H molecule. 'H'' indicates the utilization of the random halting scheme, while "wo-IA' signifies the absence of illegal actions.
IBM Lagos
Noisy Simulator
Expectation Value
0.1
0.0
-0.1
-0.3
XYXX
ZIIt
IIII
XXYY
112
Z1Z1
221
Z11Z
1112
122
1211-
1212
11ZZ
YYXX
Pauli Strings
Figure 11: Comparative experiment of real hardware to classical noisy simulator: The bars in the plot
represent the expectation values of each Pauli string for H2 1 with the circuit depicted in Fig. 10. The colors represent the results for the RM Lagos (orange) and noisy simulator (green) introduced this work.
K COMPARISON TO OTHER QAS METHODS
We compare our method CRLQAS quantumDARTS (Wu et al., 2023) and RLQAS (Ostaszewski et al., 2021b) in noiseless experiments (see Table 1 and Table 3). For noisy experiments, we compare QCAS (Du et al., 2022) (see Table 2). While there is substantial amount of literature on quantum architecture search (QAS), replicating the performance of some methods is challenging due the lack of unified experimental settings. Moreover, the purpose of this work was to establish the limits of RI methods in com- bination with noisy intermediate-scan quantum hardware. Nonetheless, we provide an overview of reported results and corresponding settings of the ground state estimation task for some QAS approaches.
modified variant of qubit-ADAPT VQE (Tang al., 2021),
In the QAS literature, we identified two works that include the ground state energy estimation prob- lem their experiments. (Rattew et al., 2019) presents numerical results for their EA-based QAS method within chemical accuracy, while (Wang et al., 2022) evaluates their one-shot super-cireuit method under simulated circuit noise, significantly impacting estimation accuracy
(Rattew et al., 2019) proposed Evolutionary Variational Quantum Eigensolver (EVQE) that evolutionary programming techniques to dynamically generate and optimize quantum circuits (by adding removing gates). The gate set in this algorithm is composed of 1-qubit universal gates and 2-qubit controlled universal gates, which after the optimization, are decomposed into CNOT gates and I-qubit universal rotation gates. The noiseless experiments involved molecules, 6-qubit LiH with varying interatomic bond distances and 7-qubit BeH, with bond distance of 1.3A. In
31

---

the noiseless case, examining Fig. 3 in (Rattew et al., 2019), the best-observed depth and CNOT gate count were approximately 35 and 40, respectively (exact numbers not explicitly stated in the paper). During EVQE training, the achieved error was slightly less than 1 x 10 (no exact number provided). On the other hand, CRIQAS (IA, wo-RH) median statistics achieves 8 x 10-1 error with depth and CNOT gate count of 27 and 30, respectively (see Table 3). For the noisy experiment (where noise profiles were simulated using the Qiskit Aer QASM simulator) with a 4-qubit LiH at a bond distance of 2.2Å, referencing Fig. 7 in (Rattew et al., 2019), the averaged noisy energy error, depth, and number of CNOT gates over five seeds were approximately 0.036Ha, 8, and 6, respectively. In our case, we achieved a slightly improved noisy error of 0.0356Ha with an average of 1.33 CNOT gates and depth of 10 across three seeds. Although the mean noisy errors and depth are comparable, CRLQAS produces quantum circuit with fewer CNOT gates.
(Wang et al., 2022) introduced the TorchQuantum framework based on PyTorch (Paszke al., 2019) and develops the QuantumNAS algorithm, a one-shot super-cireuit method designed for finding noise-adaptive quantum circuits in VQA tasks. This technique involves sharing gate parameters among sub-circuits after one shot training to accelerate circuit evaluation. However, optimizing the super-circuit poses a challenge, and furthermore, there a weak correlation performances between sub-circuits with inherited parameters and those with optimal parameters from individual training. Our attempt to compare with (Wang et al., 2022) was hindered by the lack of essential information for comprehensive analysis. More specifically, details about molecular geometries and quantum circuits prevented us from conducting a quantitative experiment for comparison, as it would require rumning their algorithm from scratch for the molecules considered in our study. Given that we already compared to the QCAS (Du et al., 2022) algorithm (see Table 2), which is similar QuantumNAS,, and considering the aforementioned reasons, we did include this work in our comparative experiments. However, a qualitative comparison was possible for the H2 molecule in a 2- 2-qubit system. Examining Fig. 17, the best result achieved in their experiments had an energy error of H.H. In our H2 simulations, we achieved mean noisy energy error over three independent seeds of 1.116 x 10-3 4-344 x 10-1 (for TBM Mumba median), which is three orders of magnitude better than their results. Additionally, we outperformed them for IEM Mumbai max and 10 times max noise levels, with mean noisy energy errors of 0.907 = 0.008 and 0.0912 + 0.004, respectively. Unfortunately, the manuscript does not provide information about the number of gates, depth, or parameters achieved for this experiment.
L LIST OF MOLECULES
Given the geometrical description of the molecule, and using the Kom--Opeehheimer approximation, fix finite basis set, in case STO-3G, discretize the system. Subsequently, utilize the OPENFERMION open-source electronic structure package (McClean et al., 2020), with PSI4 serving as the backend computational chemistry software, to generate the molecular Hamiltonians. We describe the essential details to generate the Hamiltonians in Table 6 below.
Table 6: List of molecules considered in our simulations. The coordinates of the configuration are given in angstrom.
Molecule Fermion to qubit mapping
Configuration
Number of qubits
Jordan-Wigner Wigner
H (0, O, 0);
2
(Jordan & Wigner, 1993)
H (0, 0,0.4114
H (0, O, O);
H2
Jordan- Wigner
3
H (0, 0.7.7114)
H (0, -0.35);
Jordan- Wigner
4
H (0, 0.355
Parity (Seeley al., 2012)
Li (0, 0);
LiH
4
H (0,0,3.4)
Li (0, 0, 0):
Hordan-Wigner
6
H (.,0,
H (- 0.002, -0.002, 0);
H₂O
Jordan-Wigner
O (0.835, 0.452, 0);
8
H (1.477, -0.273,
32


Reviews:
[{"review_id": "gsMdWNXcu6", "reviewer": "Reviewer_K92h", "summary": "Variational quantum algorithms (VQAs) stand for a promising candidate of practical quantum algorithms in the NISQ era. Nevertheless, the performance of VQA is significantly influenced by the choice of ansatz and noise effects. \n\nIn this paper, the authors adopt curriculum reinforcement learning for quantum architecture search (QAS), aiming to select ansatz with good performance under hardware errors. Specifically, they devise a CRLQAS algorithm with tensor-based 3-D encoding of quantum circuits, a random halting scheme to reduce the circuit length, and an variant of Adam-SPSA for better convergence. In addition, this work accecelerate simulation of noisy quantum circuits by employing pauli transfer matrix formalism. \n\nNumerical experiments on quantum chemistry tasks demonstrate the performance of the proposed method under noiseless and noisy circumstances.", "questions": "- Please proofread the manuscript and correct the numerous mistakes in grammar and notation. Also there is a repetition in the reference section, where \"Abhinav Kandala, Antonio Mezzacapo, Kristan Temme, Maika Takita, Markus Brink, Jerry M Chow, and Jay M Gambetta. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. nature, 549(7671):242–246, 2017\" is listed twice.\n- Some descriptions are unclear to me. In Section 3.3, what does \"$n_s$ is the number of successes\" mean? Details in Section 3.4 could also be better stated. Moreover, from a reader's point of view, the organization of Section 3 is scattered and lacks emphasis on major contributions. \n- Does the model need retraining for different noise models to deliver competitive results? I am curious about the efficiency for adaptation to new noise models. Also I believe experiments on real quantum machines comparing the proposed method with other algorithms would make the work more solid.\n- Although the authors claim to tackle the general employment of VQAs, numerical experiments on applications are actually limited to VQE for quantum chemistry tasks. The authors should avoid overstatement, or apply the method to general VQA settings and tasks.\n- There is almost no discussion of limitations. What is the weakness of this method and what are the potential future directions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Originality: As far as I know, this is the first work to consider noise models from real quantum machines in RL-based QAS. Previous work focus on noiseless scenarios or only touch on simple noisy configurations. The setting of this work is closer to real applications in NISQ era, hence making it an advance over the previous state-of-the-art.\n- Quality: The paper proposes several techniques which are effective and easy to realize. The tensor-based binary encoding is straightforward but efficient and compact. Pruning of illegal actions successfully narrow the search space. Random Halting is introduced to reduce the training episodes, which is a major drawback of previous work. The offline computation with Pauli-transfer matrices and JAX achieves considerable acceleration for noise simulation.\n- Clarity: The main ideas and techniques in the paper are mostly clear, but many issues remain (see Weaknesses & Questions). \n- Significance: This paper seeks to tackle an important problem by considering noise of current devices in QAS for VQA, which is well-motivated. Literature discussing the negative impacts of noise on VQA has been thoroughly cited. The numerical demonstration of achieving chemical accuracy and outperforming previous methods is impressive.", "weaknesses": "- The writing is poor, full of mistakes in grammar and notation. For example, the first sentence in the second paragraph of Section 3.6 is incomplete. There are erroneous notations such as  $T_{s}^{e}$ v.s $T_{s^{e}}$ in Section 3.3, wrong format of citation like (Wang et al., 2021) in introduction, inconsistent proper nouns such as \"ansatz\" v.s. \"ans{\\\"a}tze\", and numerous syntax errors. These issues impair the quality of the paper. \n- Originality: This paper extends the ideas from previous work (Ostaszewski et al., 2021). Some techniques such as feedback driven curriculum learning and Adam-SPSA seem to be the direct adaptation of existing methods.\n- Although several techniques are proposed, the paper may lack insights into which component principally improves the results, especially under noisy circumstances. It is unclear to me which technique is specialized for facilitating noise adaptations, rather than general improvement over existing methods.\n- Related to the previous question, in the noisy simulation section, the paper conducts no comparison with other RL-based QAS methods. Therefore it is unclear what is the improvement of this algorithm over previous methods in noisy environments. Moreover, for molecules such as LiH-4, the experiments only consider shot and 1-qubit depolarizing noise, which is far from real noise models. This undermines the persuasiveness of this work. \n- Important related work not discussed. Despite little consideration of noise in previous reinforcement learning methods for QAS, there have been many attempts in traditional architecture field. The paper may need more comparison with other QAS algorithms under hardware erors, especially those targeting noise-aware search, for example Wang et al. QuantumNAS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAVvyzOHWd", "reviewer": "Reviewer_sjih", "summary": "This paper tackles the challenge of identifying optimal quantum circuits within the constraints of noisy intermediate-scale quantum (NISQ) devices. It introduces a novel curriculum-based reinforcement learning algorithm for quantum architecture search (CRLQAS), which innovatively employs a 3-D architecture encoding, an episode halting mechanism, and an enhanced simultaneous perturbation stochastic approximation optimizer for more efficient exploration and convergence. By incorporating Pauli transfer matrix formalism to improve simulation times, the proposed CRLQAS demonstrates superior performance in automating the design of quantum circuit architectures, outstripping existing methods in both noiseless and noisy conditions, specifically for quantum chemistry applications.", "questions": "1. Can the authors provide the source code that was mentioned as a part of the paper's contributions to facilitate replication and further research by peers?\n\n2. Would it be possible to expand the comparative analysis section of the paper to include methodologies based on SMT solvers, such as the approach detailed by B. Tan and J. Cong in ICCAD 2020, to provide a broader context and demonstrate the relative effectiveness of the proposed algorithm?\n\n3. Considering the potential for confusion within the interdisciplinary community, could the authors consider using a more precise term than \"architecture\" to describe the optimization of parameterized quantum circuits, thereby avoiding conflation with low-level hardware design?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper excels in its strategic approach to quantum computing in the NISQ era by giving paramount importance to the role of quantum noise and its effect on circuit architecture. It introduces a pioneering curriculum-based reinforcement learning (RL) algorithm, CRLQAS, which specifically addresses the challenges posed by noise in quantum systems. The use of the Pauli transfer matrix formalism in this algorithm marks a substantial improvement in the simulation of noisy quantum circuits, significantly reducing computational time and increasing simulation fidelity. \n\nIn addition to the quantum-specific advancements, the paper contributes to the field of RL by implementing a 3-D architecture encoding and an episode halting mechanism within its curriculum-based framework, enhancing the RL agent's ability to discover shorter and more hardware-efficient circuit architectures rapidly. The introduction of a novel optimization technique further refines the learning process, facilitating faster convergence and potentially leading to more robust quantum computing solutions. These combined strengths showcase the paper's dual focus on developing an efficient simulation that carefully accounts for quantum noise and on refining RL techniques to ensure that the architecture search yields practical, hardware-efficient designs.", "weaknesses": "One area for improvement in the paper is the absence of the promised source code, which limits the ability of others to replicate and build upon the work. It is essential for the authors to provide the source code to enhance the transparency and applicability of their research. Therefore, it is recommended that the authors include a link to the open-sourced code or an appendix containing the code in any subsequent revision of the paper. This addition would be a valuable resource for the community and would greatly facilitate further research and validation of the proposed methods.\n\nThe paper could be enhanced by broadening the scope of its comparative analysis with existing methodologies. While the current work focuses on a specific approach to quantum circuit optimization, there are alternative methods worth considering, such as those based on SMT (Satisfiability Modulo Theories) solvers. For instance, the technique presented by B. Tan and J. Cong in \"Optimal layout synthesis for quantum computing,\" (ICCAD, 2020) could serve as a valuable benchmark. It would be beneficial for the authors to include a comparison with these kinds of potentially optimal circuit compilers to contextualize the effectiveness of their proposed method when applied to analogous problems.\n\nFurthermore, the terminology used in the paper, specifically the term \"architecture,\" may lead to ambiguity within the broader scientific community. In the realm of computer architecture, \"architecture\" typically refers to the low-level hardware design, as discussed in sources like the one available on IEEE Xplore. Since this work is centered around the optimization of parameterized quantum circuits, and not the hardware itself, clarity could be improved by selecting a term that more accurately describes the subject of optimization. A term like \"circuit design\" or \"circuit configuration\" may better capture the essence of the research without the potential for misinterpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A27PewPOhO", "reviewer": "Reviewer_wt33", "summary": "The paper presents a new combined RL algorithm tailored for the domain of quantum architecture search (QAS). AN empirical study shows benefits over state-of-the-art approaches in this very early field of research.", "questions": "see above...\n\nWhat is the distinct impact of the \"illegal actions\" adaptation?\n\nWhy is the result even better upon \"closer inspection\"? (p7, very bottom)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper does a reasonable job introducing the issue of quantum architecture search and, from the presented difficulties of the domain, manages to derive its adaptations to standard RL employed to tackle this domain. The adaptations presented are sound and straight-forward. The results give a nice guideline for future research in that direction.", "weaknesses": "As with most approaches in the field of quantum computing, the evaluation is rather weak does not allow to draw too strong a conclusion (which, luckily, the authors did not draw). The presentation of the sequence of experiments is rather convoluted (and not helped by putting the plots very late in the document). More clear-cut scenario descriptions would be helpful.\n\nThe discussion lacks an explanation how adaptations which were largely motivated by the accommodation of noise models then manage to outperform the state-of-the-art in the noiseless case.\n\nAlso, I would have liked a singled-out study on the impact of the \"illegal actions\" adaptations. It appears that it might do a lot of the heavy lifting, which would then change the overall story of the paper.\n\nMany discussions are left to the Appendix and several forward-references point to it. The structure should be improved here to make a clearer distinction between important and less important points.\n\nThe writing still has several problems including:\n- p1, second paragraph, last sentence: no verb\n- p2, fig1, caption: \"(i)\" without \"(ii)\"\n- p3: \"Methods like this\" should read \"Methods like these\"\n- p4, eq2: parameters/variables are not sufficiently explained in the text, at least not in a concise manner\n- p4, last paragraph before 3.1, first sentence: It is unclear how the \"environment\" is deterministic/stochastic. Refer to transition function, reward function etc.\n- p5, section3.3, throughout: Typo \"T_{s^e}\" instead of \"T _s^e\"?\n- p4, section3.4: do not put comma after \"where\"\n- p6, section3.6: Use \"it is\", not \"it's\"\n- p6, section3.6: \"the Appendix\" randomly appears in a sentence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mehW4IELbO", "reviewer": "Reviewer_WndL", "summary": "In the era of noisy intermediate-scale quantum computing, selecting effective circuits in line with current device constraints is pivotal. This study accelerates quantum circuit simulations using the Pauli transfer matrix method and introduces a novel Curriculum-based Reinforcement Learning Quantum Architecture Search (CRLQAS) approach for optimal circuit structure determination.", "questions": "1. What is the function you used to fit the spin orbital of the molecule?\n\n2. What is the computational cost for your framework? Say, based on the GPU acceleration, on what specific classical hardware, how long does it take? Please specify in detail.\n\n3. Have you tried to implement your final result on real hardware? - I believe it should be easy for you since you already get your optimal arch and parameters from noisy simulation. You can just run this circuit on real quantum hardware and obtain the result.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A detailed analysis of the error rate.\n2. Using the Pauli transfer matrix method to accelerate simulation and consider the noise in the training process.", "weaknesses": "1. The paper does not specify the function used to fit the spin orbital of the molecule. If the 'sto-3g' function is used, the claim of achieving chemical accuracy might be an overstatement. While it's not impossible, further clarification is necessary.\n\n2. I'm concerned about the computational cost and efficiency of the reinforcement learning approach, especially when adding the noise injection method. Even with GPU acceleration, reinforcement learning is inherently expensive. Given that the noise model is derived from \"current\" calibration data and the gate-level noise model is only an approximation, there are concerns about its accuracy. For instance, although noise might seem stable over a short time frame, over a more extended period (e.g., a week), one can observe significant shifts. If the framework has a high computational cost, it may yield unreliable results if the calibration data changes rapidly during training.\n\n3. Some parts of the paper could be clearer. For example, Table 4 should specify which molecule task is being discussed. There are several sections in the paper with similar clarity issues, making it harder to understand.\n\n4. I observed that only one quantum architecture search work is considered in the comparative experiments. There are many other relevant works in the quantum architecture search for VQE, such as:\n\n[1] Wang, Hanrui, et al. \"Quantumnas: Noise-adaptive search for robust quantum circuits.\" 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022.\n\n[2] Rattew, Arthur G., et al. \"A domain-agnostic, noise-resistant, hardware-efficient evolutionary variational quantum eigensolver.\" arXiv preprint arXiv:1910.09694 (2019).\n\n[3] Liu, Xiaoyuan, et al. \"Layer VQE: A variational approach for combinatorial optimization on noisy quantum computers.\" IEEE Transactions on Quantum Engineering 3 (2022): 1-20.\n\n[4] Cheng, Jinglei, et al. \"TopGen: Topology-Aware Bottom-Up Generator for Variational Quantum Circuits.\" arXiv preprint arXiv:2210.08190 (2022).\n\nWhile there is a brief comparison with previous reinforcement learning-based architecture search papers, the fact that they don't consider noise models means it's expected for your approach to perform better. However, the increased computational cost is also a concern.\n\n5.There's a notable absence of results on real quantum hardware. Testing only on a noisy simulator might not be entirely convincing, especially when using reinforcement learning, which is assumed to achieve the upper bound of performance. Given that you utilize a noise model from the current noisy environment and consider only one seed, real hardware testing is crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kK0CCiWoNG", "reviewer": "Reviewer_3EHX", "summary": "This paper introduces a novel quantum architecture search (QAS) algorithm based on curriculum reinforcement learning (CRL) for designing parameterized quantum circuits (PQCs) under realistic noisy environments. The main contributions include a tensor-based binary encoding scheme, a mechanism of illegal actions to prune the search space and avoid redundant gates, and a random halting technique to encourage the agent to find shorter circuits. Additionally, it presents a variant of simultaneous perturbation stochastic approximation (SPSA) algorithm with adaptive momentum and variable sample budget for faster and robust optimization, and a fast GPU simulation framework using Pauli transfer matrix formalism to fuse gates with their noise models. The effectiveness of CRLQAS is demonstrated on quantum chemistry tasks of finding the ground state energy of various molecules, such as H2, LiH, and H2O, in both noiseless and noisy settings.", "questions": "1. As mentioned in the Intro, current NISQ devices are characterized by limited qubit connectivity and susceptibility to noise, but how does the proposed approach deal with qubit connectivity?\n2. The proposed method is based on reinforcement learning, but only one related method is mentioned in this paper, and there are several QNSA works based on reinforcement learning that are not mentioned, such as [2,3,4]. Furthermore, compared with these methods, what is the innovation of CRLQAS, and whether it can reflect the advantage of the experiment? \n\n[2] Kuo, E. J., Fang, Y. L. L., & Chen, S. Y. C. Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.07715, 2021.\n\n[3] Chen, S. Y. C. (2023, August). Quantum Reinforcement Learning for Quantum Architecture Search. In Proceedings of the International Workshop on Quantum Classical Cooperative (pp. 17-20), 2023.\n\n[4] Sun, Y., Ma, Y., & Tresp, V. Differentiable quantum architecture search for quantum reinforcement learning. arXiv preprint arXiv:2309.10392, 2023.\n\n3. In Table 4, why is CRLQAS(wo-RH) better than CRLQAS? What is the meaning of random halting (RH)? \n\n4. What are the advantages and disadvantages of the proposed method and the Predictor-based QAS methods [5,6]?\n\n[5] Zhimin He, Xuefen Zhang, Chuangtao Chen, Zhiming Huang, Yan Zhou, and Haozhen Situ. A gnn-based predictor for quantum architecture search. Quantum Information Processing, 22(2): 128, 2023b.\n\n[6] Shi-Xin Zhang, Chang-Yu Hsieh, Shengyu Zhang, and Hong Yao. Neural predictor based quantum architecture search. Machine Learning: Science and Technology, 2(4):045027, 2021", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novel QAS algorithm: The paper proposes a curriculum-based reinforcement learning QAS (CRLQAS) algorithm that can automatically construct parametrized quantum circuits for variational quantum algorithms in realistic noisy environments. The algorithm introduces several novel features, such as tensor-based binary circuit encoding, illegal actions, random halting, feedback-driven curriculum learning, and Adam-SPSA optimizer.\n- Experiments on real quantum devices: The paper demonstrates the efficiency and robustness of CRLQAS under different noise models inspired by real IBM quantum devices, and provides comprehensive descriptions of the experimental setup and the source code for reproducibility.", "weaknesses": "1. (Major) The absence of the comparison of some key methods in terms of theory and experiments. This paper emphasizes the design for quantum hardware errors, but there is no mention or comparison of some methods of QNAS that are also designed for noise, such as QuantumNAS (a noise-adaptive quantum circuit search) [1]. In addition, as far as I know, QCAS (Du et al., 2022) is tested on depolarizing errors, and QuantumDARTS (Wu et al., 2023) is evaluated readout errors. However, these methods under noise are not compared in the experiments.\n\n[1] Wang, H., Ding, Y., Gu, J., Lin, Y., Pan, D. Z., Chong, F. T., & Han, S. QuantumNAS: Noise-adaptive search for robust quantum circuits. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA).\n\n\n2. (Minor) The representations of some tables are not clear enough. For example, if the best result is bolded, then ‘7.21*10^-8’ and ‘2.9*10^-4’ should be bolded in Table 1. In Table 4, CRLQAS (RH) has two rows of results but they're not marked what do they mean.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)