Reinforcement learning for optimization of variational quantum circuit architectures
Authors: Mateusz Ostaszewski
Lea Marion Trenkwalder
Wojciech Masarczyk
Eleanor Scerri
Vedran Dunjko
Venue: NeurIPS 2021 Poster (2021)
Keywords: Quantum Computing
Reinforcement Learning
Quantum Chemistry

Abstract: 	The study of Variational Quantum Eigensolvers (VQEs) has been in the spotlight in recent times as they may lead to real-world applications of near-term quantum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the corresponding circuit. At the same time, near-term restrictions limit the depth of the circuit we can expect to run. Thus, the optimization of the VQE ansatz requires maximizing the expressivity of the circuit while maintaining low depth. In recent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem have not yet been extensively investigated. In this work, we propose a reinforcement learning algorithm that autonomously explores the space of possible ansatzes, identifying economic circuits which still yield accurate ground energy estimates. The algorithm uses a feedback-driven curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning algorithm and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of our algorithm on the problem of estimating the ground-state energy of lithium hydride (LiH) in various configurations. In this well-known benchmark problem, we achieve chemical accuracy and state-of-the-art results in terms of circuit depth.

Reinforcement learning for optimization of variational quantum circuit architectures
Mateusz Ostaszewski,
Lea M. Trenkwalder, Institute for Theoretical Physics, University of Innsbruck Innsbruck, Austria
Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Gliwice, Poland.
mm wwwwwsstwwwwsiiccooooooo @gmail. com
lea. www.trewwwwwlerrubbb........ ac ac.
Wojciech Masarczyk,
Eleanor Scerri,
Warsaw University of Technology, Warsaw, Poland.
Leiden University, Leiden, The Netherlands.
wojciech woojcicc.mmsarrrree wtw.lecc..asaarczyk@mmall.. com
scerri@lorenta leidenuniv.nn ml
Vedran Dunjko, Leiden University, Leiden, The Netherlands.
v. v.dunjko@liace haann:ooolaac...eeemmiiiii.....
Abstract
The study of Variational Quantum Eigensolvers (VQEs) has been the spotlight in recent times they may lead to real-world applications of near-term quan- tum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the cor- responding circuit. At the same time, near-term restrictions limit the depth of the circuit we can expect to run. Thus, the optimization of the VQE ansatz requires maximizing the expressivity of the circuit while maintaining low depth. In re- cent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem have not yet been extensively investigated. In this work, we propose reinforcement learning algo- rithm that autonomously explores the space of possible ansatzes, identifying eco- nomic circuits which still yield accurate ground energy estimates. The algorithm uses a bredback-drivers curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning al- gorithm and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of estimating the ground-state energy of lithium hydride (LiH) in various config- urations. In this well-known benchmark problem, we achieve chemical accuracy and state-of-the-ar results in terms of circuit depth.
our algorithm on the problem
Introduction
As we are entering the so-called Noisy Intermediate Scale Quantum (NISQ) [1] technology era, the search for more suitable algorithms under NISQ restrictions is becoming more important. A truly compatible NISQ application must first be amenable to architecture constraints and size limits. Furthermore, minimize the adverse effects of gate errors and decoherence, it is important that the circuits are as gate-frugal and as shallow as possible.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Perhaps the most promising classes of such algorithms are based on variational circuit methods, with which we have high expectations when applied to problems in quantum chemistry. A key problem in this field is the computing of ground state energies and low energy properties of chemical systems (the chemical structure problem). This problem is believed to intractable in general, yet the quantum Variational Quantum Eigensolver (VQE) [2] algorithm can provide solutions in regimes which lie beyond the reach of classical algorithms, while maintaining NISQ-friendly properties [3, 4].
VQE is hybrid quantum-classien algorithm, where a parametrized quantum state is prepared on a quantum computer, the parameters of which are selected using classical optimization methods.
The objective is to prepare the state (x(()) which can be used to approximate the ground state of given Hamiltonian H by the variational principle
Emin S min (nin(((()))))))
(1)
where Emin is the true ground state energy of H. The parametrized state is prepared by applying (()), which is parametrized quantum circuit (typically with a fixed architecture), where the angles Ø- (01...0.) specify the rotation angles of the local unitary rotations present in the circuit. This circuit, known as the ansatz is applied to an initial state (e/o), usually chosen to be the fiducial "all zero" state (00...0), to prepare the state (0) = U(0) wo)-
well established that the structure of the ansatz can dramatically influence the VQE's perfor- mance [4, 5], as the closeness of the estimated ground state to the true one depends on the state manifold accessible by the ansatz. Thus finding new architecture construction methods could lead to breakthroughs in quantum variational methods for chemistry (e.g. for strongly-correlat sys- tems, for which standard ansatzes might fail), but also other domains which utilize variational circuits such as machine learning and optimization [6. 7, 8, 9].
Currently, the foremost ansatzes fall primarily in classes: chemistry-inspire (e-g- the unitary coupled-cluster ansatz [2. 10)) and hardware-inspired (e.g. the hardware efficient ansatz [11]). Ar- chitectures from both of these classes entail using fixed architecture [2, 10, 11, 12] determining the unitary (+). and hence the corresponding ansatz. The ansatz circuit is then usually decomposed into two-qubit CNOT and one-qubit rotation gates parametrized by (0,, 02.0...... 0%) .o........ be optimized by a classical subroutine. However, the architecture itself can also optimized. This results in a hard structure optimization problem, it is combinatorial optimization problem that must balance two competing factors. On the one hand, the ansatz needs to be expressive enough to guarantee good approximation of the ground state energy. On the other hand, the depth and size of the circuit need to be controlled in order for the latter to be compatible with NISQ restrictions.
E [m]" to
Contributions In this work, we propose a general optimization procedure for VQE based on deep reinforcement learning (RL) which is designed to yield quantum circuits that are both gate and depth-efficient. We supplement this RL approach with curriculum learning [13, 14), a powerful machine learning method for solving complex problems by leveraging the solutions of previously- solved simpler instances. Specifically, we introduce a feedback driven curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning algorithm. We apply our architecture to the well-established benchmarking problem of finding the ground state energy of the LiH molecule [15, 3] and observe chemical accuracy while maintaining a low-depth quantum circuit and achieving sate--of-hheart results in terms of gate effi- ciency. Apart from ground state estimation, the proposed method has a wider range of applications, the algorithm can be applied to any variational-cireuit based algorithm.
The rest of the paper is organized as follows. In Section 2, we discuss the related works. In Section 3, we introduce our method with simplistic assumptions about the exact value of the ground state energy. At the end of this section, we provide a method circumventing this unrealistic requirement. As an initial proof-of-concepp in Section 4, we present the results assuming access to the exact value of energy. In Section 5, we provide the results for the full method that utilizes a rough proxy of energy obtained with classical methods. We conclude the paper with a discussion of possible future directions.

---

2 Related Work
The automatic construction of quantum circuits with the use of heuristics/machine learning methods
is a topic that has recently been gaining more attention [16, 17, 18, 19, 20]. In the recent article [21], the authors explore various methods of deep reinforcement learning for quantum architecture search. In contrast to our approach the authors generate circuits which are designed to prepare multi-qubir GHZ states. Several works concern the construction of quantum circuits in the context of quan- tum chemistry. In [22] the authors propose the Quantum Architecture Search algorithm, based on the AutoML. method - Neural Architecure Search, which is benchmarked on the H2 molecule simpler system than LiH. On the other hand, more complex molecules are considered in [23], the authors use evolutionary algorithms to optimize the structure of VQF: circuits for quantum chemistry and combinatorial optimization. The gate set in this algorithm is composed of one-qubit universal gates and two-qubit controlled universal gates, which after the optimization procedure, are decom- posed into CNOT gates and one-qubit universal rotation gates. Another evolutionary strategy was explored in [24], where a multi-objective genetic algorithm is used to optimize the structure of the VQE ansatz in the context of quantum chemistry problems. The topology of the circuits is optimized by a non-dominated sorting genetic algorithm while the parameters are globally tuned by Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [25]. Yet another different approach, explored in the manuscript [26), involved applying a set of rules to both grow and remove quantum gates during the optimization. The last three manuscripts focus solving problems in quantum chemistry by constructing the shortest possible quantum circuits.
3 Methods
In this section, we discuss our approach for constructing VQE ansatzes using RL. We discuss the state and action representations, as well as the reward function used in this work. We then introduce a novel adaptation of curriculum learning applied to VQE circuit construction with different parameter optimization strategies.
3.1 Ansatz optimization as a reinforcement learning problem
We describe ansatz optimization reinforcement learning environment, where the states encode the current circuit. The actions encode gate that is added to the current circuit. The environmental transitions deterministic because the subsequent state is defined by the previous state appended by the taken action. At each time step f, the agent receives the reward according to the following formula:
if Fr < ift > L and 2E, otherwise
R
5
E,-1-E.
max
The goal of the agent is to reach the minimal energy Emin within predefined threshold &. The energy E, is calculated for the current circuit representing the quantum state (v)( according to:
E, ( (O)).
(3)
where o describes the angles of the rotation gates. The vector Ø needs to be separately optimized to obtain the reward R. This optimization subroutine is a crucial hyperparameter of this RL envi- ronment and explained in more detail in the following two sections. The details on the reward shaping are discussed in Supplementary Materials.
We utilize deep RL methods, thus the states and actions are represented in neural network--friendl form. Namely, each state is represented as an ordered list of layers that are composed of single quantum gates. Thus the environmental state, which is represented by this list, fully deseribes the whole circuit (with specified gate ordering). For constructing the circuits, we use CNOT and one qubit rotation gates- which are realizable on currently available quantum devices. The CNOT gates are encoded by two values indicating position of control and targer qubits (the position is enumerated starting from 0). The rotation gates are encoded by two integers (starting from 0). The first number indicates the qubit register and the second specifies the axis of rotation (X 1, Y - 2, 3). We deliberately omit the continuous parameters describing the angles of the rotation gates from the state representation. Instead, the energy estimated for the state is appended to state representation. The
3

---

CNOT Control qubit
4
4
3
4
4
R₂
CNOT Target qubit
0
0
1
0
0
2
Rotation qubit
3
1
4
4
4
R,
Rotation axis
3 1 0
0
0
Example of state representation. In this example, the maximum length of the circuit (L) is set to 5, and the number of qubits to 4. Since we count qubits from O, the lack of particular gate at each layer represented with the maximum number of qubits, which in this case is 4, therefore the last two columns represent layers with identity operators.
Figure I:
user needs to define the maximum number of circuit layers (L). If given circuit has fewer layers than L, the remaining ones are filled with identity operators (c.f. Fig. 1). The agent constructs the circuit iteratively, starting from an empty list, and adding single quantum gate at the end of the current circuit. This way the agent starts each episode with exactly the same conditions, i.e. empty state, avoiding an additional source of randomness tied to the initial state. Thus, the set of all possible actions is equivalent to the set of all possible quantum gates that the agent can select. The size of the set of all possible actions for the agent is then + the number of qubits. The agent can choose from 3[Q] single-qubit gates and from 2(2)) two-qubit gates. We use simple one-hot encoding for the actions.
[Q[(IQ) + 2), where [Q] is
3.2 Specification of the agent and the environment
We employ a Double Deep-Q network [27] (DDQN) with an c-greedy policy and an ADAM opti- mizer for better stability. More technical details about the Reinforcement Learning procedure are described in the Experiments section (Section 4). As described in the previous section, to obtain a reward R for each eircuit, an optimization subroutine needs be applied to determine the continu- ous values of the rotation gate angles. We use well-developed methods for continuous optimization such as Constrained Optimization By Linear Approximation (COBYLA) [28] and Rotosolve [29]. Whilst we could use an agent for this purpose as well, we abstain from doing so for two main rea- sons. Firstly, in this work, we aim at assessing the efficacy RL for circuit structure optimization, rather than for parameter searching. The latter is investigated extensively in other literature and thus treated as a separate subroutine that can be solved by other, more developed, tools. The second rea- son is practical, as training the agent to reliably predict values of angles would result in significantly longer training periods.
For the above reasons, our approach is hybrid: the agent learns how to, and is rewarded for, con- structing particular circuits given set of parameters determined by an independent optimization algorithm. We apply the angle optimization subroutine only after the steps in which the agent ap- pends a rotation gate the circuit, thus eliminating the necessity of including the parameters in the state representation. The number of angles optimized at a given step is a hyperparameter in our method, the choice of which will be analyzed in Sec. 4.2. In this work, we consider optimizing all angles at once (global strategy), well as optimizing a few angles at time (local strategy). These different experimental settings allow us check whether the rough energy approximation obtained from only optimizing subset of the circuit angles is sufficient to create sufficiently good circuits.
3.3 Feedback-driven curriculum learning
As explained, the agent is rewarded for creating circuits that pass given threshold. In our exper- iments, we set the threshold to so-called chemical accuracy, which corresponds to an energy that differs from the ground state energy by less than 0.0016 Hartree. Even in cases where the exact value of minimum energy is known, setting the value of threshold & directly to chemical precision from the beginning of the training may result in the agent failing due to poor exploration. A possible solution is to introduce the agent to tasks of increasing difficulty known curriculum learning [13] which was found be beneficial for neural network training [14]. However, this approach requires the user to design& task sequence which, without priori knowledge of the relationship between

---

E2
Episodes
Figure 2: Illustration of the feedback-driven (orange) procedure, showing the effects of two amor- tization events (blue), the value of which is represented by The first event follows a non-zero change in threshold from & to &,, i.e. the agent manages to find better energy estimate during training. The second amortization event illustrates what happens when the agent fails to improve on the current threshold ε2 (or the improvement is smaller than the amortization value): the threshold increases suddenly due to the resetting of the amortization value. Note that the final threshold, after the second amortization drops to zero, also be lower than Ez-
the threshold value and the corresponding task difficulty, may lead to the agent reaching undesired local minima.
Inspired by the body of works exploring automatic curriculum learning reviewed in RL [30], we propose feedback-driven curriculum learning method in which the difficulty of the current task dynamically adjusted based on the current performance of the agent. Hence, the agent does not rely a human-defined task schedule and builds its knowledge gradually paced by the agent's per- formance. The proposed method, illustrated in Fig. 2, starts with an arbitrary threshold & that is decreased according to two rules. The first rule decreases the threshold greedily the best energy obtained so far &2 after each G episodes. Additionally, the amortization factor ō is added to the threshold the moment of a greedy shift and each time the agent fails solve multiple consec- utive episodes. To gradually decrease the amount of added amortization 5, after few successful episodes we decrease the amortization 8 by a certain factor until the agent reaches the energy that was greedily obtained at first. Since in this feedback-drive -drive curriculum learning method the thresh- old successively decreased, we will also refer to this approach as moving threshold.
Note that due to the dynamic nature of our reward system we can substitute the true value of Emain which is in practical cases unknown, with arbitrary value that is lower than minimum energy Emin and set threshold & to a value that guarantees that the exact energy falls into the range Emin E (a, a + E). Note that, in principle, the method does pose any restrictions the starting threshold and the agent can start learning from any point, so we can set the & arbitrarily big to make sure that the Emin in the range. The a value can also be in the same vein, however, a simple method to roughly estimate the lower bound of energy can be utilized instead. If the agent is able to create circuits with chemical precision relative to any (reasonably) higher energy, the training with ever-increasing difficulty will ultimately lead to the chemical accuracy without computing the exact energy. Experiments proving the validity of this argument are presented in Section 4.2.
4
Experiments
We start this section with a description of the chemical problems on which we evaluate the proposed approach. Next, we will describe the experimental setup and present results.
4.1 Experimental setup
In our analysis of RL for VQE circuit synthesis, we focus the problem of finding the ground state energy of the lithium hydride (LiH) molecule for various intramolecular distances, as well as Hamiltonians on differing qubit numbers, which stem from different approximations of the true chemical problem. All experiments are divided into two parts. the first part, we explain how the choice of optimizer and number of optimized parameters per step impact the performance of the RL
'ee tested multiple hard-coded schedules for shifting the threshold. However, the algorithm did not pass the desired lowest threshold while using these pre-detined sequences of thresholds.
5

---

approach. This part is evaluated on less challenging problem instances, i.c. with smaller quantum system, action space, and state space. The second part, on the other hand, will be performed on the same molecule but simulated on a higher dimensional system. We assume that the number of measurements sufficiently large that sampling noise is negligible. Whilst this a significant assumption to make, and current limitation of our method, in this work we are interested in a proof of concept for RL-based circuit synthesis. Thus we do not include the additional overhead incurred by finite measurements, which is left for future work.
Quantum chemistry problems. As mentioned carlier, in this work we focus on finding the ground state energy of the LiH molecule, although approach design is not limited to this molecule. The Hamiltonians are computed in STO-3G basis. In the first part of our analysis, we consider a simpler approximation of LiH molecule which, after taking into account the symmetry of the molecule and removing the orbitals with weak interaction, results in a Hamiltonian defined on 4 qubits. Thus, the state space and action space significantly smaller than the ones for the full LiH Hamiltonian. In this model, we utilize the parity mapping to convert molecular Hamiltonian to qubit Hamiltonian [31]. We examine proposed method on three values of LiH bond distances, 1.2A, 2.2À and 3.4A. In the second experiment, we use the larger Hamiltonian which only takes into account the symmetry of the molecule, and is therefore defined on 6 qubits, for which we use the Jordan- Jordan-Wigner mapping [32]. We opted to switch from the parity to the Jordan- Wigner mapping when considering the 6 qubit Hamiltonian in order to compare our results with previous literature which tackled this problem using a different approach [23]. We only focus on single geometry for this case, i.e. for bond distance 2.2 All Hamiltonians were generated using the Qiskit library [33].
Implementation ? In all experiments we utilize n-step DDQN algorithm, with the discount factor set 7 = 0.88, and the probability of a random action being selected is set by an 6 e-greedy policy, with decayed in each step by factor of 0.99995 from its initial value = 1, down to a minimal value ε 0.05. The memory replay buffer size is set to 20,000. The target network in the DDQN training procedure is updated after every 500 actions. After each training episode, we included a testing phase where the probability of random action set to e and the experience replay procedure is turned off, i.c. the experiences obtained during the testing phase are not included in the memory replay buffer.
In our experiments, we use differing step sizes in the n-step trajectory rollout updates [34], which specify exactly how the Q-function approximations is set to n = 1 for the first part of the experiments. The reported results correspond to the experiment performed with the value greedily after 2000 episodes with an amortization radius of 0.0001. After 50 successfully solved episodes, the amortization radius is decreased by 0.00001. The initial threshold value is set to & = 0.005. Simulations of quantum circuits were performed using Qulacs library [35] (MIT License). The hyperparameters were selected through coarse grain search. The employed network is fully connected network with 5 hidden layers with 1000 neurons each for the 4-qubit case and 2000 neurons each for the 6-qubit case. The maximal number of gates is equal to 40. All experiments were performed on three computing clusters - 4 Titan RTX GPUs, 4 Titan V GPUs, and 4 Tesla V100 GPUs
updated. This hyperparameter of the model
6. In the moving threshold approach, the threshold is changed
Evaluation. validate the capabilities of the reinforcement learning approach we compare it with well-established architectures, namely the Hardware Efficient (HE) [11] ansatz and UCCSD [36, 37] ansatz. The number of layers of the HE ansatz is tailored to each Hamiltonian considered. We report the smallest number of layers for which we have achieved chemical accuracy using our chosen optimizers. Moreover, the naive approach of UCCSD is used, and it is likely possible to find shallower implementation of this ansatz, but for the purposes of this manuscript, it is an acceptable benchmark.
In all experiments, we compare the minimal depth and the number of gates of the obtained circuits (for more details see Supplementary Materials). By depth, we mean the length of the longest path between input and the output along qubit wires, without taking into account quantum gate depen- dency. Each RL. experiment runs on 10 trials, i.e. experiments run on 10 different random seeds. For the first part of experiments, we consider two optimization methods: Rotosolve and COBYLA.
'The code is available on httt:://ggittab.co////stttssssss ottp://sss...ssoooooooooooooaata wwt.sootttiiiiitn.ommm of _QE_ Aurcuit_architer

---

HE
1.2 avg
2.2 avg
3.1 avg
UCCSD
1.2 min
2.2 min
3.4 min
275
20
30
mumber of gates
depth
-20
10-
10
local
local
global
global Rotosolve
loral COBYLA
local Rotosolve
global
giobal Rotosolve
COBYLA
Rotosolve
COBYLA
COBYLA
Figure 3: Comparison of minimum depth (left) and gate count (right) of the quantum circuits on which chemical accuracy has been achieved for different strategies. The above results are for the 4- qubit LiH Hamiltonian for several intramolecular separations, shown in the legend. The experiments testing the RL approach were run 10 times on different seeds for the networks' parameters and the the
E-greedy random actions. Each bar labeled by "avg" in the legend represents the average results from the different trials, whereas each one labeled by "min" " is the smallest value obtained over the trials. For the local optimization strategy with bond distance 3, 3,4A, the agent found circuits achieving chemical accuracy in two out of ten trials using COB YLA and three out of ten using Rotosolve.
Both of these optimization methods be applied with local strategy, where the optimizer up- dates the last five rotation gates after each agent step, or a global strategy, where after each time step full update is performed. For the local optimization strategy, set the number of available iterations of Rotosolve to 5, whilst for the global strategy, the number of iterations is chosen to be 25. By iterations (for Rotosolve) we mean one complete cycle updating all the parameters under consideration. We chose 100 iterations for COBYLA for both strategies. Due to the fact that the number of iterations is fixed, convergence is not always guaranteed. However, in most cases, these values were sufficient and more iterations did not improve the energy estimate.
In the LiH 4-qubit experiments, were able to establish the strategy and the optimizer use in the LiH experiments with 6-qubits. For the setup with 6-qubit, we chose global optimization each step and COBYLA with 200 iterations.
4.2 Results
LiH - 4-qubit different bond distances. In this section, we discuss the results of the LiH exper- iments with qubits. Fig. 3, the depth (left) and the total number of gates (right) of the cireuits which achieve chemical accuracy are presented, with the average taken over the minimal values from the different trials. For bond distances 1.2Á and 2.2Å, the agent proposed quantum circuits that were shallower and contained fewer gates than the standard approaches, in every trial. For bond distance 3.4A, the agent using local COBYLA optimization found quantum circuits satisfying chemical ac- curacy in 2 out of 10 trials, whereas the agent using local Rotosolve did so in 3 out of 10 trials. On the other hand, when using global optimization, the agent found circuits satisfying chemical accu- racy in every trial, regardless of the optimizer used. In almost all cases, the average minimal depth and gate count were less than those obtained using standard approaches. i.c. HE and UCCSD, the exception being when the agent used local Rotosolve, which resulted in circuits deeper than the HE ansatz. Given the above results, COBYLA seems to outperform Rotosolve for this task. Moreover, taking into account the number of successful trials, conclude that approach is significantly improved when all angles are optimized.
LiH 6-qubits moving threshold. In this section, we discuss the results of the LiH experiments with 6 qubits. We evaluated the fixed threshold approach on the 6-qubit LiH Hamiltonian case,
7

---

which, however, did not give any positive results. The experiments where global optimization after each step was used did not yield any circuits achieving chemical accuracy. This seems to stem from the reward function with the chemical accuracy threshold being too sparse. Whilst gradually de- creasing the threshold the obvious next step, we opted for the curriculum learning approach based on the moving thresholds, in order to decrease the threshold automatically rather than manually. As one can see in Table 1, curriculum agents provide better results than standard approaches. To the best of our knowledge, the only previous work tackling this problem for the 6 qubit LiH Hamiltonian [23]. Whilst we cannot quantitatively compare our results with their work due to differing gate compilations, note out that approach seems to generate circuits roughly 5 times shallower than the ones obtained in [23].
Comparing the minimum depth and the number of gates obtained using the threshold RL approach with those obtained using the standard HE and UCCSD ansatzes for circuits which achieve chemical accuracy. The experiments were performed on the 6-qubit LiH molecule with bond distance 2.2Á. For the RL data, "avg" denotes the average over minimum depths over different trials, whereas "min" denotes the minimum value achieved over all trials. For the standard ansatzes, the minimum depth and number of gates are obviously fixed by the architectures themselves. The RL approach successfully generated circuits achicving chemical accuracy in 7 out of 10 trials.
Table 1:
avg depth
min depth
avg gates
min # gates
RL global COBYLA HE
16
11
35
27
17
17
63
63
UCCSD
377
377
610
610
By looking closely at how the moving threshold guides the agent in the direction of chemical accu- racy, we can analyze the mean difference between the exact energy and the energy estimate using the agent's circuit from the end of each episode. In Fig. 4 we plot this error for particular case at dif- ferent scales understand better what happens different training stages. The blue dots represent the error in the agent energy estimates, whilst the orange curve represents the moving threshold. The left of Fig. 4 shows this error for the entire training period but does not provide much information due to the large deviation in the error. On the other hand, in the right plot of Fig. 4, we focus more on episodes which resulted in errors closer to chemical accuracy. The agent guided by a decreasing moving threshold, with the threshold occasionally increasing abruptly due to amortization, which helps the agent adapt new lower threshold.
exact energy + chemical accuracy
threshold
0.02
0.003
0.002
error
0.01
0.001
0.00
0.000
20000
40000
20000
40000
episodes
episodes
Figure 4: An example of final errors during training on 6-qubit LiH molecule, i.e. the difference between minimal energy and energy obtained from the circuits at the end of each episode, with the blue dots representing the error after every episode and the energy threshold, the orange curve show- ing the energy threshold used to guide the agent, and the red line marking the chemical accuracy. The different plots focus on different error ranges illustrate the effects of decreasing the threshold, as well as the sudden increases in the latter due to the introduction of amortization values.

---

exact energy + chemical accuracy
exact energy
lower bound
threshold
7.842
7.884
energs
7.844
10
7.845
10000
20000 episodes
30000
40000
0
10000
20000 episodes
30000
40000
Figure 5; An example of final energies applied to the moving threshold approach without exact energy knowledge 4-qubit LiH molecule. The blue dots represent the energy estimate from the agent's final circuit at the end of each episode, and the orange curve shows the energy threshold used guide the agent. The agent manages to reach the exact energy (yellow line) within chemical accuracy (red line) despite the provided estimate (purple line) being well below the exact value.
5 Learning procedure with lower-bound approximation to the ground-state energy
So far we have relied on the assumption that the agent uses the energy erro with respect to the exact solution as learning metric. This assumption in practice is of course not reasonable, estimating the energy to this precision is the main objective of VQE algorithms to start with. One way to relax the assumption of knowing the exact energy is to provide reasonable information about the lower and upper bounds, and classical and cheaper methods, respectively, and which incurs only a logarithmic overhead in the precision by using binary search for the best achievable energy. Another solution stems from our moving threshold approach.
The moving threshold approach discussed in the previous sections not only allows to solve more demanding problems, like the 6-qubit Hamiltonian but also removes the requirement knowing the exact enengy. In order demonstrate this method, performed an additional experiment the 4- qubit LiH Hamiltonian at bond distance 2.2Á. Instead of using the exact energy, i.e. R 7.8444Ha, of the absolute values of the Hamiltonian's 's Pauli coefficients, i.e. for
used the negative Hamiltonian H Σوا cy we take Σド= |cyl guide the agent. This is an example of very rough casy-to-compute lower bound on the energy but requires the Hamiltonian be local. For this LiH geometry, this weight sum evaluates to - -10.0604Ha, We set the amortization radius to 0.005, the number of episodes after greedily shifting the threshold 500, and the amortization radius is decreased after every 25 successfully solved episodes. The initial threshold value we set to = 4. We used global COBYLA with 100 iterations to optimize the angles of the quantum circuit.
As one can see in Fig. 5, the agent successfully manages to reach the exact energy within chemical accuracy, despite having lower bound below the exact value. Moreover, it can be that the initial threshold is well above the error that the agent reaches, which also removes the requirement of human input the threshold schedule. Similar experiments were performed on bond distances 1.2A and 3.4A with the same results.
6 Discussion
The molecular electronic structure problem is a promising near-term application for quantum com- puters. The performance of a chosen variational ansatz depends on the structure and depth of its corresponding circuit. Near-term quantum devices pose restrictions on the depth of the VQE circuit. Thus, VQE structure optimization for near-term quantum computers is a multi-objective optimiza- tion problem that requires maximizing the expressivity and reducing the depth of the corresponding

---

circuits. Reinforcement learning by design aims at maximizing the discounted sum of rewards, and as consequence, this will naturally lead finding shorter better- teater-performing solutions. This property of RI. is one of the main reasons we selected this method.
In this work, we present deep reinforcement learning architecture with the feedback-driven cur- riculum learning to optimize the structure of VQE circuits. The goal of this approach is to design circuits that estimate the ground state energy of molecules within chemical accuracy while keep- ing the circuit depth as low as possible. In our approach, instead of confronting the RL agent with the scll-scale problem from the beginning, the agent first starts training on a simpler instance and autonomously adjusts the task complexity until chemical accuracy is achieved. In particular, we demonstrate that our approach yields the ground state energy of LiH within chemical accuracy for several bond distances. While other VQE approaches also reach chemical accuracy for this bench- mark, our approach consistently outperforms the others in terms of circuit depth. Hence, the unique combination of deep RL and feedback-driven curriculum learning for structure learning yields an interesting new approach for solving VQE problems.
We emphasize that whilst we focus specifically on the molecular electronic structure problem for LiH, the reinforcement learning approach presented in this paper can be adapted for optimizing other VQE architectures [38, 7J. The method is also directly compatible with other reinforcement learning algorithms, and the angle optimization algorithms are not restricted the methods investigated in this paper.
In summary, the methods proposed in this work provide state-of-the-art results in electronic structure problem experiments and lay the foundations for the application of deep reinforcement learning methods to VQE optimization problems. Moreover, is often the case when methods employed, we envision numerous follow-up avenues which will reveal the true capacities of such automated ML methods for VQE-type problems.
Broader Impact
Quantum computers may offer significant improvements in chemistry, with applications in drug and material design that could have a widespread positive impact on society (e.g. in the discovery of novel effective pharmaceuticals).
Our work presents novel approaches for enhancing VQE-based methods targeting quantum chem- istry problems and thus contributes to this objective. In particular, this research focuses the use of reinforcement learning to automatically program existing quantum devices. Whilst our work mainly focuses finding the LiH molecule ground state energy, the benefits of such technique extend to research questions that can be reformulated as a VQE optimization problem.
We foresee no negative impact stemming from our research, no significant consequences from sys- tem failures, nor do we believe our methods leverage any bias in any data.
From the energy consumption perspective, we estimate that the single experiment takes from 12 to 48 hours of computation time using single CPU. We did not perform any experiments on QPU machine.
Acknowledgments and Disclosure of Funding
This manuscript builds on the results presented in the NeurIPS workshop paper [39]. MO acknowl- edge the support of the Foundation for Polish Science (FNP) under grant number POIR. 04. 04.000 00-17C1/18-00 LMT acknowledges the support from the Austrian Science Fund (FWF) through the projects DK-ALM-W 1ALLM-W1559.N22 N27 and SFB BeyondC F7102. This work was also supported by the Dutch Research Cownci()WW/OCC)) as part of the Quantum Software Consortium programme (project number 024.003.037). VD and ES acknowledges the support of SURFsara through the QC4QC project. This research was partially funded by the Grant of Priority Research Domain at Warsaw University of Technology Artificial Intelligence and Robotics. The authors also wanted to thank Arthur G. Rattew for helpful discussions.
10

---

References
[1] John Preskill. Quantum computing in the NISQ era and beyond. Quantum, 2:79, 2018.
[2] Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter J Love, Alán Sspuru-Guzik and Jeremy L O'brien. A variational eigenvalue solver on a pho- tonic quantum processor. Nature Communications, 5:4213, 2014.
[3] Abhinav Kandala, Antonio Mezzacapo, Kristan Temme, Maika Takita, Markus Brink, Jerry M Chow, and Jay M Gambetta. Hardware-efficien variational quantum eigensolver for small molecules and quantum magnets. Nature, 549(7671):242- 2222, 2017.
[4] Harper R. Grimsley, Sophia E. Economou, Edwin Barnes, and Nicholas J. Mayball. An adap- tive variational algorithm for exact molecular simulations on a quantum computer. Nature Communications, 10(1):3007, Jul 2019.
[5] Ho Lun Tang, O. Shkolnikov, George S. Barron, Harper R. Grimsley, Nicholas J. Mayhall, Edwin Barnes, and Sophia E. Economou. qubit-ADAPT-VQE: An adaptive algorithm for constructing hardware-efficient ansatze on a quantum processor. arXiv e-prints, November 2019.
[6] Pierre-Lue Dollaire-Demen and Nathan Killoran. Quantum generative adversarial networks. Phys. Rev. A, 98:012324, Jul 2018.
[7] Marcello Benedetti, Erika Lloyd, Stefan Sack, and Mattia Fiorentini. Parameterized quantum circuits as machine learning models. Quantum Science and Technology, 4(4):043001, nov 2019.
[8] Leo Zhou, Sheng- Tao Wang, Soonwon Choi, Hannes Pichler, and Mikhail Lukin. Quantum approximate optimization algorithm: Performance, mechanism, and implementation on near- term devices. Phys. Rev. X. 10:021067, Jun 2020.
[9] Jonathan Romero and Alán Aspuru-Guzik. Variational quantum generators: Generative adversarial quantum machine learning for continuous distributions. Advanced Quantum Technologies, 4(1):2000003, 2021.
[10] M.-H. Yung. J. Casanova, A. Mezzacapo, J. McClean, L. Lamata, A. Aspuru-Guzik, and E. Solano. From transistor to trapped-ion computers for quantum chemistry. Scientific Reports, 4(1):3589, Jan 2014.
[11] Abhinay Kandala, Antonio Mezzacapo, Kristan Temme, Maika Takita, Markus Brink, Jerry M. Chow, and Jay M. Gambetta. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. Nature, 549(7671):242- Sep 2017.
[12] Marcello Benedetti, Mattia Fiorentini, and Michael Lubasch. Hardware-efficient variational quantum algorithms for time evolution, 2020.
[13] Jeffrey I.. Elman. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71-99, 1993.
[14] Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ACM International Conference Proceeding Series, volume 382, pages 1-8, New York, New York, USA, 2009, ACM Press.
[15] IG Csizmadia, BT Suteliffe, and MP Barnett. group orbital study of lithium hydride. Canadian Journal of C 42(7):1645-1663, 1964.
[16] Steven Herbert and Akash Sengupta. Using reinforcement learning to find efficient qubit routing policies for deployment near-term quantum computers. arXiv preprint arXiv:1812.11611 2018.
[17] M Sohaib Alam. Quantum logic gate synthesis as a markov decision process. arXiv preprint Miti:1912.12009 2019.

---

[18] Mohammad Pirhooshyaran and Tamas Terlaky, Quantum circuit design search. arXiv preprint arXiv:2012.04046, 2020.
[19] Thomas Fösel, Murphy Yuezhen Niu, Florian Marquardt, and Li Li. Quantum circuit opti- mization with deep reinforcement learning, 2021.
[20] Shi-Xin Zhang, Chang-Yu Yu Hsieh, Shengyu Zhang, and Hong Yao. Neural predictor based quantum architecture search, arXiv preprint arXiv:2103.065: 2021.
[21] En-Jui Kuo, Yao-Lung I. Fang, and Samuel Yen-Chi Chen. Quantum architecture search via deep reinforcement learning. arXiv preprint arXiv:2104.07715, 2021.
[22] Yuxuan Du, Tao Huang. Shan You, Min-Hsiu Hsich, and Dacheng Tao. Quantum circuit archi- tecture search: error mitigation and trainability enhancement for variational quantum solvers. arXiv preprint arXiv:2010. 10217, 2020.
[23] Arthur G. Rattew, Shaohan Hu, Marco Pistoia, Richard Chen, and Steve Wood. A domain- agnostic, noise-resistant, hardware-efficient evolutionary variational quantum eigensolver. arXiv: Quantum Physics, 2019.
[24] D Chivilikhin, A Samarin, V Ulyantsev, Iorsh, AR Oganov, and O Kyriienko. Mog-vqe: vqe: Mul- tiobjective genetic variational quantum eigensolver. arXiv preprint arXiv:2007.04424 2020.
[25] Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evo-
lution strategies. evol. Comput., 9(2):19--195 June 2001.
[26] M Bilkis, M Cerezo, Guillaume Verdon, Patrick Coles, and Lukasz Cincio. semi- agnossgi ansatz with variable structure for quantum machine learning, arXiv preprint arXiv:2103.06712 2021.
[27] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013.
[28] Michael JD Powell. A direct search optimization method that models the objective and con- straint functions by linear interpolation. In Advances in optimization and numerical analysis, pages 51-67, Springer, 1994.
[29] Mateusz Ostaszewski, Edward Grant, and Marcello Benedetti. Structure optimization for pa- rameterized quantum circuits. Quantum, 5:391, 2021.
[30] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre Yves Oudeyer. Au- tomatic Curriculum Learning For Deep RL: A Short Survey. IJCAI International Joint Conference on Artificial Intelligence, 2021 2021-January:48199445
mar 2020.
[31] Jacob T. Seeley, Martin J. Richard, and Peter J. Love. The bravyi-kitaev transforma- tion for quantum computation of electronic structure. The Journal of Chemical Physics, 137(22):224199, 2012.
[32] Jordan and E. Wigner. Über das paulische "quivalenzverrbot. Zeitschrift für Physik, 4709):331-651, Sep 1928.
[33] Qiskit: An open-source framework for quantum computing, 2019.
[34] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cam- bridge, MA, second edition, 2018
[35] Yasunari Suzuki, Yoshiaki Kawase, Yuya Masumura, Yuria Hiraga, Masahiro Nakadai, Jiabao Chen, Ken M Nakanishi, Kosuke Mitarai, Ryosuke Imai, Shiro Tamiya, et al. Qulacs: a fast and versatile quantum circuit simulator for research purpose. arXiv preprint arXiv:2011.13522 2020.
[36] Mark R. Hoffmann and Jack Simons. A unitary multiconfigurational coupled-cluster method: Theory and applications. The Journal of Chemical Physics, (5(2):993-1002 1988.
12

---

[37] Alternative coupled-cluster ansätze ii. the unitary coupled- coupled-cluster clluster method. Letters, 155(1):133 140, 1989.
Chemical Physics
[38] Marco Cerezo, Andrew Arrasmith, Ryan Babbush, Simon Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, al. Variational quan- tum algorithms. arXiv preprint arXiv:2012.09265, 2020.
[39] Mateusz Ostaszewski, Wojciech Masarezyk, Lea Trenkwalder, Eleanor Scerri, and Vedran Dunjko. Curriculum reinforcement learning for optimization of variational quantum circuit chitectures. NeurlPS 2020, Third Workshop on Machine Learning and the Physical Sciences, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See the abstract and introduction
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Did you describe the limitations of your work? [Yes] See Section 4.1
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] Code and instructions are in the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Figure 3
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Section paragraph "Implementation details."
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4 para- graphs "Quantum chemistry problems." and "Implementation details."
(b) Did you mention the license of the assets? [Yes] See Section 4 paragraphs "Quantum
chemistry problems." " and "Implementation details."
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'te re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "w-L165gde57", "reviewer": "Reviewer_uF7i", "summary": "# Overview\n\nThe paper proposes a framework incorporating deep reinforcement learning for resolving Ansatz optimization problems, which is with wide impacts on quantum chemistry. The authors have a good introduction and related work section, which covers most standing works in quantum reinforcement learning and variational circuit learning. \n\nCompared with Rotosolve and COBYLA, the proposed DDQN-based method attains state-of-the-art results on estimating the ground-state energy of lithium hydride.\n\nOverall, I like the idea but the baseline comparison and some time complexity could better incorporate to add the depth of this paper. The current version is more application-oriented and less novel as a general ML framework considering existing works in the neural architecture search (NAS) community. \n\n- Justification of DDQN and comparison \n\nWhen discussing \"why selecting DDQN for the RL\", there is very little discussion. \n\nSince this work is not the first works on applying the heuristic method for quantum circuit architectures search (QCAS), the contribution is more on framing the Ansatz optimization and the benchmark results. It would be more valuable to address the selection and justification of the algorithm. (e.g., DRL-based vs Neuroevolution-based or at least the variants inside DQNs)\n\nFor the general audience in the NeurIPS community, instead of only showing the DDQN-based method for QCAS, providing more in-depth discussion between variants of the DRL algorithm (e.g., DQN, DQN with dueling, DDQN with dueling) and its complexity could gain better values of this paper. \n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n- The framing of using architecture optimization for VQC on the specific Ansatz optimization is with a wide impact on Quantum chemistry and biology.\n\n- Compared with a sequential trust-region algorithm (COBYLA) and gradient-based Rotosolv; the proposed method shows a new state-of-the-art performance\n\n- I like the discussion and motivation in the introduction and related sections.\n\n\n## Cons\n\n- The baseline could be more comprehensive considering (1) the variants insides DQNs and its time complexity discussion or (2) in a great impact of discrete optimization from gradient-free method (e.g., Neuroevolution) and DRL, where Neuroevolution is much possible to find the global minimum. [R1]\n\n- The training time and algorithm complexity needs to be added to the discussion \n\n- When the current quantum circuit is running on CPU-simulated QPU, it would be better to show the proposed framework is also working for QPU machines with little quantum error correction. \n\n- The selection of DDQN and its writing could be further motivated. \n\n## Recommendation\n\nOverall, I think this is a submission between borderline and reject threshold. \n\nConsidering the Pros and Cons, I give a borderline rejection on this work. \n\n- The reviewer encourages the review to improve their baseline discussion and provide more in-depth discussion to the general ML community. \n\n## Discussion \n\n- How robust is this searching algorithm compared with the (1) Rotosolve and COBYLA (2) neural-network-based discrete searching methods (DQNs and Neuroevolutons [R1 to R3]) with the same model capacity. (e.g., the same amount of trainable parameters). If (a) machine noise affects the circuit or (b) additive random noise affects Ansatz optimization?\n\nR1. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning, NeurIPS workshop 2018\n\nR2. Evolving Neural Networks through a Reverse Encoding Tree, IEEE Congress on Evolutionary Computation (CEC) 2020\n\nR3. Deep neuroevolution of recurrent and discrete world models, ACM GECCO 2019\n\n## Minors\n\n- Figure 4 is not properly cut (e.g., threshold). Besides, Please use pdf figure if it is possible. \n\n- please improve the quality of the figures in general \n\n### Post-Rebuttal\n\nI have read the author's response. My concerns on the baseline studies still stand. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- When the checklist is not fully passed, the authors may consider adding a board impact discussion section in their revised version\n\nDid you discuss any potential negative social impacts of your work? -> No\n\n- As a QML paper, the energy of simulation cost of CPU hours and QPU cost could be further incorporated to echo the recent NeurIPS policy. \n\nFor 2 (a) and (b) in the checklist, the paper would select [N/A] instead of [No] ", "review_text": "", "extra_scores": {"ethical_concerns": "The current experiment is running on CPU-based simulation. No major concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "Nv-JFE8UvrF", "reviewer": "Reviewer_qT3k", "summary": "The variational quantum eigensolver (VGE) is a method that utilizes a Noisy Intermediate Scale Quantum (NISQ) computer to find the ground state of quantum systems. At its core, an Ansatz for the ground-state wave-function is parameterized in terms of a quantum circuit, i.e. a series of single-qubit and two-qubit gates acting on a reference state, and the optimal parameters of the circuit are determined by a classical algorithm that uses the NISQ to evaluate the performance (energy) of the Ansatz.In this manuscript, the authors propose a reinforcement learning (RL) based approach to identify the small circuits (i.e. with the low number of gates, and thus less prone to errors) that still allow to reach chemical accuracy. The RL method is based on double deep-Q learning and curriculum learning. The authors employ their method to find the ground state energy of LiH approximating the electronic Hamiltonian as a 4 or 6 qubits Hamiltonian. For 3 different bond distances, the method proposed by the authors identifies circuits that achieve chemical accuracy, but whose size is lower as compared to other Ansatze", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel method, based on reinforcement learning, to identify the smallest circuits that achieve chemical accuracy. The problem is relevant and timely. While the overallpresentation of the method and of the results is clear, it could be improved in thefollowing three ways:\n\n- The specific choice of the reward \n\n- a crucial ingredient in reinforcement learning– is provided in Eq. (2), but it is not commented. The only comment, given in line 102, states that “The goal of the agent is to reach the minimal energy E_min within a predefined threshold \\xi.” However, this does not explain why the agent should identify small circuits. Since the agent actually tries to minimize the discounted sum of the rewards, it would be beneficial to the reader to comment how such a choice of the reward allows the agent to find small circuits, and perhaps also to justify the apparently arbitrary value “+5” and “-5” in the reward. (this can be omitted)\n- Related to the previous question, in Fig. 3 the authors quantify the performanceof their method using both the “depth” of the circuit, and the \n“number of gates”.While they seem roughly proportional, the authors should comment as to which one of these two figures of merit is actually being minimized by the RL agent.\n\n- Some implementational details of the RL method are missing, such as the architecture and size of the neural network used to parameterize the Q-value functions, and a comment of the choice of L for the various results presented.\n\nHowever, my main concern regards the actual applicability of such method. As the authors correctly point out, it is crucial to identify small circuits in the NISQ era. However, as shown in figures 4 and 5, the method requires order 10s of thousands of episodes, and each step of each episode further requires an optimization over the theta parameters using a separate algorithm (such as Rotosolve or COBYLA), which in turn requires tens or hundreds of iterations. Furthermore, such experiments are carried out for 10 trials, since not all trials converge. Therefore, the total number of energy calculations that the NISQ computer must perform seems much higher than what would be required using standard Ansatzes (where it is only required to optimize over the theta parameters, which presumably can be performed with a computational cost similar to a single optimization using Rotosolve or COBYLA). Could the authors comment such observation? Would a RL method that optimizes also over theta help to decrease such computational cost?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NQVSoNQt0kf", "reviewer": "Reviewer_PYKT", "summary": "This paper studies the ansatz for Variational Quantum Eigensolver (VQE) quantum circuits. The authors propose a deep reinforcement learning framework to generate the ansatz for VQE circuit, aiming to achieve low estimated energy and shallow circuit depth. The authors leverage curriculum learning to gradually reduce the energy threshold (increase difficulty) to avoid learning failure. The method is evaluated on LiH molecule in different settings and shows shallower circuit than baseline HE and UCCSD circuits. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is well-written and fairly easy to follow. Furthermore, the proposed RL framework is technically sound and has achieved decent empirical results on the benchmark molecule.\n\n2. The problem is a two-level optimization one. The outer level is to design the ansatz, inner level is to train the parameters (angle of rotation gates). My concern is that the RL framework is too time-consuming because reward of each episode requires full training of the circuit once again. According to the experiments, the episodes are larger than 40,000, so how long does it take to obtain the results in paper?\n\n3. When the molecule requires more qubits, how good is the scalability of the proposed method? The cost of training parameters is increasing exponentially, and large number of RL episodes is unlikely affordable.\n\n4. For curriculum learning, the author qualitatively claim that the fixed threshold and hard-coded threshold schedules cannot provide positive results. More quantitative comparisons such as learning curves are helpful. It is a little surprising that no hard-coded schedule can yield positive results. What kinds of schedules have you tried?\n\n5. The rationale behind reward function (2) and curriculum learning is not convincing. For my understanding, if the L is large enough, the Et will ultimately be smaller than $\\xi$. Why not gradually reduce the L (with hard-coded schedule)? Why use L instead of gate count as the criterion?\n\n6. Though it is generally assumed that larger L brings larger noise impact. Some results on real QC device comparing HE, UCCSD and proposed method can make the work more solid.\n\n7. There are several existing manuscripts on optimizing VQE circuit ansatz (some of them also use RL), some qualitative/quantitative comparisons are helpful. \n\nReferences:\n(1) Du, Yuxuan, et al. \"Quantum circuit architecture search: error mitigation and trainability enhancement for variational quantum solvers.\" arXiv preprint arXiv:2010.10217 (2020).\n(2) Kuo, En-Jui, Yao-Lung L. Fang, and Samuel Yen-Chi Chen. \"Quantum Architecture Search via Deep Reinforcement Learning.\" arXiv preprint arXiv:2104.07715 (2021).\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not have noticeable negative societal impact. As for limitations, the RL framework may not be scalable enough for large quantum circuits.", "review_text": "", "extra_scores": {"ethical_concerns": "The paper does not have noticeable negative ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "ERRLenxXflf", "reviewer": "Reviewer_KxuB", "summary": "This work demonstrates that reinforcement learning with intrinsic motivation can be effectively used to search over the space of ansatzes for the variational quantum eigensolver (VQE) algorithm. This algorithm is used in quantum chemistry to approximate the ground state energy of molecules (e.g. LiH). The author(s) describe how DDQN trained with Adam can learn to optimize a VQE-relevant reward function, in an environment which runs an optimization subroutine (COBYLA or rotosolve) at each step to optimize the rotation angles of the ansatz's gates. Their LiH experiments used a simulated noiseless quantum environment, and they benchmarked the performance against two well-known ansatzes. The results show that in simulation, RL has promising potential for circuit depth improvement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality: \nThe originality of this work lies in the novel application of well-known reinforcement learning techniques, to the contemporary variational quantum eigensolver technique used in quantum chemistry.\nAppears to be missing seemingly relevant open-source works which include discussion of variational quantum circuit design. A cursory Google search finds multiple works which discuss VQE and RL, some discussion about these works would be nice. Due to the lack of this discussion, its not clear if this work is standing absolutely outside of previously conducted research or how it compares otherwise. Nonetheless, overall originality - good.\n\nQuality:\nThe paper is light on technical method description. Although both RL and VQE are well known among experts in their respective fields, this work would benefit from a more detailed overview. The claims however do appear to be supported by the author(s) in-house experiments. The methods are used appropriately and the authors are honest about the strengths and weaknesses of their results. Overall quality - acceptable but could be improved.\n\nClarity:\nThe paper overall is quite readable, likely even for those who have little exposure to quantum topics (i.e. VQE). The organization is logical and straightforward. Since the author(s) experiments are of central importance for this work, and since the technical implementation details of those experiments has been (probably necessarily) omitted, some form of releasing the associated code (e.g. GitHub) would help inform the interested reader who is not privy to the reviewer's supplementary material. Plus, this would help make the results reproducible. At minimum, it would benefit by showing an algorithm covering psuedocode for the overall training process. Overall clarity - acceptable.\n\nSignificance:\nThe demonstration that RL has promising potential for shortening quantum circuit depth is an important result. Similar results have been shown on the front of using RL to improve optimization passes during real-device quantum circuit compilation. In my personal opinion, I believe other researchers in the field of quantum chemistry will find these results useful. To my knowledge, this paper does present state of the art results for this particular problem. Overall significance - moderate to high in the quantum field, acceptable in the applied ML field.\n\nOther comments and questions:\nJudging from the reward function definition Eqn. (2), it seems like once E_t is less than \\xi, the RL agent could continuously suggest the identity gate to receive higher rewards. This would waste an arbitrary amount of circuit depth, performing the identity over and over.\n\nIts not very clear what is \"intrinsic\" about the reward function threshold decreasing according to a schedule. As far as my understanding goes, intrinsic motivation encapsules rewards/policies which are thought to be baseline useful, but not directly related to the \"extrinsic\" reward/task at hand. For an unrelated example, a legged-robot may be given an \"extrinsic\" reward for crossing a room quickly, and a separate intrinsic reward for making coordinated movements, regardless of its progress across the room. By supplementing the extrinsic task with an intrinsic reward to be coordinated, learning policies which exhibit graceful motion are more easily learned, in principle.\n\nProviding the DDQN network structure would be nice.\n\nLine 319 is a misleading statement. RL by design does not aim at obtaining the shortest optimal solution (In fact, it seems like there's incentive for the circuit to be longer here to collect more rewards -- see the comment on repeating the identity gate above).\nRL is actually a quite roundabout method for obtaining a solution to an optimization problem.\nBy design, a reinforcement learning agent is tasked to maximize some form of return, which may or may not translate to a good solution to the problem at hand (e.g. a combinatorial optimization problem).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work were discussed, and there does not appear to be any clear negative societal impact of improving the state of knowledge in computational quantum chemistry.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is the first demonstration of (1) deep RL for variational optimization of quantum circuits which (2) discovers a circuit which demonstrates high accuracy on a "real world" problem - computing the ground state energy of lithium hydride - using fewer gates than other approaches. While both (1) and (2) have been done before, none of the reviewers have convincingly argued that the combination has not been done before. There was significant disagreement between the reviewers and I believe some misunderstanding over the novelty of the approach. Many of the reviewers would have liked to see experiments with a wider variety of deep RL algorithms. While this would certainly make the paper stronger, and is justified for a well-studied benchmark problem, an exhaustive comparison between closely related methods is not always needed for a novel application, as Reviewer KxuB points out.

Others pointed out that the application is not entirely novel - other papers have looked at optimizing circuits for quantum chemistry applications using non-DRL methods (arXiv:2010.10217) and have looked at optimizing quantum circuits for other applications using DRL (arXiv:2104.07715). On more closely inspecting these papers, it is clear that the first one is looking at simpler systems than this paper (the hydrogen molecule, with two electrons, rather than lithium hydride, with 4 electrons) and the second paper is only looking at simple 2 or 3 qubit toy systems (Bell states and GHZ states). Therefore this paper is an advance over the previous state of the art simply by virtue of looking at more difficult systems. The authors should rewrite the paper to emphasize this aspect of the novelty of their work and properly contextualize their work in the wider context of the field, but this should be straightforward to do.

Many reviewers raised concerns about scaling, noting that running 40,000 iterations on real quantum hardware would be extremely challenging. This is a valid concern, and one which is common to other applications of deep RL where generating data on real hardware is expensive (e.g. robotics). However, I don't believe it should be disqualifying for the paper. Training deep RL models in simulation and then transferring to real hardware is an active area of research, and deep RL papers are published all the time which only work in simulated settings like OpenAI Gym but would be impractical to run in a realistic amount of time on real hardware.

Relatedly, reviewers also raised concerns about how the learned circuits were not evaluated on real hardware. A closely related paper (arXiv:2010.10217) was evaluated on real hardware, and found a significant decline in performance due to noise. I expect something similar would be seen with this method, and the authors should be sure to discuss this in the paper. But, despite the availability of some small research devices on some cloud platforms, these devices are still far from widespread and are very experimental. While it would certainly make the paper much stronger, I don't think experiments on real hardware should be a requirement for a paper published at a machine learning venue. That would put authors working on quantum algorithms at an unfair disadvantage compared to those working on ML algorithms that run on conventional hardware.

Overall, while there were many valid criticisms of the paper, I think that the paper should be accepted based on the novelty of the results. This novelty could be better explained by the authors, and I hope that they get a chance to read these comments and improve the paper based on them. Rigorous comparison against more RL baselines and other quantum architecture search methods and experiments on real quantum hardware would make this paper significantly stronger - but if those were present, then I would likely be recommending this paper for a spotlight rather than simply a poster.
Decision: Accept (Poster)