Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The variational quantum eigensolver (VGE) is a method that utilizes a Noisy Intermediate Scale Quantum (NISQ) computer to find the ground state of quantum systems. At its core, an Ansatz for the ground-state wave-function is parameterized in terms of a quantum circuit, i.e. a series of single-qubit and two-qubit gates acting on a reference state, and the optimal parameters of the circuit are determined by a classical algorithm that uses the NISQ to evaluate the performance (energy) of the Ansatz.In this manuscript, the authors propose a reinforcement learning (RL) based approach to identify the small circuits (i.e. with the low number of gates, and thus less prone to errors) that still allow to reach chemical accuracy. The RL method is based on double deep-Q learning and curriculum learning. The authors employ their method to find the ground state energy of LiH approximating the electronic Hamiltonian as a 4 or 6 qubits Hamiltonian. For 3 different bond distances, the method proposed by the authors identifies circuits that achieve chemical accuracy, but whose size is lower as compared to other Ansatze
main_review: The authors propose a novel method, based on reinforcement learning, to identify the smallest circuits that achieve chemical accuracy. The problem is relevant and timely. While the overallpresentation of the method and of the results is clear, it could be improved in thefollowing three ways:

- The specific choice of the reward 

- a crucial ingredient in reinforcement learning– is provided in Eq. (2), but it is not commented. The only comment, given in line 102, states that “The goal of the agent is to reach the minimal energy E_min within a predefined threshold \xi.” However, this does not explain why the agent should identify small circuits. Since the agent actually tries to minimize the discounted sum of the rewards, it would be beneficial to the reader to comment how such a choice of the reward allows the agent to find small circuits, and perhaps also to justify the apparently arbitrary value “+5” and “-5” in the reward. (this can be omitted)
- Related to the previous question, in Fig. 3 the authors quantify the performanceof their method using both the “depth” of the circuit, and the 
“number of gates”.While they seem roughly proportional, the authors should comment as to which one of these two figures of merit is actually being minimized by the RL agent.

- Some implementational details of the RL method are missing, such as the architecture and size of the neural network used to parameterize the Q-value functions, and a comment of the choice of L for the various results presented.

However, my main concern regards the actual applicability of such method. As the authors correctly point out, it is crucial to identify small circuits in the NISQ era. However, as shown in figures 4 and 5, the method requires order 10s of thousands of episodes, and each step of each episode further requires an optimization over the theta parameters using a separate algorithm (such as Rotosolve or COBYLA), which in turn requires tens or hundreds of iterations. Furthermore, such experiments are carried out for 10 trials, since not all trials converge. Therefore, the total number of energy calculations that the NISQ computer must perform seems much higher than what would be required using standard Ansatzes (where it is only required to optimize over the theta parameters, which presumably can be performed with a computational cost similar to a single optimization using Rotosolve or COBYLA). Could the authors comment such observation? Would a RL method that optimizes also over theta help to decrease such computational cost?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies the ansatz for Variational Quantum Eigensolver (VQE) quantum circuits. The authors propose a deep reinforcement learning framework to generate the ansatz for VQE circuit, aiming to achieve low estimated energy and shallow circuit depth. The authors leverage curriculum learning to gradually reduce the energy threshold (increase difficulty) to avoid learning failure. The method is evaluated on LiH molecule in different settings and shows shallower circuit than baseline HE and UCCSD circuits. 
main_review: 1. The paper is well-written and fairly easy to follow. Furthermore, the proposed RL framework is technically sound and has achieved decent empirical results on the benchmark molecule.

2. The problem is a two-level optimization one. The outer level is to design the ansatz, inner level is to train the parameters (angle of rotation gates). My concern is that the RL framework is too time-consuming because reward of each episode requires full training of the circuit once again. According to the experiments, the episodes are larger than 40,000, so how long does it take to obtain the results in paper?

3. When the molecule requires more qubits, how good is the scalability of the proposed method? The cost of training parameters is increasing exponentially, and large number of RL episodes is unlikely affordable.

4. For curriculum learning, the author qualitatively claim that the fixed threshold and hard-coded threshold schedules cannot provide positive results. More quantitative comparisons such as learning curves are helpful. It is a little surprising that no hard-coded schedule can yield positive results. What kinds of schedules have you tried?

5. The rationale behind reward function (2) and curriculum learning is not convincing. For my understanding, if the L is large enough, the Et will ultimately be smaller than $\xi$. Why not gradually reduce the L (with hard-coded schedule)? Why use L instead of gate count as the criterion?

6. Though it is generally assumed that larger L brings larger noise impact. Some results on real QC device comparing HE, UCCSD and proposed method can make the work more solid.

7. There are several existing manuscripts on optimizing VQE circuit ansatz (some of them also use RL), some qualitative/quantitative comparisons are helpful. 

References:
(1) Du, Yuxuan, et al. "Quantum circuit architecture search: error mitigation and trainability enhancement for variational quantum solvers." arXiv preprint arXiv:2010.10217 (2020).
(2) Kuo, En-Jui, Yao-Lung L. Fang, and Samuel Yen-Chi Chen. "Quantum Architecture Search via Deep Reinforcement Learning." arXiv preprint arXiv:2104.07715 (2021).



limitations_and_societal_impact: The paper does not have noticeable negative societal impact. As for limitations, the RL framework may not be scalable enough for large quantum circuits.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: # Overview

The paper proposes a framework incorporating deep reinforcement learning for resolving Ansatz optimization problems, which is with wide impacts on quantum chemistry. The authors have a good introduction and related work section, which covers most standing works in quantum reinforcement learning and variational circuit learning. 

Compared with Rotosolve and COBYLA, the proposed DDQN-based method attains state-of-the-art results on estimating the ground-state energy of lithium hydride.

Overall, I like the idea but the baseline comparison and some time complexity could better incorporate to add the depth of this paper. The current version is more application-oriented and less novel as a general ML framework considering existing works in the neural architecture search (NAS) community. 

- Justification of DDQN and comparison 

When discussing "why selecting DDQN for the RL", there is very little discussion. 

Since this work is not the first works on applying the heuristic method for quantum circuit architectures search (QCAS), the contribution is more on framing the Ansatz optimization and the benchmark results. It would be more valuable to address the selection and justification of the algorithm. (e.g., DRL-based vs Neuroevolution-based or at least the variants inside DQNs)

For the general audience in the NeurIPS community, instead of only showing the DDQN-based method for QCAS, providing more in-depth discussion between variants of the DRL algorithm (e.g., DQN, DQN with dueling, DDQN with dueling) and its complexity could gain better values of this paper. 


main_review: ## Pros

- The framing of using architecture optimization for VQC on the specific Ansatz optimization is with a wide impact on Quantum chemistry and biology.

- Compared with a sequential trust-region algorithm (COBYLA) and gradient-based Rotosolv; the proposed method shows a new state-of-the-art performance

- I like the discussion and motivation in the introduction and related sections.


## Cons

- The baseline could be more comprehensive considering (1) the variants insides DQNs and its time complexity discussion or (2) in a great impact of discrete optimization from gradient-free method (e.g., Neuroevolution) and DRL, where Neuroevolution is much possible to find the global minimum. [R1]

- The training time and algorithm complexity needs to be added to the discussion 

- When the current quantum circuit is running on CPU-simulated QPU, it would be better to show the proposed framework is also working for QPU machines with little quantum error correction. 

- The selection of DDQN and its writing could be further motivated. 

## Recommendation

Overall, I think this is a submission between borderline and reject threshold. 

Considering the Pros and Cons, I give a borderline rejection on this work. 

- The reviewer encourages the review to improve their baseline discussion and provide more in-depth discussion to the general ML community. 

## Discussion 

- How robust is this searching algorithm compared with the (1) Rotosolve and COBYLA (2) neural-network-based discrete searching methods (DQNs and Neuroevolutons [R1 to R3]) with the same model capacity. (e.g., the same amount of trainable parameters). If (a) machine noise affects the circuit or (b) additive random noise affects Ansatz optimization?

R1. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning, NeurIPS workshop 2018

R2. Evolving Neural Networks through a Reverse Encoding Tree, IEEE Congress on Evolutionary Computation (CEC) 2020

R3. Deep neuroevolution of recurrent and discrete world models, ACM GECCO 2019

## Minors

- Figure 4 is not properly cut (e.g., threshold). Besides, Please use pdf figure if it is possible. 

- please improve the quality of the figures in general 

### Post-Rebuttal

I have read the author's response. My concerns on the baseline studies still stand. 


limitations_and_societal_impact: - When the checklist is not fully passed, the authors may consider adding a board impact discussion section in their revised version

Did you discuss any potential negative social impacts of your work? -> No

- As a QML paper, the energy of simulation cost of CPU hours and QPU cost could be further incorporated to echo the recent NeurIPS policy. 

For 2 (a) and (b) in the checklist, the paper would select [N/A] instead of [No]
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work demonstrates that reinforcement learning with intrinsic motivation can be effectively used to search over the space of ansatzes for the variational quantum eigensolver (VQE) algorithm. This algorithm is used in quantum chemistry to approximate the ground state energy of molecules (e.g. LiH). The author(s) describe how DDQN trained with Adam can learn to optimize a VQE-relevant reward function, in an environment which runs an optimization subroutine (COBYLA or rotosolve) at each step to optimize the rotation angles of the ansatz's gates. Their LiH experiments used a simulated noiseless quantum environment, and they benchmarked the performance against two well-known ansatzes. The results show that in simulation, RL has promising potential for circuit depth improvement.
main_review: 
Originality: 
The originality of this work lies in the novel application of well-known reinforcement learning techniques, to the contemporary variational quantum eigensolver technique used in quantum chemistry.
Appears to be missing seemingly relevant open-source works which include discussion of variational quantum circuit design. A cursory Google search finds multiple works which discuss VQE and RL, some discussion about these works would be nice. Due to the lack of this discussion, its not clear if this work is standing absolutely outside of previously conducted research or how it compares otherwise. Nonetheless, overall originality - good.

Quality:
The paper is light on technical method description. Although both RL and VQE are well known among experts in their respective fields, this work would benefit from a more detailed overview. The claims however do appear to be supported by the author(s) in-house experiments. The methods are used appropriately and the authors are honest about the strengths and weaknesses of their results. Overall quality - acceptable but could be improved.

Clarity:
The paper overall is quite readable, likely even for those who have little exposure to quantum topics (i.e. VQE). The organization is logical and straightforward. Since the author(s) experiments are of central importance for this work, and since the technical implementation details of those experiments has been (probably necessarily) omitted, some form of releasing the associated code (e.g. GitHub) would help inform the interested reader who is not privy to the reviewer's supplementary material. Plus, this would help make the results reproducible. At minimum, it would benefit by showing an algorithm covering psuedocode for the overall training process. Overall clarity - acceptable.

Significance:
The demonstration that RL has promising potential for shortening quantum circuit depth is an important result. Similar results have been shown on the front of using RL to improve optimization passes during real-device quantum circuit compilation. In my personal opinion, I believe other researchers in the field of quantum chemistry will find these results useful. To my knowledge, this paper does present state of the art results for this particular problem. Overall significance - moderate to high in the quantum field, acceptable in the applied ML field.

Other comments and questions:
Judging from the reward function definition Eqn. (2), it seems like once E_t is less than \xi, the RL agent could continuously suggest the identity gate to receive higher rewards. This would waste an arbitrary amount of circuit depth, performing the identity over and over.

Its not very clear what is "intrinsic" about the reward function threshold decreasing according to a schedule. As far as my understanding goes, intrinsic motivation encapsules rewards/policies which are thought to be baseline useful, but not directly related to the "extrinsic" reward/task at hand. For an unrelated example, a legged-robot may be given an "extrinsic" reward for crossing a room quickly, and a separate intrinsic reward for making coordinated movements, regardless of its progress across the room. By supplementing the extrinsic task with an intrinsic reward to be coordinated, learning policies which exhibit graceful motion are more easily learned, in principle.

Providing the DDQN network structure would be nice.

Line 319 is a misleading statement. RL by design does not aim at obtaining the shortest optimal solution (In fact, it seems like there's incentive for the circuit to be longer here to collect more rewards -- see the comment on repeating the identity gate above).
RL is actually a quite roundabout method for obtaining a solution to an optimization problem.
By design, a reinforcement learning agent is tasked to maximize some form of return, which may or may not translate to a good solution to the problem at hand (e.g. a combinatorial optimization problem).

limitations_and_societal_impact: The limitations of the work were discussed, and there does not appear to be any clear negative societal impact of improving the state of knowledge in computational quantum chemistry.
review_title: null

Meta-review: This paper is the first demonstration of (1) deep RL for variational optimization of quantum circuits which (2) discovers a circuit which demonstrates high accuracy on a "real world" problem - computing the ground state energy of lithium hydride - using fewer gates than other approaches. While both (1) and (2) have been done before, none of the reviewers have convincingly argued that the combination has not been done before. There was significant disagreement between the reviewers and I believe some misunderstanding over the novelty of the approach. Many of the reviewers would have liked to see experiments with a wider variety of deep RL algorithms. While this would certainly make the paper stronger, and is justified for a well-studied benchmark problem, an exhaustive comparison between closely related methods is not always needed for a novel application, as Reviewer KxuB points out.

Others pointed out that the application is not entirely novel - other papers have looked at optimizing circuits for quantum chemistry applications using non-DRL methods (arXiv:2010.10217) and have looked at optimizing quantum circuits for other applications using DRL (arXiv:2104.07715). On more closely inspecting these papers, it is clear that the first one is looking at simpler systems than this paper (the hydrogen molecule, with two electrons, rather than lithium hydride, with 4 electrons) and the second paper is only looking at simple 2 or 3 qubit toy systems (Bell states and GHZ states). Therefore this paper is an advance over the previous state of the art simply by virtue of looking at more difficult systems. The authors should rewrite the paper to emphasize this aspect of the novelty of their work and properly contextualize their work in the wider context of the field, but this should be straightforward to do.

Many reviewers raised concerns about scaling, noting that running 40,000 iterations on real quantum hardware would be extremely challenging. This is a valid concern, and one which is common to other applications of deep RL where generating data on real hardware is expensive (e.g. robotics). However, I don't believe it should be disqualifying for the paper. Training deep RL models in simulation and then transferring to real hardware is an active area of research, and deep RL papers are published all the time which only work in simulated settings like OpenAI Gym but would be impractical to run in a realistic amount of time on real hardware.

Relatedly, reviewers also raised concerns about how the learned circuits were not evaluated on real hardware. A closely related paper (arXiv:2010.10217) was evaluated on real hardware, and found a significant decline in performance due to noise. I expect something similar would be seen with this method, and the authors should be sure to discuss this in the paper. But, despite the availability of some small research devices on some cloud platforms, these devices are still far from widespread and are very experimental. While it would certainly make the paper much stronger, I don't think experiments on real hardware should be a requirement for a paper published at a machine learning venue. That would put authors working on quantum algorithms at an unfair disadvantage compared to those working on ML algorithms that run on conventional hardware.

Overall, while there were many valid criticisms of the paper, I think that the paper should be accepted based on the novelty of the results. This novelty could be better explained by the authors, and I hope that they get a chance to read these comments and improve the paper based on them. Rigorous comparison against more RL baselines and other quantum architecture search methods and experiments on real quantum hardware would make this paper significantly stronger - but if those were present, then I would likely be recommending this paper for a spotlight rather than simply a poster.
Decision: Accept (Poster)