Reinforcement learning for optimization of variational quantum circuit architectures

1 Introduction . As we are entering the so-called Noisy Intermediate Scale Quantum ( NISQ ) [ 1 ] technology era , the search for more suitable algorithms under NISQ restrictions is becoming ever more important . A truly compatible NISQ application must first be amenable to architecture constraints and size limits . Furthermore , to minimize the adverse effects of gate errors and decoherence , it is important that the circuits are as gate-frugal and as shallow as possible . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Perhaps the most promising classes of such algorithms are based on variational circuit methods , with which we have high expectations when applied to problems in quantum chemistry . A key problem in this field is the computing of ground state energies and low energy properties of chemical systems ( the chemical structure problem ) . This problem is believed to be intractable in general , yet the quantum Variational Quantum Eigensolver ( VQE ) [ 2 ] algorithm can provide solutions in regimes which lie beyond the reach of classical algorithms , while maintaining NISQ-friendly properties [ 3 , 4 ] . VQE is a hybrid quantum-classical algorithm , where a parametrized quantum state is prepared on a quantum computer , the parameters of which are selected using classical optimization methods . The objective is to prepare the state |ψ ( ~θ ) 〉 which can be used to approximate the ground state of a given Hamiltonian H by the variational principle Emin ≤ min ~θ ( 〈ψ ( ~θ ) |H|ψ ( ~θ ) 〉 ) , ( 1 ) where Emin is the true ground state energy of H . The parametrized state is prepared by applying U ( ~θ ) , which is a parametrized quantum circuit ( typically with a fixed architecture ) , where the angles ~θ = ( θ1 ... θn ) specify the rotation angles of the local unitary rotations present in the circuit . This circuit , known as the ansatz is applied to an initial state |ψ0〉 , usually chosen to be the fiducial “ all zero ” state |00 ... 0〉 , to prepare the state |ψ ( ~θ ) 〉 = U ( ~θ ) |ψ0〉 . It is well established that the structure of the ansatz can dramatically influence the VQE ’ s performance [ 4 , 5 ] , as the closeness of the estimated ground state to the true one depends on the state manifold accessible by the ansatz . Thus finding new architecture construction methods could lead to breakthroughs in quantum variational methods for chemistry ( e.g . for strongly-correlated systems , for which standard ansatzes might fail ) , but also in other domains which utilize variational circuits such as machine learning and optimization [ 6 , 7 , 8 , 9 ] . Currently , the foremost ansatzes fall primarily in two classes : chemistry-inspired ( e.g . the unitary coupled-cluster ansatz [ 2 , 10 ] ) and hardware-inspired ( e.g . the hardware efficient ansatz [ 11 ] ) . Architectures from both of these classes entail using a fixed architecture [ 2 , 10 , 11 , 12 ] determining the unitary U ( ~θ ) , and hence the corresponding ansatz . The ansatz circuit is then usually decomposed into two-qubit CNOT and one-qubit rotation gates parametrized by ( θ1 , θ2 , . . . , θn ) ∈ [ 0 , 2π ] n to be optimized by a classical subroutine . However , the architecture itself can also be optimized . This results in a hard structure optimization problem , as it is a combinatorial optimization problem that must balance two competing factors . On the one hand , the ansatz needs to be expressive enough to guarantee a good approximation of the ground state energy . On the other hand , the depth and size of the circuit need to be controlled in order for the latter to be compatible with NISQ restrictions . Contributions In this work , we propose a general optimization procedure for VQE based on deep reinforcement learning ( RL ) which is designed to yield quantum circuits that are both gate and depth-efficient . We supplement this RL approach with curriculum learning [ 13 , 14 ] , a powerful machine learning method for solving complex problems by leveraging the solutions of previouslysolved simpler instances . Specifically , we introduce a feedback-driven curriculum learning method that autonomously adapts the complexity of the learning problem to the current performance of the learning algorithm . We apply our architecture to the well-established benchmarking problem of finding the ground state energy of the LiH molecule [ 15 , 3 ] and observe chemical accuracy while maintaining a low-depth quantum circuit and achieving state-of-the-art results in terms of gate efficiency . Apart from ground state estimation , the proposed method has a wider range of applications , the algorithm can be applied to any variational-circuit-based algorithm . The rest of the paper is organized as follows . In Section 2 , we discuss the related works . In Section 3 , we introduce our method with simplistic assumptions about the exact value of the ground state energy . At the end of this section , we provide a method circumventing this unrealistic requirement . As an initial proof-of-concept , in Section 4 , we present the results assuming access to the exact value of energy . In Section 5 , we provide the results for the full method that utilizes a rough proxy of energy obtained with classical methods . We conclude the paper with a discussion of possible future directions . 2 Related Work . The automatic construction of quantum circuits with the use of heuristics/machine learning methods is a topic that has recently been gaining more attention [ 16 , 17 , 18 , 19 , 20 ] . In the recent article [ 21 ] , the authors explore various methods of deep reinforcement learning for quantum architecture search . In contrast to our approach the authors generate circuits which are designed to prepare multi-qubit GHZ states . Several works concern the construction of quantum circuits in the context of quantum chemistry . In [ 22 ] the authors propose the Quantum Architecture Search algorithm , based on the AutoML method – Neural Architecure Search , which is benchmarked on the H2 molecule – a simpler system than LiH . On the other hand , more complex molecules are considered in [ 23 ] , the authors use evolutionary algorithms to optimize the structure of VQE circuits for quantum chemistry and combinatorial optimization . The gate set in this algorithm is composed of one-qubit universal gates and two-qubit controlled universal gates , which after the optimization procedure , are decomposed into CNOT gates and one-qubit universal rotation gates . Another evolutionary strategy was explored in [ 24 ] , where a multi-objective genetic algorithm is used to optimize the structure of the VQE ansatz in the context of quantum chemistry problems . The topology of the circuits is optimized by a non-dominated sorting genetic algorithm while the parameters are globally tuned by Covariance Matrix Adaptation Evolution Strategy ( CMA-ES ) [ 25 ] . Yet another different approach , explored in the manuscript [ 26 ] , involved applying a set of rules to both grow and remove quantum gates during the optimization . The last three manuscripts focus on solving problems in quantum chemistry by constructing the shortest possible quantum circuits . 3 Methods . In this section , we discuss our approach for constructing VQE ansatzes using RL . We discuss the state and action representations , as well as the reward function used in this work . We then introduce a novel adaptation of curriculum learning applied to VQE circuit construction with different parameter optimization strategies . 3.1 Ansatz optimization as a reinforcement learning problem . We describe ansatz optimization as a reinforcement learning environment , where the states encode the current circuit . The actions encode a gate that is added to the current circuit . The environmental transitions are deterministic because the subsequent state is defined by the previous state appended by the taken action . At each time step t , the agent receives the reward R according to the following formula : R = 5 if Et < ξ −5 if t ≥ L and Et ≥ ξ max ( Et−1−EtEt−1−Emin , −1 ) otherwise , ( 2 ) The goal of the agent is to reach the minimal energy Emin within a predefined threshold ξ . The energy Et is calculated for the current circuit representing the quantum state |ψ ( ~θ ) 〉 according to : Et = 〈ψ ( ~θ ) |H|ψ ( ~θ ) 〉 , ( 3 ) where ~θ describes the angles of the rotation gates . The vector ~θ needs to be separately optimized to obtain the reward R. This optimization subroutine is a crucial hyperparameter of this RL environment and it is explained in more detail in the following two sections . The details on the reward shaping are discussed in Supplementary Materials . We utilize deep RL methods , thus the states and actions are represented in a neural network-friendly form . Namely , each state is represented as an ordered list of layers that are composed of single quantum gates . Thus the environmental state , which is represented by this list , fully describes the whole circuit ( with a specified gate ordering ) . For constructing the circuits , we use CNOT and one qubit rotation gates – which are realizable on currently available quantum devices . The CNOT gates are encoded by two values indicating position of control and target qubits ( the position is enumerated starting from 0 ) . The rotation gates are encoded by two integers ( starting from 0 ) . The first number indicates the qubit register and the second specifies the axis of rotation ( X – 1 , Y – 2 , Z – 3 ) . We deliberately omit the continuous parameters describing the angles of the rotation gates from the state representation . Instead , the energy estimated for the state is appended to state representation . The user needs to define the maximum number of circuit layers ( L ) . If a given circuit has fewer layers than L , the remaining ones are filled with identity operators ( c.f . Fig . 1 ) . The agent constructs the circuit iteratively , starting from an empty list , and adding a single quantum gate at the end of the current circuit . This way the agent starts each episode with exactly the same conditions , i.e . an empty state , avoiding an additional source of randomness tied to the initial state . Thus , the set of all possible actions is equivalent to the set of all possible quantum gates that the agent can select . The size of the set of all possible actions for the agent is then 3|Q|+2 ( |Q| 2 ) = |Q| ( |Q|+2 ) , where |Q| is the number of qubits . The agent can choose from 3|Q| single-qubit gates and from 2 ( |Q| 2 ) two-qubit gates . We use a simple one-hot encoding for the actions . 3.2 Specification of the agent and the environment . We employ a Double Deep-Q network [ 27 ] ( DDQN ) with an -greedy policy and an ADAM optimizer for better stability . More technical details about the Reinforcement Learning procedure are described in the Experiments section ( Section 4 ) . As described in the previous section , to obtain a reward R for each circuit , an optimization subroutine needs to be applied to determine the continuous values of the rotation gate angles . We use well-developed methods for continuous optimization such as Constrained Optimization By Linear Approximation ( COBYLA ) [ 28 ] and Rotosolve [ 29 ] . Whilst we could use an agent for this purpose as well , we abstain from doing so for two main reasons . Firstly , in this work , we aim at assessing the efficacy of RL for circuit structure optimization , rather than for parameter searching . The latter is investigated extensively in other literature and thus treated as a separate subroutine that can be solved by other , more developed , tools . The second reason is practical , as training the agent to reliably predict values of angles would result in significantly longer training periods . For the above reasons , our approach is hybrid : the agent learns how to , and is rewarded for , constructing particular circuits given a set of parameters determined by an independent optimization algorithm . We apply the angle optimization subroutine only after the steps in which the agent appends a rotation gate to the circuit , thus eliminating the necessity of including the parameters in the state representation . The number of angles optimized at a given step is a hyperparameter in our method , the choice of which will be analyzed in Sec . 4.2 . In this work , we consider optimizing all angles at once ( global strategy ) , as well as optimizing a few angles at a time ( local strategy ) . These different experimental settings allow us to check whether the rough energy approximation obtained from only optimizing a subset of the circuit angles is sufficient to create sufficiently good circuits .

Summary: # Overview  The paper proposes a framework incorporating deep reinforcement learning for resolving Ansatz optimization problems, which is with wide impacts on quantum chemistry. The authors have a good introduction and related work section, which covers most standing works in quantum reinforcement learning and variational circuit learning.   Compared with Rotosolve and COBYLA, the proposed DDQN-based method attains state-of-the-art results on estimating the ground-state energy of lithium hydride.  Overall, I like the idea but the baseline comparison and some time complexity could better incorporate to add the depth of this paper. The current version is more application-oriented and less novel as a general ML framework considering existing works in the neural architecture search (NAS) community.   - Justification of DDQN and comparison   When discussing "why selecting DDQN for the RL", there is very little discussion.   Since this work is not the first works on applying the heuristic method for quantum circuit architectures search (QCAS), the contribution is more on framing the Ansatz optimization and the benchmark results. It would be more valuable to address the selection and justification of the algorithm. (e.g., DRL-based vs Neuroevolution-based or at least the variants inside DQNs)  For the general audience in the NeurIPS community, instead of only showing the DDQN-based method for QCAS, providing more in-depth discussion between variants of the DRL algorithm (e.g., DQN, DQN with dueling, DDQN with dueling) and its complexity could gain better values of this paper.