Parametrized Quantum Policies for Reinforcement Learning

1 Introduction . Hybrid quantum machine learning models constitute one of the most promising applications of near-term quantum computers [ 1 , 2 ] . In these models , parametrized and data-dependent quantum computations define a hypothesis family for a given learning task , and a classical optimization algorithm is used to train them . For instance , parametrized quantum circuits ( PQCs ) [ 3 ] have already proven successful in classification [ 4–8 ] , generative modeling [ 9 , 10 ] and clustering [ 11 ] problems . Moreover , recent results have shown proofs of their learning advantages in artificially constructed tasks [ 6 , 12 ] , some of which are based on widely believed complexity-theoretic assumptions [ 12–15 ] . All these results , however , only consider supervised and generative learning settings . Arguably , the largest impact quantum computing can have is by providing enhancements to the hardest learning problems . From this perspective , reinforcement learning ( RL ) stands out as a field that can greatly benefit from a powerful hypothesis family . This is showcased by the boost in learning performance that deep neural networks ( DNNs ) have provided to RL [ 16 ] , which enabled systems like AlphaGo [ 17 ] , among other achievements [ 18 , 19 ] . Nonetheless , the true potential of near-term quantum approaches in RL remains very little explored . The few existing works [ 20–23 ] have failed so far at solving classical benchmarking tasks using PQCs and left open the question of their ability to provide a learning advantage . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contributions In this work , we demonstrate the potential of policies based on PQCs in solving classical RL environments . To do this , we first propose new model constructions , describe their learning algorithms , and show numerically the influence of design choices on their learning performance . In our numerical investigation , we consider benchmarking environments from OpenAI Gym [ 24 ] , for which good and simple DNN policies are known , and in which we demonstrate that PQC policies can achieve comparable performance . Second , inspired by the classification task of Havlíček et al . [ 6 ] , conjectured to be classically hard by the authors , we construct analogous RL environments where we show an empirical learning advantage of our PQC policies over standard DNN policies used in deep RL . In the same direction , we construct RL environments with a provable gap in performance between a family of PQC policies and any efficient classical learner . These environments essentially build upon the work of Liu et al . [ 14 ] by embedding into a learning setting the discrete logarithm problem ( DLP ) , which is the problem solved by Shor ’ s celebrated quantum algorithm [ 25 ] but widely believed to be classically hard to solve [ 26 ] . Related work Recently , a few works have been exploring hybrid quantum approaches for RL . Among these , Refs . [ 20 , 21 ] also trained PQC-based agents in classical RL environments . However , these take a value-based approach to RL , meaning that they use PQCs as value-function approximators instead of direct policies . The learning agents in these works are also tested on OpenAI Gym environments ( namely , a modified FrozenLake and CartPole ) , but do not achieve sufficiently good performance to be solving them , according to the Gym specifications . Ref . [ 27 ] shows that , using some of our design choices for PQCs in RL ( i.e. , data re-uploading circuits [ 28 ] with trainable observable weights and input scaling parameters ) , one can also solve these environments using a value-based approach . An actor-critic approach to QRL was introduced in Ref . [ 22 ] , using both a PQC actor ( or policy ) and a PQC critic ( or value-function approximator ) . In contrast to our work , these are trained in quantum environments ( e.g. , quantum-control environments ) , that provide a quantum state to the agent , which acts back with a continuous classical action . These aspects make it a very different learning setting to ours . Ref . [ 23 ] also describes a hybrid quantum-classical algorithm for value-based RL . The function-approximation models on which this algorithm is applied are however not PQCs but energy-based neural networks ( e.g. , deep and quantum Boltzmann machines ) . Finally , our work provides an alternative approach to take advantage of quantum effects in designing QRL agents compared to earlier approaches [ 29–33 ] , which are mainly based on ( variations of ) Grover ’ s search algorithm [ 34 ] or quantum annealers [ 35 ] to speed up sampling routines . Code An accompanying tutorial [ 36 ] , implemented as part of the quantum machine learning library TensorFlow Quantum [ 37 ] , provides the code required to reproduce our numerical results and explore different settings . It also implements the Q-learning approach for PQC-based RL of Skolik et al . [ 27 ] 2 Parametrized quantum policies : definitions and learning algorithm . In this section , we give a detailed construction of our parametrized quantum policies and describe their associated training algorithms . We start however with a short introduction to the basic concepts of quantum computation , introduced in more detail in [ 38 , 39 ] . 2.1 Quantum computation : a primer . A quantum system composed of n qubits is represented by a 2n-dimensional complex Hilbert space H = ( C2 ) ⊗n . Its quantum state is described by a vector |ψ⟩ ∈ H of unit norm ⟨ψ|ψ⟩ = 1 , where we adopt the bra-ket notation to describe vectors |ψ⟩ , their conjugate transpose ⟨ψ| and inner-products ⟨ψ|ψ′⟩ inH . Single-qubit computational basis states are given by |0⟩ = ( 1 , 0 ) T , |1⟩ = ( 0 , 1 ) T , and their tensor products describe general computational basis states , e.g. , |10⟩ = |1⟩ ⊗ |0⟩ = ( 0 , 0 , 1 , 0 ) . A quantum gate is a unitary operation U acting on H. When a gate U acts non-trivially only on a subset S ⊆ [ n ] of qubits , we identify it to the operation U ⊗ 1 [ n ] \S . In this work , we are mainly interested in the single-qubit Pauli gates Z , Y and their associated rotations Rz , Ry : Z = ( 1 0 0 −1 ) , Rz ( θ ) = exp ( −iθ 2 Z ) , Y = ( 0 −i i 0 ) , Ry ( θ ) = exp ( −iθ 2 Y ) , ( 1 ) for rotation angles θ ∈ R , and the 2-qubit Ctrl-Z gate = diag ( 1 , 1 , 1 , −1 ) . A projective measurement is described by a Hermitian operator O called an observable . Its spectral decomposition O = ∑ m αmPm in terms of eigenvalues αm and orthogonal projections Pm defines the outcomes of this measurement , according to the Born rule : a measured state |ψ⟩ gives the outcome αm and gets projected onto the state Pm |ψ⟩ / √ p ( m ) with probability p ( m ) = ⟨ψ|Pm |ψ⟩ = ⟨Pm⟩ψ . The expectation value of the observable O with respect to |ψ⟩ is Eψ [ O ] = ∑ m p ( m ) αm = ⟨O⟩ψ . 2.2 The RAW-PQC and SOFTMAX-PQC policies . At the core of our parametrized quantum policies is a PQC defined by a unitary U ( s , θ ) that acts on a fixed n-qubit state ( e.g. , |0⊗n⟩ ) . This unitary encodes an input state s ∈ Rd and is parametrized by a trainable vector θ . Although different choices of PQCs are possible , throughout our numerical experiments ( Sec . 3 and 4.2 ) , we consider so-called hardware-efficient PQCs [ 40 ] with an alternatinglayered architecture [ 28 , 41 ] . This architecture is depicted in Fig . 2 and essentially consists in an alternation of Denc encoding unitaries Uenc ( composed of single-qubit rotations Rz , Ry ) and Denc + 1 variational unitaries Uvar ( composed of single-qubit rotations Rz , Ry and entangling Ctrl-Z gates ) . For any given PQC , we define two families of policies , differing in how the final quantum states |ψs , θ⟩ = U ( s , θ ) |0⊗n⟩ are used . In the RAW-PQC model , we exploit the probabilistic nature of quantum measurements to define an RL policy . For |A| available actions to the RL agent , we partition H in |A| disjoint subspaces ( e.g. , spanned by computational basis states ) and associate a projection Pa to each of these subspaces . The projective measurement associated to the observable O = ∑ a aPa then defines our RAW-PQC policy πθ ( a|s ) = ⟨Pa⟩s , θ . A limitation of this policy family however is that it does not have a directly adjustable greediness ( i.e. , a control parameter that makes the policy more peaked ) . This consideration arises naturally in an RL context where an agent ’ s policy needs to shift from an exploratory behavior ( i.e. , close to uniform distribution ) to a more exploitative behavior ( i.e. , a peaked distribution ) . To remedy this limitation , we define the SOFTMAX-PQC model , that applies an adjustable softmaxβ non-linear activation function on the expectation values ⟨Pa⟩s , θ measured on |ψs , θ⟩ . Since the softmax function normalizes any real-valued input , we can generalize the projections Pa to be arbitrary Hermitian operators Oa . We also generalize these observables one step further by assigning them trainable weights . The two models are formally defined below . Definition 1 ( RAW- and SOFTMAX-PQC ) . Given a PQC acting on n qubits , taking as input a state s ∈ Rd , rotation angles ϕ ∈ [ 0 , 2π ] |ϕ| and scaling parameters λ ∈ R|λ| , such that its corresponding unitary U ( s , ϕ , λ ) produces the quantum state |ψs , ϕ , λ⟩ = U ( s , ϕ , λ ) |0⊗n⟩ , we define its associated RAW-PQC policy as : πθ ( a|s ) = ⟨Pa⟩s , θ ( 2 ) where ⟨Pa⟩s , θ = ⟨ψs , ϕ , λ|Pa|ψs , ϕ , λ⟩ is the expectation value of a projection Pa associated to action a , such that ∑ a Pa = I and PaPa′ = δa , a′ . θ = ( ϕ , λ ) constitute all of its trainable parameters . Using the same PQC , we also define a SOFTMAX-PQC policy as : πθ ( a|s ) = eβ⟨Oa⟩s , θ∑ a′ e β⟨Oa′ ⟩s , θ ( 3 ) where ⟨Oa⟩s , θ = ⟨ψs , ϕ , λ| ∑ i wa , iHa , i|ψs , ϕ , λ⟩ is the expectation value of the weighted Hermitian operatorsHa , i associated to action a , β ∈ R is an inverse-temperature parameter and θ = ( ϕ , λ , w ) . Note that we adopt here a very general definition for the observables Oa of our SOFTMAX-PQC policies . As we discuss in more detail in Appendix C , very expressive trainable observables can in some extreme cases take over all training of the PQC parameters ϕ , λ and render the role of the PQC in learning trivial . However , in practice , as well as in our numerical experiments , we only consider very restricted observables Oa = ∑ i wa , iHa , i , where Ha , i are ( tensor products of ) Pauli matrices or high-rank projections on computational basis states , which do not allow for these extreme scenarios . In our PQC construction , we include trainable scaling parameters λ , used in every encoding gate to re-scale its input components . This modification to the standard data encoding in PQCs comes in light of recent considerations on the structure of PQC functions [ 42 ] . These additional parameters allow to represent functions with a wider and richer spectrum of frequencies , and hence provide shallow PQCs with more expressive power .

Summary: ## Summary   The paper proposes a policy-based training framework with variational quantum circuits (VQC). Although the VQC-based deep reinforcement learning algorithm has been first studied in [20] and [23], this paper further addresses a policy-based optimization instead of a value-based method (e.g., DQN) in [20].   The introduction and related work parts are good covering most of the important references in the field.   The paper further discusses two variants of the parameterized quantum circuit (PQC), raw-PQC and softmax-PQC to compared with a DNN-based RL method in three DRL environments from OpenAI gym.   - Noted I am not sure how fair is the selection DNN-based RL method in terms of model capacity according to the current version.   The authors further discuss some points included of (1) Efficient policy sampling and (2) quantum advantages.   Overall, the paper is well-written and motivated. The major weakness is on the baseline selection and some scalable discussion on quantum advantages.   - baseline selection  The works in [22] using a continuous quantum Deep RL method, called Quantum DDPG. What is the advantage between the proposed softmax PQC compared with Quantum DDPG?  - scalable discussion  I just notice that the value-based method paper [20] uses a simple tangible radio RL environment to discuss the trade-off between parameter complexity (Figure 11 in [20]) versus different classical and quantum RL methods.   If the author could provide experimental supports or discussion versus that existing analysis on the QA on parameters, the discussion could be more comprehensive.