Parametrized Quantum Policies for Reinforcement Learning

1 Introduction . Hybrid quantum machine learning models constitute one of the most promising applications of near-term quantum computers [ 1 , 2 ] . In these models , parametrized and data-dependent quantum computations define a hypothesis family for a given learning task , and a classical optimization algorithm is used to train them . For instance , parametrized quantum circuits ( PQCs ) [ 3 ] have already proven successful in classification [ 4–8 ] , generative modeling [ 9 , 10 ] and clustering [ 11 ] problems . Moreover , recent results have shown proofs of their learning advantages in artificially constructed tasks [ 6 , 12 ] , some of which are based on widely believed complexity-theoretic assumptions [ 12–15 ] . All these results , however , only consider supervised and generative learning settings . Arguably , the largest impact quantum computing can have is by providing enhancements to the hardest learning problems . From this perspective , reinforcement learning ( RL ) stands out as a field that can greatly benefit from a powerful hypothesis family . This is showcased by the boost in learning performance that deep neural networks ( DNNs ) have provided to RL [ 16 ] , which enabled systems like AlphaGo [ 17 ] , among other achievements [ 18 , 19 ] . Nonetheless , the true potential of near-term quantum approaches in RL remains very little explored . The few existing works [ 20–23 ] have failed so far at solving classical benchmarking tasks using PQCs and left open the question of their ability to provide a learning advantage . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contributions In this work , we demonstrate the potential of policies based on PQCs in solving classical RL environments . To do this , we first propose new model constructions , describe their learning algorithms , and show numerically the influence of design choices on their learning performance . In our numerical investigation , we consider benchmarking environments from OpenAI Gym [ 24 ] , for which good and simple DNN policies are known , and in which we demonstrate that PQC policies can achieve comparable performance . Second , inspired by the classification task of Havlíček et al . [ 6 ] , conjectured to be classically hard by the authors , we construct analogous RL environments where we show an empirical learning advantage of our PQC policies over standard DNN policies used in deep RL . In the same direction , we construct RL environments with a provable gap in performance between a family of PQC policies and any efficient classical learner . These environments essentially build upon the work of Liu et al . [ 14 ] by embedding into a learning setting the discrete logarithm problem ( DLP ) , which is the problem solved by Shor ’ s celebrated quantum algorithm [ 25 ] but widely believed to be classically hard to solve [ 26 ] . Related work Recently , a few works have been exploring hybrid quantum approaches for RL . Among these , Refs . [ 20 , 21 ] also trained PQC-based agents in classical RL environments . However , these take a value-based approach to RL , meaning that they use PQCs as value-function approximators instead of direct policies . The learning agents in these works are also tested on OpenAI Gym environments ( namely , a modified FrozenLake and CartPole ) , but do not achieve sufficiently good performance to be solving them , according to the Gym specifications . Ref . [ 27 ] shows that , using some of our design choices for PQCs in RL ( i.e. , data re-uploading circuits [ 28 ] with trainable observable weights and input scaling parameters ) , one can also solve these environments using a value-based approach . An actor-critic approach to QRL was introduced in Ref . [ 22 ] , using both a PQC actor ( or policy ) and a PQC critic ( or value-function approximator ) . In contrast to our work , these are trained in quantum environments ( e.g. , quantum-control environments ) , that provide a quantum state to the agent , which acts back with a continuous classical action . These aspects make it a very different learning setting to ours . Ref . [ 23 ] also describes a hybrid quantum-classical algorithm for value-based RL . The function-approximation models on which this algorithm is applied are however not PQCs but energy-based neural networks ( e.g. , deep and quantum Boltzmann machines ) . Finally , our work provides an alternative approach to take advantage of quantum effects in designing QRL agents compared to earlier approaches [ 29–33 ] , which are mainly based on ( variations of ) Grover ’ s search algorithm [ 34 ] or quantum annealers [ 35 ] to speed up sampling routines . Code An accompanying tutorial [ 36 ] , implemented as part of the quantum machine learning library TensorFlow Quantum [ 37 ] , provides the code required to reproduce our numerical results and explore different settings . It also implements the Q-learning approach for PQC-based RL of Skolik et al . [ 27 ] 2 Parametrized quantum policies : definitions and learning algorithm . In this section , we give a detailed construction of our parametrized quantum policies and describe their associated training algorithms . We start however with a short introduction to the basic concepts of quantum computation , introduced in more detail in [ 38 , 39 ] . 2.1 Quantum computation : a primer . A quantum system composed of n qubits is represented by a 2n-dimensional complex Hilbert space H = ( C2 ) ⊗n . Its quantum state is described by a vector |ψ⟩ ∈ H of unit norm ⟨ψ|ψ⟩ = 1 , where we adopt the bra-ket notation to describe vectors |ψ⟩ , their conjugate transpose ⟨ψ| and inner-products ⟨ψ|ψ′⟩ inH . Single-qubit computational basis states are given by |0⟩ = ( 1 , 0 ) T , |1⟩ = ( 0 , 1 ) T , and their tensor products describe general computational basis states , e.g. , |10⟩ = |1⟩ ⊗ |0⟩ = ( 0 , 0 , 1 , 0 ) . A quantum gate is a unitary operation U acting on H. When a gate U acts non-trivially only on a subset S ⊆ [ n ] of qubits , we identify it to the operation U ⊗ 1 [ n ] \S . In this work , we are mainly interested in the single-qubit Pauli gates Z , Y and their associated rotations Rz , Ry : Z = ( 1 0 0 −1 ) , Rz ( θ ) = exp ( −iθ 2 Z ) , Y = ( 0 −i i 0 ) , Ry ( θ ) = exp ( −iθ 2 Y ) , ( 1 ) for rotation angles θ ∈ R , and the 2-qubit Ctrl-Z gate = diag ( 1 , 1 , 1 , −1 ) . A projective measurement is described by a Hermitian operator O called an observable . Its spectral decomposition O = ∑ m αmPm in terms of eigenvalues αm and orthogonal projections Pm defines the outcomes of this measurement , according to the Born rule : a measured state |ψ⟩ gives the outcome αm and gets projected onto the state Pm |ψ⟩ / √ p ( m ) with probability p ( m ) = ⟨ψ|Pm |ψ⟩ = ⟨Pm⟩ψ . The expectation value of the observable O with respect to |ψ⟩ is Eψ [ O ] = ∑ m p ( m ) αm = ⟨O⟩ψ . 2.2 The RAW-PQC and SOFTMAX-PQC policies . At the core of our parametrized quantum policies is a PQC defined by a unitary U ( s , θ ) that acts on a fixed n-qubit state ( e.g. , |0⊗n⟩ ) . This unitary encodes an input state s ∈ Rd and is parametrized by a trainable vector θ . Although different choices of PQCs are possible , throughout our numerical experiments ( Sec . 3 and 4.2 ) , we consider so-called hardware-efficient PQCs [ 40 ] with an alternatinglayered architecture [ 28 , 41 ] . This architecture is depicted in Fig . 2 and essentially consists in an alternation of Denc encoding unitaries Uenc ( composed of single-qubit rotations Rz , Ry ) and Denc + 1 variational unitaries Uvar ( composed of single-qubit rotations Rz , Ry and entangling Ctrl-Z gates ) . For any given PQC , we define two families of policies , differing in how the final quantum states |ψs , θ⟩ = U ( s , θ ) |0⊗n⟩ are used . In the RAW-PQC model , we exploit the probabilistic nature of quantum measurements to define an RL policy . For |A| available actions to the RL agent , we partition H in |A| disjoint subspaces ( e.g. , spanned by computational basis states ) and associate a projection Pa to each of these subspaces . The projective measurement associated to the observable O = ∑ a aPa then defines our RAW-PQC policy πθ ( a|s ) = ⟨Pa⟩s , θ . A limitation of this policy family however is that it does not have a directly adjustable greediness ( i.e. , a control parameter that makes the policy more peaked ) . This consideration arises naturally in an RL context where an agent ’ s policy needs to shift from an exploratory behavior ( i.e. , close to uniform distribution ) to a more exploitative behavior ( i.e. , a peaked distribution ) . To remedy this limitation , we define the SOFTMAX-PQC model , that applies an adjustable softmaxβ non-linear activation function on the expectation values ⟨Pa⟩s , θ measured on |ψs , θ⟩ . Since the softmax function normalizes any real-valued input , we can generalize the projections Pa to be arbitrary Hermitian operators Oa . We also generalize these observables one step further by assigning them trainable weights . The two models are formally defined below . Definition 1 ( RAW- and SOFTMAX-PQC ) . Given a PQC acting on n qubits , taking as input a state s ∈ Rd , rotation angles ϕ ∈ [ 0 , 2π ] |ϕ| and scaling parameters λ ∈ R|λ| , such that its corresponding unitary U ( s , ϕ , λ ) produces the quantum state |ψs , ϕ , λ⟩ = U ( s , ϕ , λ ) |0⊗n⟩ , we define its associated RAW-PQC policy as : πθ ( a|s ) = ⟨Pa⟩s , θ ( 2 ) where ⟨Pa⟩s , θ = ⟨ψs , ϕ , λ|Pa|ψs , ϕ , λ⟩ is the expectation value of a projection Pa associated to action a , such that ∑ a Pa = I and PaPa′ = δa , a′ . θ = ( ϕ , λ ) constitute all of its trainable parameters . Using the same PQC , we also define a SOFTMAX-PQC policy as : πθ ( a|s ) = eβ⟨Oa⟩s , θ∑ a′ e β⟨Oa′ ⟩s , θ ( 3 ) where ⟨Oa⟩s , θ = ⟨ψs , ϕ , λ| ∑ i wa , iHa , i|ψs , ϕ , λ⟩ is the expectation value of the weighted Hermitian operatorsHa , i associated to action a , β ∈ R is an inverse-temperature parameter and θ = ( ϕ , λ , w ) . Note that we adopt here a very general definition for the observables Oa of our SOFTMAX-PQC policies . As we discuss in more detail in Appendix C , very expressive trainable observables can in some extreme cases take over all training of the PQC parameters ϕ , λ and render the role of the PQC in learning trivial . However , in practice , as well as in our numerical experiments , we only consider very restricted observables Oa = ∑ i wa , iHa , i , where Ha , i are ( tensor products of ) Pauli matrices or high-rank projections on computational basis states , which do not allow for these extreme scenarios . In our PQC construction , we include trainable scaling parameters λ , used in every encoding gate to re-scale its input components . This modification to the standard data encoding in PQCs comes in light of recent considerations on the structure of PQC functions [ 42 ] . These additional parameters allow to represent functions with a wider and richer spectrum of frequencies , and hence provide shallow PQCs with more expressive power .

Summary: I thank the authors for their response. I think the empirical results are encouraging and the idea is reasonably novel, so will keep the score as is.  ---  The paper discusses a parametrized quantum computations as a policy trained via policy gradient. The policy is parametrized with a quantum computer whereas parameter updates are done by a classical computer. Situations where the quantum policy can have advantage over classical ones are discussed and evaluated. In PQC-generated environments, PQC-policies have a significant advantage over classical ones.