Title: Differentiable Quantum Computing for Large-scale Linear Control
Venue: NeurIPS 2024
Accepted: yes
Keywords: quantum computing

quantum algorithms

optimal control

differentiable physics

reinforcement learning
Abstract: As industrial models and designs grow increasingly complex, the demand for optimal control of large-scale dynamical systems has significantly increased. However, traditional methods for optimal control incur significant overhead as problem dimensions grow. In this paper, we introduce an end-to-end quantum algorithm for linear-quadratic control with provable speedups. Our algorithm, based on a policy gradient method, incorporates a novel quantum subroutine for solving the matrix Lyapunov equation. Specifically, we build a *quantum-assisted differentiable simulator* for efficient gradient estimation that is more accurate and robust than classical methods relying on stochastic approximation. Compared to the classical approaches, our method achieves a *super-quadratic* speedup. To the best of our knowledge, this is the first end-to-end quantum application to linear control problems with provable quantum advantage.

Differentiable Quantum Computing for Large-scale Linear Control

Abstract

As industrial models and designs grow increasingly complex, the demand for optimal control of large-scale dynamical systems has significantly increased. However, traditional methods for optimal control incur significant overhead as problem dimensions grow. In this paper, we introduce an end-to-end quantum algorithm for linear-quadratic control with provable speedups. Our algorithm, based on a policy gradient method, incorporates a novel quantum subroutine for solving the matrix Lyapunov equation. Specifically, we build a quantum-assisted differentiable simulator for efficient gradient estimation that is more accurate and robust than classical methods relying on stochastic approximation. Compared to the classical approaches, our method achieves a super-quadratic speedup. To the best of our knowledge, this is the first end-to-end quantum application to linear control problems with provable quantum advantage.

1 Introduction

Over the past few decades, the growing complexity of modern engineering designs has made the control of large-scale dynamical systems a crucial task across various application fields, such as power grid management [56], swarm robotics [16, 18], sensor networks [21], and airline scheduling [57]. These challenges often involve high-dimensional solution spaces with tens of thousands of degrees of freedom, presenting a significant obstacle for traditional optimal control methods.

The emergence of quantum computing has expanded the potential for designing efficient algorithms in numerical optimization and machine learning [1, 36, 62]. By leveraging the principles of quantum mechanics, such as superposition and entanglement, quantum computers excel at efficient data processing, making them promising for accelerating solutions to large-scale computational challenges [31].

Although there has been some progress in quantum algorithms for some specific optimal control problems arising in quantum sciences [37, 39], a viable pathway for accelerating general large-scale optimal control problems remains unclear. A conventional approach to optimal control involves solving the Algebraic Riccati Equation (ARE, see Section 1.1 for details), which is a nonlinear matrix equation. This problem has been less explored in the

field of quantum computing for two reasons. First, most proposed quantum algorithms for algebraic and differential equations focus on linear and vector-valued problems, and extending them to nonlinear matrix equations is highly challenging. Second, while efficient quantum algorithms exist for certain weakly nonlinear problems [41], they are not powerful enough to handle the nonlinearity present in the Algebraic Riccati Equation. A breakthrough in this direction calls for novel ideas in algorithm design.

Inspired by the recent advances in differentiable physics [37, 46, 53] and reinforcement learning [19, 45], we develop an end-to-end quantum algorithm that solves a fundamental optimal control problem called the linear-quadratic regulator (LQR). Given its widely applicable mathematical formulation, LQR has been extensively researched and serves as a standard case study for various computing and learning algorithms [25]; moreover, LQR is of significant practical relevance as many real-world optimal control problems can be formulated to address through linearization techniques. Our quantum algorithm is proven to output an ε -approximate optimal solution in time ˜ O ( nε -1 . 5 ) 1 , where n is the dimension of the state vector and ε is an error tolerance parameter. The algorithm involves two major components: a quantum differentiable simulator and a quantum-accessible classical data structure. This hybrid quantum-classical framework enables us to employ a policy gradient method that exhibits a fast convergence rate for the LQR problem. Since almost all known classical methods for the LQR problem heavily rely on subroutines such as matrix factorization and matrix inversion [6, 32, 34, 35], which require at least O ( n 3 ) overhead in the problem dimension n , our new linear-time quantum algorithm, with super-quadratic speedup, offers significant promise for large-scale applications.

Notation. We use R and C to denote the set of real and complex numbers, respectively. I denotes an identity operator with an appropriate dimension. For two real vectors u, v ∈ R n , the Euclidean inner product 〈 u, v 〉 = u T v , and the norm of a vector u is ‖ u ‖ = √ u T u . Given a symmetric/Hermitian matrix M , we denote λ max ( M ) (or λ min ( M )) as the maximal/minimal eigenvalue of M . The spectral norm of a matrix M ∈ R m × n is denoted by ‖ M ‖ = sup ‖ v ‖ =1 ‖ Mv ‖ . The Frobenius norm of a matrix M ∈ R m × n is denoted by ‖ M ‖ F = ∑ i,j | M i,j | 2 = Tr [ M T M ] . We say ξ ∼ D if the random variable ξ ∈ R n is distributed according to D .

1.1 Problem Formulation

We focus on the infinite-horizon continuous-time linear-quadratic regulator (LQR) problem:

where x ( t ): [0 , ∞ ] → R n is the state vector, u ( t ): [0 , ∞ ] → R m is the control input. A and B are constant matrices of appropriate dimensions; Q and R are positive definite matrices.

Definition 2. For a controllable pair ( A,B ), the set of stabilizing feedback gains is given by

Definition 1. For a square matrix M ∈ R n × n , we say M is Hurwitz if every eigenvalue of M has a strictly negative real part.

Given a controllable pair ( A,B ), the optimal controller u ( t ) of problem (1) can be expressed as a linear function of the state vector x ( t ), namely

where the matrix K ∗ is the optimal linear feedback gain. An analytical form of the optimal feedback gain is given by K ∗ = R -1 B /latticetop P ∗ , where P ∗ is the unique positive solution to the Algebraic Riccati Equation (ARE),

For large-scale control problems where the control input is much smaller than the state vector (i.e., m /lessmuch n ), it is often desired to compute the optimal feedback gain matrix K ∗ without explicitly solving for P ∗ . To this end, we can rewrite the LQR objective function J ( x, u ) as a function solely depending on K by leveraging the linearity of the optimal controller u ( t ) = -Kx ( t ). With standard algebraic manipulation, it turns out that

where

and Σ 0 := E ξ ∼D [ ξξ /latticetop ]. Given this reformulation, the search for the optimal feedback gain K ∗ reduces to minimizing the unconstrained objective function f ( K ).

In practice, the matrices A,B,Q,R often possess sparsity structures that can be leveraged by quantum computers. We make the assumptions on the efficient quantum access model.

Assumption 1 (Sparse-access matrices) . We assume A , B , Q , and R are s -sparse, i.e., there are at most s non-zero entries in each row/column. For M ∈ { A,B,Q,R } , we assume access to an efficient procedure 2 that loads the matrix into quantum data:

where r i,k is the index of the k -th non-zero entry of the i -th row of M , c /lscript,j is the index of the /lscript -th non-zero entry of the j -th column of M , and M ij is a fixed-length binary description of the ( i, j )-th entry of M .

With quantum access to the problem data, we aim to determine the optimal linear feedback gain K ∗ so that the objective function f ( K ) is minimized, as summarized in the following problem statement:

Problem 1. Assume ( A,B ) is a controllable pair and Q , R are positive-definite. Given quantum access to A,B,Q,R in the sense of Assumption 1, we want to compute an ε -approximate solution K such that ‖ K -K ∗ ‖ F ≤ ε , where ε > 0 is prefixed.

1.2 Main Contributions

In this paper, we propose an end-to-end quantum algorithm for solving LQR problems that exhibits the desired quantum advantage in the large-scale setting (i.e., in the parameter regime m /lessmuch n ). A detailed comparison between ours and various other methods is given in Table 1. Compared with state-of-the-art classical methods, our algorithm achieves a super-quadratic speedup in terms of the state vector dimension n . To the best of our knowledge, this is the first end-to-end quantum application to linear control problems with provable speedup.

Our algorithm is based on a novel policy gradient strategy to find globally optimal solutions for linear-quadratic control problems. A brief overview of the policy gradient method for LQR is available in Section 3.2. In each iteration cycle, our algorithm executes a fast,

quantum-assisted differentiable simulator to obtain robust gradient estimates, as detailed in Theorem 31. The gradient estimates are then utilized by a classical computer to update the control policy K .

As illustrated in Figure 1, with the back-and-forth iterations between the quantum simulator and a classical computer, the control policy K converges to the optimal policy K ∗ at a linear rate, leading to an end-to-end resolution of the LQR problem.

Our quantum algorithm design can be regarded as a novel realization of the hybrid quantum-classical computing paradigm, where collaboration between classical and quantum computers significantly reduces the burden on the quantum side. Moreover, we provide explicit constructions of the quantum simulator and analyze the convergence rate of the policy gradient based on our end-to-end model. These desirable attributes make our proposed design more practical and relevant in the early fault-tolerant era [29].

Notably, we propose a new quantum algorithm for solving the Lyapunov equation, a fundamental task in optimal control theory [25]. Based on an integral representation of the solution and a rich toolbox of methods for quantum numerical linear algebra, our algorithm can produce a quantum representation of the solution matrix in a cost that

Differentiable Quantum

Computing

𝐾

Gradient-based

Optimizer

𝑓(𝐾)

Objective

Function

∇𝑓(𝐾)

Classical Module

Quantum Module

Control

Policy

LQR Policy Gradient

is polylogarithmic in the matrix dimension n (see Theorem 4), leading to an exponential speedup over existing classical methods [51]. Moreover, since the matrix Lyapunov equation is fundamental to many control problems, we foresee that our quantum algorithm may play an important role in finding speedups for other tasks.

The fast quantum algorithm for the Lyapunov equation enables us to develop a quantum gradient estimation subroutine in near-optimal cost, as detailed in Theorem 31. Compared with the conventional gradient estimation techniques based on stochastic approximation (e.g., one- and two-point gradient estimators [45]), our quantum gradient estimation benefits from the explicit exploitation of the analytical form of the gradient. Numerical experiments suggest that our gradient estimation is robust and often leads to faster convergence in practice, as demonstrated in Section 5.3.

2 Related Work

We survey related work on model-based and model-free linear-quadratic control, differentiable physics, and quantum reinforcement learning in this section.

Model-based linear-quadratic control. Model-based optimal control [17, 50] refers to the scenario where historical measurement data explicitly gives (or estimates) the problem description. In this case, the optimal linear feedback gain K ∗ can be computed by solving the algebraic Riccati equation (ARE), as detailed in Section 1.1. Commonly used numerical methods for ARE include factorization methods (e.g., Schur method [35, 48]) and iterative methods (e.g., Newton-Kleinman method [20, 32, 47]). These methods require computing matrix factorization or matrix inverse, which in general leads to a O ( n 3 ) run time (assuming m ≤ n ). Some methods for ARE can achieve O ( n ) runtime under strong assumptions such as the solution P ∗ is of low rank [8]. It is also possible to solve LQR by reformulating it as a semidefinite programming (SDP) problem [14]. We do not dive into the SPD approach as it does not demonstrate superior asymptotic scaling compared to other direct methods.

Model-free linear-quadratic control. LQR can be regarded as a continuous-time analog of the discrete Markov Decision Process (MDP) model and many techniques from reinforcement learning (RL) can be introduced to learn the optimal feedback gain (i.e., control policy ), such as policy gradient [19, 45], natural gradient [25], and policy iteration [10].

These RL-based methods are particularly useful when we have access to the observed costs but the system model can not be directly constructed.

Differentiable physics and quantum computing. The differentiable programming paradigm has been applied to many dynamical systems for learning [46] and control [53]. Those gradient-based methods can be used in reinforcement learning [61], inverse problems [27], optimization [5], design [60], etc. People have developed differentiable pipelines for various dynamics including fluids [59], rigid body [49], soft body [26], and other hybrid systems [52]. Recently, [37] have derived a differentiable analog quantum computing pipeline for quantum optimization and control. In this work, we will focus on using differentiable quantum computing to accelerate a widely studied classical problem - linear control synthesis.

Quantum reinforcement learning. Recently, quantum-accelerated reinforcement learning has attracted significant attention as it demonstrates the potential for computational speedup [43]. It has been shown that quantum computers can be used to compute policy gradients given coherent access to a Markov Decision tree model [15, 28]. Some works also discuss the quantum policy iteration method for RL, see [12, 58]. It is worth noting that the existing works are usually based on a strong quantum access model and it remains unclear what the cost of constructing such models is in an end-to-end sense.

3 Preliminaries

3.1 Introduction to Quantum Computing

All quantum states of a quantum system form a Hilbert space, which is isomorphic to C N . We may assume N = 2 n and n is a non-negative integer. An element | ψ 〉 in this Hilbert space is then noted as a N -dimensional quantum state, where

where v i ∈ C , i ∈ { 0 , 1 , · · · , N -1 } . Also, we often use 〈 ψ | to denote the conjugate transpose of | ψ 〉 . For any c = 0, c | ψ 〉 and | ψ 〉 refer to the same state, thus without loss of generality, ‖| ψ 〉‖ = 1 always holds. Specifically, a one qubit system corresponds to the aforementioned Hilbert space with n = 1.

/negationslash

Given m quantum states | ψ 1 〉 , | ψ 2 〉 , · · · , | ψ m 〉 from m quantum systems, then

is a quantum state in the space that consists m subspaces.

The evolution of a quantum state can be described by a unitary operator U , meaning

We often note these operations as gates on the quantum circuit. One important type of unitary operators are the Pauli operators, namely

They form a basis of all the linear operators acting on C 2 .

For the quantum measurement, given a quantum observable H , we can do the measurement of a quantum state | ψ 〉 . Specifically, after the measurement on | ψ 〉 , the state collapses to P m | ψ 〉 √ p m , and we get an outcome λ m with probability p m = 〈 ψ | P m | ψ 〉 , where

is the spectral decomposition of H .

3.2 Policy Gradient for LQR

For all stabilizing feedback gains K ∈ S K , the gradient of the objective function f ( K ) as defined in (5) has the following closed-form expression [38, 40]:

where P ( K ) is given in (6), and X ( K ) is determined by

The (direct) policy gradient method for LQR minimizes the objective f ( K ) via the vanilla gradient update rule K ← K -s ∇ f ( K ), where s > 0 is a fixed step size. Given sufficiently small s , it has been shown that the policy gradient method converges at a linear rate [45, Theorem 2]. In practice, however, the policy gradient is often estimated through stochastic approximation, such as one- and two-point estimation [45]. While these zeroth-order gradient estimation methods are less demanding in terms of computational cost, they tend to be sensitive to random perturbations and slow to converge.

In this paper, we propose a fast quantum algorithm that outputs a robust estimate of the gradient in ˜ O ( n ) time (assuming m /lessmuch n , see Theorem 31). Leveraging the quantum gradient estimation subroutine, we recover the linear convergence rate using robust gradient descent, as detailed in Proposition 34.

3.3 Quantum Data Structure

To perform policy gradient in the training process, the linear feedback gain K is stored in a quantum-accessible data structure as proposed in [30]. This data structure allows intermediate updates on K and efficient quantum queries to K as a block-encoded matrix. This data structure is a purely classical representation of K , and quantum access to this data structure (e.g., through qRAM [23]) is required to build the block-encoding of K . In the literature, this data structure is also known as classical-write, quantum-read qRAM [7, 54].

Definition 3 (Block-encoding) . Suppose that M is an p -qubit operator, α, ε ∈ R + and r ∈ N , then we say that the ( p + r )-qubit unitary U M is an ( α, r, ε )-block-encoding of M , if

In this paper, the growth of ancilla qubits (space complexity) is dominated by the number of elementary gates (gate complexity). Therefore, when referring to a specific block-encoding, we often omit the number of ancilla qubits (i.e., the parameter r ) for simplicity.

Lemma 1. Let K ∈ R m × n . There exists a data structure to store K with the following properties: (1) the size of the data structure is O ( mn log 2 ( mn ) ) , (2) the time to store a new entry ( i, j, ˆ K i,j ) is O ( log 2 ( mn ) ) , and (3) for any ε > 0 , a quantum algorithm can implement a ( ‖ K ‖ F , /ceilingleft log 2 n /ceilingright +2 , ε ) -block-encoding of K in time O (poly log( n, 1 /ε )) . There also exists an analogous data structure for ˆ K /latticetop .

Proof. We use the data structure as described in [30, Theorem 5.1]. To construct the block-encoding, we utilize [22, Lemma 50].

3.4 Quantum Simulation of Linear Dynamics

Quantum computers can simulate certain linear ordinary differential equations (ODEs) exponentially faster than classical computers [3, 4, 9, 22, 33]. In this paper, we present a quantum simulation subroutine based on quantum linear system solvers (QLSS) [9, 33], as described below. While this approach may not be optimal in terms of state preparation cost compared to quantum singular value transformation [22] or linear combination of Hamiltonian simulation [3, 4], it allows us to incorporate the Hurwitz stability of the system.

Theorem 2 (Informal version of Theorem 17) . Suppose that A ∈ R n × n is a Hurwitz matrix, and O A is an ( α, 0) -block-encoding of A . For an arbitrary t > 0 , we can implement a ( ζt, ε ) -block-encoding of e A t using ˜ O ( αρt · poly log(1 /ε )) queries to O A , and ˜ O ( αρt · poly log(1 /ε )) queries to additional gates. Here, the normalization factor ζ = O ( αρ ) , and the constant ρ is solely determined by A .

More details and the proof of Theorem 2 can be found in Appendix C. Note that the dependence on t in the above result can be further improved using a standard padding technique, but for simplicity, we do not discuss this minor improvement, as it does not affect our main end-to-end result. We also notice a technique called quantum eigenvalue transformation (QEVT), recently proposed by Low and Su [42]. While this method cannot be directly applied to Hurwitz-stable systems, it may be enhanced to provide a simulation algorithm with a similar cost, as discussed in Appendix D.

4 Quantum Algorithm for the Lyapunov Equation

The (continuous-time) Lyapunov equation is a linear matrix equation of the following form,

Since this equation is linear in terms of the matrix X , it is possible to derive a vectorization form of (16) and solve it using a quantum linear system algorithm [11, 24]. In this paper, we propose a new quantum algorithm for solving the Lyapunov equation based on an integral representation formula. Our algorithm directly prepares a block-encoded solution matrix X ∗ . Compared to the previous approach, our method leads to an exponentially faster quantum objective function evaluation algorithm (see Theorem 5) and a new quantum gradient estimation subroutine (see Theorem 6).

4.1 Representation

Given a positive-definite Q , there exists a unique positive-definite X ∗ satisfying (16) if and only if A is Hurwitz. The unique positive solution is given by

The integral formula (17) suggests that the solution to the Lyapunov equation can be computed using a numerical integration technique. For a finite τ > 0, we define

For any arbitrary ε > 0, we find that a τ = ˜ O (log(1 /ε )) is sufficient to ensure an ε -approximate solution. We denote κ := ‖ X ∗ ‖ /λ min (Ω).

Lemma 3 (Numerical integration) . For any ε > 0 , we have ‖ X ∗ -X τ ‖ ≤ ε , provided that

Proof. Note that ‖ X ∗ -X τ ‖ = ∥ ∥ ∥ ∫ ∞ τ e A t Ω e A /latticetop t d t ∥ ∥ ∥ ≤ ∫ ∞ τ ‖ Ω ‖‖ e A t ‖‖ e A /latticetop t ‖ d t , where ‖ e A t ‖ (or ‖ e A /latticetop t ‖ ) is upper bounded by ‖ X ∗ ‖ λ min ( X ∗ ) e -t/κ (see [44, Lemma 12]). It follows that ‖ X ∗ -X τ ‖ ≤ ‖ Ω ‖‖ X ∗ ‖ κ λ min ( X ∗ ) e -τ/κ . Therefore, an integration time as given in (19) guarantees that ‖ X ∗ -X τ ‖ ≤ ε .

4.2 Algorithm Complexity Analysis

The matrix X τ can be approximated by a trapezoidal rule with ( K +1) quadrature node points, namely,

where w k = (2 -1 k =0 ,K ) τ 2 K , t k = kτ K , and F ( t ) := e A t Ω e A /latticetop t . The summation in (20) can be computed on a quantum computer by performing linear combinations of block-encoded matrices, which we will explain shortly.

Definition 4 (Select oracle) . Let A ∈ R n × n be a Hurwitz matrix. Given an integer K > 0 and two positive scalars τ, ε > 0, we define the following unitary (named as the select oracle ):

where for each k = 0 , . . . , K , U k is a ( ζt k , ε )-block-encoding of e A t k with t k = kτ/K . Here, ζ denotes some parameter that only depends on A .

Now, we consider two select oracles:

where for each k = 0 , . . . , K , U k (or V k ) denotes a block-encoding of e A t k (or e A /latticetop t k ) with normalization factor ζt k . Let O Ω be a ( η, 0)-block-encoding of Ω, and we find that

where W k := U k O Ω V k is a ( ηζ 2 t 2 k , 2 ζηε )-block-encoding of the matrix F ( t k ). Denoting λ k := w k k 2 , then it is clear that ∑ K k =0 λ k W k is a block-encoding of X τ . Thus we can implement a block-encoded X τ on a quantum computer using a technique known as linear combination of unitaries (LCU) [22, Lemma 52]. The rigorous complexity of this quantum algorithm is given in the following theorem, for which a complete proof is provided in Appendix E.

Theorem 4. Suppose that A ∈ R n × n is Hurwitz and Ω ∈ R n × n is positive-definite. Let O A be an ( α, 0) -block-encoding of A and O Ω be an ( η, 0) -block-encoding of Ω . Then, we can implement a ( γ, ε ) -block-encoding of X ∗ , the unique solution to the Lyapunov equation (16), using a single query to O Ω , ˜ O ( α 2 √ η ε ) queries to controlled O A and its inverse, and ˜ O ( α 2 √ η ε ) queries to other additional elementary gates. Here γ = ˜ O ( α 2 η ) .

4.3 Objective Function Evaluation

As a direct consequence of Theorem 4, we can evaluate the objective function value f ( K ) for a given K ∈ S K in a cost that is logarithmic in the dimension parameter n . This result demonstrates an exponential quantum advantage for the objective function evaluation task, as any known classical algorithm for this task requires at least matrix multiplication time.

Theorem 5. Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. Let K ∈ S K be a stabilizing policy stored in a quantum-accessible data structure. We can estimate the objective function f ( K ) up to multiplicative error θ in cost ˜ O ( 1 θ 2 ) .

Proof. Given a K ∈ S K , we can evaluate the objective function f ( K ) via the formula f ( K ) = Tr[ P ( K )Σ 0 ]. Here, without loss of generality, we assume Σ 0 = I . P ( K ) has a closed-form representation as in (6), which corresponds to the unique positive solution to the Lyapunov equation

We denote A = A -BK , and Ω = Q + K /latticetop RK . By Lemma 11 and Remark 1, we can block-encode A with normalization factor s ( ‖ K ‖ F +1) and Ω with normalization factor s ( ‖ K ‖ 2 F +1), both in cost O (poly log( n, s )). Suppose that f ( K ) = a , it follows from Lemma 8 that ‖ K ‖ F ≤ O ( a ). Therefore, by Theorem 4, we can implement a ( γ, ε )-block-encoding of

P ( K ) in cost ˜ O ( a 3 ρ √ κ 5 ε ) , where γ ≤ ˜ O ( a 4 ρ 2 κ 3 ). Note that P ( K ) is a Hermitian matrix, and λ min ( P ( K )) ≥ λ min ( Q ), by invoking Theorem 25, we can estimate f ( K ) = Tr[ P ( K )] up to a multiplicative error θ in cost ˜ O ( a 3 ρ θ √ γ 3 κ 5 ε ) ≤ ˜ O ( a 13 ρ 6 κ 10 θ 2 ) . Here, the error parameter must be chosen so that ε ≤ ˜ O ( θ 2 /γ 2 ).

5 Quantum Policy Gradient for Large-Scale Control

5.1 Quantum Gradient Estimation

Definition 5. Given any K ∈ S K , we call G ∈ R m × n a θ -robust estimate of ∇ f ( K ) if it approximates the gradient ∇ f ( K ) up to a multiplicative error θ , i.e.,

Here, we utilize the close-form expression of the policy gradient ∇ f ( K ), as shown in (13), to construct a quantum algorithm for gradient estimation. The complete theorem statement and the proof are in Appendix G.

Theorem 6 (Informal version of Theorem 31) . Assume we have efficient procedures (as described in Assumption 1) to access A,B,Q,R in O (poly log( n )) time. Let K ∈ S K be a stabilizing policy stored in a quantum-accessible data structure. Provided that ‖ K -K ∗ ‖ > ε and m /lessmuch n , we can compute a θ -robust estimate of ∇ f ( K ) in cost ˜ O ( n θ 1 . 5 ε 1 . 5 ) .

5.2 Quantum Policy Gradient

Our main quantum algorithm for LQR is summarized in Algorithm 1.

Algorithm 1 Quantum policy gradient

Inputs: A,B,Q,R (problem data), K 0 ∈ S K (initial guess), σ > 0 (step size/learning rate), θ (robustness parameter), N (number of iterations)

Output: an approximate solution K N

for k ∈ { 0 , 1 , . . . , N -1 } do Compute a θ -robust estimate of ∇ f ( K k ), denoted as G k , using Theorem 6. Update the quantum-accessible data structure using the rule: K k +1 = K k -σG k . end for

We can prove that the iterates in Algorithm 1 converges to the optimal control policy K ∗ at a linear rate (Proposition 34). It follows that our algorithm can find an ε -approximate optimal policy with an end-to-end cost ˜ O ( n ε 1 . 5 ) .

Theorem 7 (Informal version of Theorem 35) . Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. Let K 0 ∈ S K be a stabilizing policy and assume that m /lessmuch n . Then, Algorithm 1 outputs an ε -approximate solution to Problem 1 in cost ˜ O ( n ε 1 . 5 ) .

5.3 Numerical Experiments

Correctness. We conduct a numerical experiment to showcase the correctness of our quantum policy gradient algorithm. Following a similar setup as in [45], a mass-springdamper system with g = 4 masses is used for constructing our LQR problem. The state

(a)

J

-

J

∗

(b)

f

(

K

)

-

f

(

K

∗

)

(c) runtime

103

classical

102

101

101

ours

10

100

10-1

e

E

10-2

10-3

classical

classical

10-4

ours

10-5

ours

10-2

0

250

500

750

0

250

500

750

10

20

3BoF40

50

60

x = [ p /latticetop , v /latticetop ] /latticetop ∈ R 2 g contains positions and velocities, with dynamic and input matrices,

where 0 , I are g × g zero and identity matrices, e i is the i th unit vector, and matrix T has 2 on the main diagonal and -1 on the first super- and sub-diagonal. In Figure 2, we run our method against the classical model-free gradient-based method [45]. It shows that our model-based policy gradient converges much faster to the ground truth ARE solution K ∗ . In the benchmark example [45], ours converges within 750 iterations, while the classical method takes 2 × 10 4 iterations ( orders of magnitude longer). In Figure 2 (c), we increase the system size by scaling g from 4 to 64. Both methods run on a classical simulator with Intel i910980XE CPU. Our method runs much faster than [45] by nearly 3 orders of magnitude . The code for both methods can be seen at https://github.com/YilingQiao/diff_lqr . Additional numerical results can be found in Appendix I.

6 Conclusion and Future Work

In this paper, we propose the first quantum algorithm for solving linear-quadratic control problems that achieves end-to-end quantum speedups. Our quantum algorithm utilizes an exponentially faster quantum linear dynamics simulator combined with a policy gradient method. Compared to classical approaches relying on matrix factorization and iterations, our method achieves super-quadratic speedup in the large-scale regime (i.e., m /lessmuch n ). Moreover, the hybrid quantum-classical algorithm design makes our algorithm a promising candidate for practical quantum advantage in the near horizon. We also provide numerical evidence to demonstrate the robustness and favorable convergence behavior of our method.

Limitations and Future Work. Accelerating optimal control and reinforcement learning using quantum computers remains an emerging research topic. Our work has focused on the theoretical aspects of quantum advantage for LQR, a classic optimal control problem of fundamental importance in both theory and practice. However, for special cases [17], we have no guarantee that our quantum algorithm still applies, since Lemma 33 may not hold. In the future, we aim to explore both the practical utility of quantum computing for such tasks and its potential for handling more complex optimal control scenarios, such as non-quadratic and nonlinear problems.

Acknowledgment

We thank Kaiqing Zhang for the helpful discussions and the anonymous reviewers for their helpful feedback. This work was supported in part by the U.S. Department of Energy Office of Science, National Quantum Information Science Research Centers, Quantum Systems Accelerator, Air Force Office of Scientific Research (Award#FA9550-21-1-0209), the U.S. National Science Foundation Career Grant (CCF-1942837), a Sloan research fellowship, Meta Research PhD Fellowship, National Science Foundation Graduate Research Fellowship (Grant No. DGE 2236417), the Simons Quantum Postdoctoral Fellowship, a Simons Investigator award (Grant No. 825053), and Dr. Barry Mersky & Captial One E-Nnovate Endowed Professorships.

References

Amira Abbas, Andris Ambainis, Brandon Augustino, Andreas Bärtschi, Harry Buhrman, Carleton Coffrin, Giorgio Cortiana, Vedran Dunjko, Daniel J Egger, Bruce G Elmegreen, et al. Quantum optimization: Potential, challenges, and the path forward, 2023. arXiv:2312.02279.

Dorit Aharonov and Amnon Ta-Shma. Adiabatic quantum state generation and statistical zero knowledge. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing , pages 20-29, 2003.

Dong An, Andrew M Childs, and Lin Lin. Quantum algorithm for linear non-unitary dynamics with near-optimal dependence on all parameters, 2023. arXiv:2312.03916.

Dong An, Jin-Peng Liu, and Lin Lin. Linear combination of Hamiltonian simulation for nonunitary dynamics with optimal state preparation cost. Physical Review Letters , 131(15):150603, 2023.

Rika Antonova, Jingyun Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. Rethinking optimization with differentiable simulation from a global perspective. In Proceedings of The 6th Conference on Robot Learning . PMLR, 2023.

William F Arnold and Alan J Laub. Generalized eigenproblem algorithms and software for algebraic Riccati equations. Proceedings of the IEEE , 72(12):1746-1754, 1984.

Brandon Augustino, Giacomo Nannicini, Tamás Terlaky, and Luis F Zuluaga. Quantum interior point methods for semidefinite optimization. Quantum , 7:1110, 2023.

Peter Benner, Zvonimir Bujanovic, Patrick Kurschner, and Jens Saak. A numerical comparison of different solvers for large-scale, continuous-time algebraic Riccati equations and lqr problems. SIAM Journal on Scientific Computing , 42(2):A957-A996, 2020.

Dominic W Berry. High-order quantum algorithm for solving linear differential equations. Journal of Physics A: Mathematical and Theoretical , 47(10):105301, 2014.

Steven J Bradtke, B Erik Ydstie, and Andrew G Barto. Adaptive linear quadratic control using policy iteration. In Proceedings of 1994 American Control Conference-ACC'94 , volume 3, pages 3475-3479. IEEE, 1994.

Shantanav Chakraborty, András Gilyén, and Stacey Jeffery. The power of block-encoded matrix powers: improved regression techniques via faster Hamiltonian simulation, 2018. arXiv:1804.01973.

El Amine Cherrat, Iordanis Kerenidis, and Anupam Prakash. Quantum reinforcement learning via policy iteration. Quantum Machine Intelligence , 5(2):30, 2023.

Andrew M Childs, Robin Kothari, and Rolando D Somma. Quantum algorithm for systems of linear equations with exponentially improved dependence on precision. SIAM Journal on Computing , 46(6):1920-1950, 2017.

Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning , pages 1029-1038. PMLR, 2018.

Arjan Cornelissen, Yassine Hamoudi, and Sofiene Jerbi. Near-optimal quantum algorithms for multivariate mean estimation. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 33-43, 2022.

Nikolaus Correll. Parameter estimation and optimal control of swarm-robotic systems: A case study in distributed task allocation. In 2008 IEEE International Conference on Robotics and Automation , pages 3302-3307. IEEE, 2008.

Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633-679, 2020.

Karthik Elamvazhuthi and Spring Berman. Optimal control of stochastic coverage strategies for robotic swarms. In 2015 ieee international conference on robotics and automation (icra) , pages 1822-1829. IEEE, 2015.

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International conference on machine learning , pages 1467-1476. PMLR, 2018.

Franziska Feitzinger, Timo Hylla, and Ekkehard W Sachs. Inexact Kleinman-Newton method for Riccati equations. SIAM Journal on Matrix Analysis and Applications , 31(2):272-288, 2009.

Greg Foderaro, Pingping Zhu, Hongchuan Wei, Thomas A Wettergren, and Silvia Ferrari. Distributed optimal control of sensor networks for dynamic target tracking. IEEE Transactions on Control of Network Systems , 5(1):142-153, 2016.

András Gilyén, Yuan Su, Guang Hao Low, and Nathan Wiebe. Quantum singular value transformation and beyond: exponential improvements for quantum matrix arithmetics. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages 193-204, 2019.

Vittorio Giovannetti, Seth Lloyd, and Lorenzo Maccone. Quantum random access memory. Physical review letters , 100(16):160501, 2008.

Aram W Harrow, Avinatan Hassidim, and Seth Lloyd. Quantum algorithm for linear systems of equations. Physical review letters , 103(15):150502, 2009.

Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Başar. Toward a theoretical foundation of policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems , 6:123-158, 2023.

Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan RaganKelley, and Frédo Durand. DiffTaichi: Differentiable programming for physical simulation. In ICLR , 2020.

Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable simulation for system identification and visuomotor control. International Conference on Learning Representations (ICLR) , 2021.

Sofiene Jerbi, Arjan Cornelissen, M¯ aris Ozols, and Vedran Dunjko. Quantum policy gradient algorithms, 2022. arXiv:2212.09328.

Amara Katabarwa, Katerina Gratsea, Athena Caesura, and Peter D Johnson. Early fault-tolerant quantum computing, 2023. arXiv:2311.14814.

Iordanis Kerenidis and Anupam Prakash. Quantum recommendation systems, 2016. arXiv:1603.08675.

Youngseok Kim, Andrew Eddins, Sajant Anand, Ken Xuan Wei, Ewout Van Den Berg, Sami Rosenblatt, Hasan Nayfeh, Yantao Wu, Michael Zaletel, Kristan Temme, et al. Evidence for the utility of quantum computing before fault tolerance. Nature , 618(7965):500505, 2023.

David Kleinman. On an iterative technique for Riccati equation computations. IEEE Transactions on Automatic Control , 13(1):114-115, 1968.

Hari Krovi. Improved quantum algorithms for linear and nonlinear differential equations. Quantum , 7:913, 2023.

Peter Lancaster and Leiba Rodman. Algebraic Riccati equations . Clarendon press, 1995.

Alan Laub. A Schur method for solving algebraic Riccati equations. IEEE Transactions on automatic control , 24(6):913-921, 1979.

Jiaqi Leng, Ethan Hickman, Joseph Li, and Xiaodi Wu. Quantum Hamiltonian Descent, 2023. arXiv:2303.01471.

Jiaqi Leng, Yuxiang Peng, Yi-Ling Qiao, Ming Lin, and Xiaodi Wu. Differentiable analog quantum computing for optimization and control. Advances in Neural Information Processing Systems , 35:4707-4721, 2022.

William Levine and Michael Athans. On the determination of the optimal constant output feedback gains for linear multivariable systems. IEEE Transactions on Automatic control , 15(1):44-48, 1970.

Xiantao Li and Chunhao Wang. Efficient quantum algorithms for quantum optimal control. In International Conference on Machine Learning , pages 19982-19994. PMLR, 2023.

Fu Lin, Makan Fardad, and Mihailo R Jovanovic. Augmented lagrangian approach to design of structured optimal state feedback gains. IEEE Transactions on Automatic Control , 56(12):2923-2929, 2011.

Jin-Peng Liu, Herman Øie Kolden, Hari K Krovi, Nuno F Loureiro, Konstantina Trivisa, and Andrew M Childs. Efficient quantum algorithm for dissipative nonlinear differential equations. Proceedings of the National Academy of Sciences , 118(35):e2026805118, 2021.

Guang Hao Low and Yuan Su. Quantum eigenvalue processing, 2024. arXiv:2401.06240.

Nico Meyer, Christian Ufrecht, Maniraman Periyasamy, Daniel D Scherer, Axel Plinge, and Christopher Mutschler. A survey on quantum reinforcement learning, 2022. arXiv:2211.03464.

Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R Jovanović. Convergence and sample complexity of gradient methods for the model-free linearquadratic regulator problem, 2019. arXiv:1912.11899.

Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R Jovanović. Convergence and sample complexity of gradient methods for the model-free linearquadratic regulator problem. IEEE Transactions on Automatic Control , 67(5):2435-2450, 2021.

Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning , pages 7805-7817. PMLR, 2021.

Kirsten Morris and Carmeliza Navasca. Iterative solution of algebraic Riccati equations for damped systems. In Proceedings of the 45th IEEE Conference on Decision and Control , pages 2436-2440. IEEE, 2006.

Kirsten A Morris. Introduction to feedback control . Academic Press, Inc., 2000.

Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Efficient differentiable simulation of articulated bodies. In ICML , 2021.

Max Simchowitz, Horia Mania, Stephen Tu, Michael I Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. In Conference On Learning Theory , pages 439-473. PMLR, 2018.

Valeria Simoncini. Computational methods for linear matrix equations. SIAM Review , 58(3):377-441, 2016.

Sanghyun Son, Yi-Ling Qiao, Jason Sewall, and Ming C. Lin. Differentiable hybrid traffic simulation. ACM Transactions on Graphics (TOG) , 2022.

Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 20668-20696. PMLR, 2022.

Joran van Apeldoorn and András Gilyén. Quantum algorithms for zero-sum games, 2019. arXiv:1904.03180.

Joran Van Apeldoorn, András Gilyén, Sander Gribling, and Ronald de Wolf. Quantum SDP-solvers: Better upper and lower bounds. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 403-414. IEEE, 2017.

R Vinifa and A Kavitha. Linear quadratic regulator based current control of grid connected inverter for renewable energy applications. In 2016 International Conference on Energy Efficient Technologies for Sustainability (ICEETS) , pages 106-111. IEEE, 2016.

Xin Wen, Xuting Sun, Yige Sun, and Xiaohang Yue. Airline crew scheduling: Models and algorithms. Transportation research part E: logistics and transportation review , 149:102304, 2021.

Simon Wiedemann, Daniel Hein, Steffen Udluft, and Christian Mendl. Quantum policy iteration via amplitude estimation and grover search-towards quantum advantage for reinforcement learning, 2022. arXiv:2206.04741.

Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. In International Conference on Learning Representations , 2023.

Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojciech Matusik, Shinjiro Sueda, and Pulkit Agrawal. An End-to-End Differentiable Framework for Contact-Aware Robot Design. In Proceedings of Robotics: Science and Systems , Virtual, July 2021.

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. In International Conference on Learning Representations , 2021.

Yao Zhang and Qiang Ni. Recent advances in quantum machine learning. Quantum Engineering , 2(1):e34, 2020.

Appendices

A LQR theory

Definition 6. Given a positive number a > 0, we define the sublevel set S K ( a ) := { K ∈ R m × n : f ( K ) ≤ a } . We have the following useful bound on K .

Lemma 8. Over the sublevel set S K ( a ) of the LQR objective function f ( K ) , we have

Lemma 9. For any K ∈ S K ( a ) , we have

where ν is the same as in (28).

Proof. By [45, Lemma 2], we have

Lemma 10 (PL condition) . Fix a > 0 . For any K ∈ S K ( a ) , we have

where µ f > 0 is a constant that only depends on the problem data and a .

Proof. See [45, Remark 2].

B Implementation of block-encoded matrices

Lemma 11. Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. For a fixed a > 0 , suppose that K ∈ S K is a stabilizing policy stored in a quantum-accessible data structure. Then, we can implement

Proof. By [22, Lemma 48], we can implement a ( s, ε )-block-encoding of A (or B,Q,R ) in cost O (poly log( n, s/ε )). Also, due to [22, Lemma 50], we can implement a ( ‖ K ‖ F , ε )-block-encoding of K in cost O (poly log( n, 1 /ε )). Therefore, by [22, Lemma 52, 53], a ( s ( ‖ K ‖ F +1) , ε )-block-encoding of A -BK can be implement in cost O (poly log( n, s/ε )). Similarly, we can implement a ( s ( ‖ K ‖ 2 F + 1) , ε )-block-encoding of Q + K /latticetop RK in cost O (poly log( n, s/ε )).

Remark 1. We observe that the block-encodings of A -BK and Q + K /latticetop RK can be implemented with high precisions, i.e., in cost O (poly log(1 /ε )). To simplify our technical arguments, we assume these block-encodings can be implemented with no error, i.e., we can implement a ( s ( ‖ K ‖ F +1) , 0)-block-encoding of A -BK (or a ( s ( ‖ K ‖ 2 F +1) , 0)-block-encoding of Q + K /latticetop RK in cost O (poly log( n, s )).

where the constant

C Matrix exponential based on quantum linear system solver

C.1 Block-encoding for matrix inverse

Lemma 12 (Modified from [42], Lemma 11) . Let C be a matrix such that C/α C is blockencoded by O C with some normalization factor α C . Then we can implement a ( O ( α C -1 ) , ε ) -block-encoding of C -1 using

queries to O C . Here κ C is the condition number of C .

Suppose we have an ( α, ε )-block-encoding U M that block-encodes M . Denote we have ‖ Λ ‖ ≤ ε . Then we can see U M as an ( α, 0)-block-encoding of M +Λ /α . So with the lemma above, we have the theorem below:

queries to U M .

Proof. From the lemma above we know we can implement a (˜ α M -1 , ε 2 )-block-encoding as ˜ U M -1 for ( M +Λ /α ), where ˜ α M -1 = O ( α M -1 ). To analyze the error,

C.2 Introduction to quantum linear system solver

The first quantum algorithm for solving linear differential equations was proposed by [9]. Since this work was done, several refinements have been made. Among these, [33] significantly loosen the requirement of performing the algorithm. We notice that this whole algorithm can be written in the form of block-encoding. Basically, for a linear differential equation upon appropriate discretization method and the method of line, one may construct a big matrix A s.t.

Then it is easy to see that the matrix inverse A -1 satisfies

Because u (0) is arbitrarily chosen, we conclude

Suppose A -1 is block-encoded in some U A -1 , we have

Here /lscript satisfy 2 /lscript = T/h , where T is the simulation time and h is the step size. Then it is obvious that U A -1 is also a block-encoding of e khA .

C.3 Matrix exponential construction

Lemma 14 (Modified from [33]) . If we have a block-encoding U L that block-encodes L defined as

then the ( α L -1 , ε ) -block-encoding U L -1 that block-encodes L -1 satisfies

for any r ≥ m +1 . Let 2 s -1 = m , thus

which means that (( X ⊗ I s -1 ⊗ I a ⊗ I ) U L -1 ) is a block-encoding of e TA with its normalization factor as α L -1 and error being at most ε .

Proof. The proof of the correctness of L can be found in [33].

Now we turn to construct the block-encoding for L .

Lemma 15. Assume we can query the ( α A , 0) -block-encoding U A that encodes A and hα A = O (1) , then we can construct a ( O ( k 1 . 5 ) , k 1 / 2 /epsilon1 ) -block-encoding for L defined in (40) using O ( k log(1 //epsilon1 )) queries of U A and same queries to additional elementary gates.

Proof. First we need to block-encode M 1 . We will use the ADD operator defined as

and the controlled-rotation U R as

Notice that

Post-select on the second register and the ancilla qubits of U A , we have

If we apply the operator on a state, namely

So we can apply one more multi-controlledX gate to flip the flag register for the control register being | 0 〉 , then the whole thing becomes a ( α A h, 0)-block-encoding for M 1 , using just one query to U A . We can then easily generate the block-encoding of I -M 1 by LCU, with a 1 + α A h normalization factor. By the analysis in [33], we know ‖ I -M 1 ‖‖ ( I -M 1 ) -1 ‖ ≤ 2 k . Using Lemma 12, we can implement a ( O ( k ) , /epsilon1 1 )-block-encoding of ( I -M 1 ) -1 using O ( k log(1 /ε 1 )) queries to the block-encoding of I -M 1 thus the same queries to U A .

For the block-encoding of M 2 , since it is sparse, we may directly implement it through the sparse input model. We may just assume the normalization factor to be √ k and there is no error. Then we can implement a ( O ( k 3 / 2 ) , √ kε 1 )-block-encoding of ( I -M 1 ) -1 M 2 using O ( k log(1 /ε 1 )) queries to U A and other additional gates.

For the block-encoding for N , we can see the normalization factor would be the same scaling as M 2 ( I -M 1 ) -1 , thus we implemented a ( O ( k 3 / 2 ) , O ( √ kε 1 ))-block-encoding of N .

Now if we use QSVT to perform the matrix inverse, just as described in Theorem 13, we can implement an ( α L -1 , ε 2 + α 2 L -1 α L -ε 1 α L -1 √ kε 1 )-block-encoding U L -1 that encodes L -1 in cost O ( κ L log(1 /ε 2 )) queries to U L , i.e. O ( κ L log(1 /ε 2 ) k log(1 /ε 1 )) queries to U A .

In order to perform the inverse of L , we need to know the condition number of L .

Lemma 16 (Modified from [33], Theorem 3 & 4) . Suppose E is the solution operator block-encoded by U L -1 that approximates e AT and m is the number of steps. Let

where we have

and

Proof. The proof can be seen at [33, Theorem 4].

Theorem 17. Let matrix A be a Hurwitz matrix with sup t ‖ exp( tA ) ‖ bounded by some constant ρ , and O A is a ( α A , 0) -block-encoding of A . Then we can construct a ( ζT, /epsilon1 ) -blockencoding of e AT using

queries to O A and same queries to other additional elementary gates. Here ζ = O ( α A ρ ) .

Proof. Firstly notice that m = O ( α A T ) and k = O ( log( Tα A ρ/δ ) log log( Tα A ρ/δ ) ) . For a given accuracy ε , we need δ ≤ ε , which could be given by letting

Choose ε 1 = ˜ O ( ε/T 2 ) and ε 2 = ε/ 2, from the discussion above we know we can implement an ( α L -1 , ε )-block-encoding of e TA . Since α L -1 = O ( mρ (1 + δ )), the queries we need is ˜ O ( α A ρT · poly log(1 /ε )).

Remark 2. If we directly use the theorem above to prepare the state | e AT | ψ 〉〉 for some input state | ψ 〉 , the dependence on T would be T 2 , which does not match the optimal scaling. This is actually due to our usage of QSVT to block-encode matrices inverses. If we adopt the common technique of padding in the quantum linear system solver, we can easily improve the current scaling into T 3 / 2 . However, for simplicity, we do not implement these standard improvements in this work, as they does not affect our end-to-end complexity result.

D Quantum eigenvalue transformation for linear differential equations

Here we exploit a technique known as quantum eigenvalue transformation (QEVT) by Low and Su [42] to simulate linear dynamics ˙ x = A x . The algorithm (QEVT) requires that the dynamics is stable under A + A /latticetop ≤ 0, which is stronger than A is Hurwitz stable.

Notice that A + A /latticetop ≤ 0 is the stability under the common 2-norm, and A satisfying the Lyapunov equation A X + X A /latticetop ≤ 0 (16) indicates the stability under the inner product induced by the matrix X . We hence follow [42], use a sequence of polynomial functions (known as the Faber polynomials) to approximate the time-evolution operator e A t .

Lemma 18 (Faber truncation of matrix exponentials, [42, Lemma 27]) . Suppose we have a matrix A where its numerical range W ( A ) := {〈 ψ | A | ψ 〉 ∣ ∣ ‖| ψ 〉‖ = 1 } is enclosed by a Faber region E with associated conformal maps Φ : E c →D c , Ψ : D c →E c and Faber polynomials F s ( z ) . Given t > 0 , let e tz = ∑ ∞ j =0 β j F j ( z ) be the Faber expansion of the complex exponential function e tz . Assume that E is convex and symmetric with respect to the real axis, lying on the left half of the complex plane, then for sufficiently large s ,

where c is a constant determined by the conformal maps.

By Lemma 18, given any ε > 0, it suffices to choose s ∼ O ( t log ( 1 ε ) ) such that

Definition 7. For some matrix A of size N × N with its normalization factor α A , denote the identity matrix of size N × N as I N , and

as the lower shift matrix of size N × N . We then define

and

Note that the conformal map Ψ has its Laurent expansion, and so are Ψ ′ ( ω ) and ω Ψ( ω -1 ). Furthermore, Ψ ′ ( ω ) and ω Ψ( ω -1 ) only contains terms with non-negative exponents. Thus Ψ( L -1 N ) and Ψ ′ ( L -1 N ) are well-defined even though L -1 N is not invertible itself.

With the matrices PAD( A ) and PAD( B ), we can compute the matrix PAD( A ) -1 PAD( B ), say

Note that this matrix has (2 s ) × (2 s ) blocks, and each block of size N × N .

Definition 8. For a matrix A whose numerical range is enclosed by a Faber region E with associated conformal maps Φ : E c →D c , Ψ : D c →E c and Faber polynomials F s ( z ), consider some Faber expansion till s -th order, we then define

as an upper bound on the derivative of Faber polynomials.

Lemma 19. We are able to have a ( ξ, ε ) -block-encoding to ( PAD ( A )) -1 using O ( ξ log ( 1 ε ) ) queries to the block-encoding of A and ˜ O (1) additional elementary gates. Here ξ ∼ O ( ρs ) .

Proof. Since we know that PAD( A ) can be block-encoded with an O (1) normalization factor with arbitrary precision using a single query to the block-encoding of A and a polylogarithmic number of elementary gates, and ‖ (PAD( A )) -1 ‖ = O ( sρ ), the rest of the proof is done by Lemma 12.

Theorem 20 (Modified version of Theorem 10. [42]) . Let matrix A have its numerical range W ( A ) := {〈 ψ | A | ψ 〉 ∣ ∣ ‖| ψ 〉‖ = 1 } enclosed by a Faber region E with associated conformal maps Φ : E c →D c , Ψ : D c →E c and Faber polynomials F s ( z ) . Let p ( z ) = ∑ s -1 k =0 β k F k ( z ) be the Faber expansion of a degree-( s -1) polynomial p and O β | 0 〉 = ∑ s -1 k =0 β k | n -1 -k 〉 / ‖ β ‖ be the oracle preparing the coefficients. Then, we can construct a ( ξ ′ ‖ β ‖ , ‖ β ‖ ε ) -block-encoding of ∑ s -1 k =0 β k F k ( A/α A ) using

queries to O A , one query to O β and ˜ O (1) additional gate. Here, ξ ′ ≤ O ( ρs ) .

Proof. First, we observe that

The X is the quantum Pauli-X gate. Here the circuit has 3 registers, the first one only has one qubit, the second has log 2 s qubits, and the third one matches the size of A .

By [42, Theorem 9], we can implement a block-encoding of PAD( A ) using a single query to the block-encoded matrix A/α A . Also, given a constant

we know that ‖ PAD( A ) -1 ‖ = O ( ρs ). By Lemma 19, we can construct a ( ξ, ε )-block-encoding of PAD( A ) -1 using O ( ξ log(1 //epsilon1 )) queries to the block-encoding of A/α A .

Also, assume we can implement a block-encoding of PAD( B ) with a O (1) normalization factor. Combining these two block-encodings together, we obtain a ( ξ ′ , ε )-block-encoding of PAD( A ) -1 PAD( B ), where ξ ′ ≤ O ( ρs ). Then, by (62), the block-encoding we implemented turns out to be a ( ξ ′ ‖ β ‖ , ‖ β ‖ ε )-block-encoding of ∑ s -1 k =0 β k F k ( A/α A ).

Theorem 21 (Block-encoding of matrix exponential) . Suppose that A ∈ R m × n is a Hurwitz matrix, and O A is a ( α A , 0) -block-encoding of A . Then we can implement a ( ζt, ε ) -blockencoding of e At using

queries to O A , one query to a state preparation oracle O β and ˜ O (1) additional gate. Here ζ = O ( α A poly log(1 //epsilon1 )) .

Proof. First we use Lemma 18 to e α A ( A/α A ) t , then we know the polynomial should be of degree O ( α A t log ( 1 ε 1 ) ) in order to have a ε 1 accuracy. Then leverage Theorem 20, we need O ( ρ ( α A t log ( 1 ε 1 )) log ( 1 ε 2 )) queries to O A to construct a block-encoding of the polynomial. Since the Faber polynomial expansion of e At converges absolutely, without loss of generality, we assume ‖ β ‖ is O (1). Choosing ε 1 = ε 2 and ε 2 = ε 2 ‖ β ‖ , we have a ( ζt, ε )-block-encoding of e At .

E Proof of Theorem 4

Lemma 22 (Select oracle) . Suppose that A is Hurwitz, and O A is an ( α, 0) -block-encoding of A . Let K be a positive integer and τ, ε > 0 are two real-valued scalars. The select oracle as defined in Definition 4 can be implemented using

queries to controlled O A , and ˜ O ( αρτK log( K ) · poly log ( 1 ε )) queries to other additional elementary gates.

Proof. Firstly, t k here attains t k = kτ K . By Theorem 17, for each k = 0 , . . . , K , we can implement a ( ζt k , ε )-block-encoding of e At k using ˜ O ( αρt k · poly log(1 /ε )) queries to O A . We denote this block-encoding as U k . Notice that the select oracle select( A , ε ) can be represented by

where each unitary operator [( I -| k 〉 〈 k | ) ⊗ I + | k 〉 〈 k | ⊗ U k ] in this product is a k -controlled version of U k that can be implemented with ˜ O ( αρt k log( K ) · poly log(1 /ε )) queries to the controlled O A and other additional gates. Therefore, the overall query complexity is

where the last step follows from ∑ K k =0 t k = ∑ K k =0 kτ K = ( K +1) τ 2 .

In the following proof, we will need a state preparation oracle

where λ k = w k k 2 , ‖ λ ‖ 1 = ∑ K k =0 | λ k | and w k = (2 -1 k =0 ,K ) τ 2 K . Since λ k can be expressed as a smooth function of k for k ∈ { 0 , 1 , · · · , K } , this state preparation oracle can be implemented in O (poly log( K )) cost.

Now, we are ready to prove Theorem 4.

Proof. The global error of the trapezoidal rule (see (20)) is proportional to

where

Therefore, to achieve precision ε 1 , the total number of quadrature points is

Now, we consider the following two select oracles, as defined in Definition 4:

where U k (or V k ) is a ( ζt k , ε 2 )-block-encoding of e A t k (or e A /latticetop t k ) for 0 ≤ k ≤ K . Recall that O Ω is a ( η, 0)-block-encoding of Ω, it turns out that

where W k := U k U Ω V k is a ( ηζ 2 t 2 k , 2 ζt k ηε 2 )-block-encoding of the matrix F ( t k ).

Let O λ be the state preparation oracle defined in (68). By the LCU technique [22, Lemma 52], we can implement a ( γ, ∑ K k =0 2 | w k | ηζt k ε 2 )-block-encoding (where γ := ∑ K k =0 | w k | ζ 2 t 2 k η ) of ∑ K k =0 w k e At k Ω e A /latticetop t k using a single query to select( A,ε 2 ), select( A /latticetop , ε 2 ), U Ω , O λ and its inverse.

Notice that thus

Given a positive scalar ε > 0, and set the integration time

Note that then if we choose

where δ > 0 is a constant positive number. By Lemma 3 and the error analysis above, we implement a ( γ, ε )-block-encoding of X ∗ , with

It follows that the overall cost of this block-encoding is

queries to O A and same queries to other additional elementary gates.

F Quantum trace estimation

Here we show how to estimate the trace of a positive definite block-encoded matrix. The main result is in Theorem 25.

Lemma 23. Let U H be an ( α, m, ε H ) -block-encoding of a positive-definite Hermitian matrix H ∈ C n × n and let ε ∈ (0 , 1 3 ] . Let λ min > 0 be a known lower bound on the smallest eigenvalue of H . Provided that ε H ≤ O ( λ min ε 2 log(1 /ε ) ) is sufficiently small (where ε > 0 is a given number), we can construct a (1 , m +2 , ε ) -block-encoding of √ H/α 36 using O ( α λ min log(1 /ε ) ) applications of U H and U † H , a single application of controlledU H , and O ( α ( m +1) λ min log(1 /ε ) ) other oneand two-qubit gates. The description of this block-encoding circuit can be computed classically in time O ( poly ( α λ min log(1 /ε ) )) .

Proof. Define ˜ λ min := λ min /α , such that ˜ λ min I /precedesequal H/α /precedesequal I . We first find a polynomial approximation of f ( x ) := √ x 36 over x ∈ [ ˜ λ min , 1] using [22, Corollary 66]. To use this corollary, let x 0 := 1, r := 1 -˜ λ min , δ := ˜ λ min , and observe that f ( x 0 + x ) = 1 6 √ 1 + x = 1 6 ∑ ∞ /lscript =0 ( 1 / 2 /lscript ) x /lscript whenever | x | ≤ r + δ = 1. Also note that 1 6 ∑ ∞ /lscript =0 ∣ ∣ ∣ ( 1 / 2 /lscript ) ∣ ∣ ∣ ≤ 1 3 =: B . Then there is an efficiently computable polynomial P 0 ∈ C [ x ] of degree d = O ( 1 ˜ λ min log(1 /ε ) ) such that ‖ f ( x ) -P 0 ( x ) ‖ ≤ ε 2 for x ∈ [ ˜ λ min , 1] and ‖ P 0 ( x ) ‖ ≤ ε 2 + B ≤ 1 2 for x ∈ [ -1 , 1]. It follows that defining P ( x ) := Re ( P 0 ( x )) gives | f ( x ) -P ( x ) | [ ˜ λ min , 1] ≤ ε 2 and | P ( x ) | [ -1 , 1] ≤ 1 2 .

Next, we use [22, Theorem 56] to construct construct a unitary V that is a (1 , m +2 , ε/ 2)encoding of P ( H/α ) with the desired complexity, using the promise that ε H is sufficiently small to satisfy 4 d √ ε H /α ≤ ε/ 4. V is then a (1 , m +2 , ε )-block-encoding of √ H/α 36 since

where λ H ∈ R n is the vector of eigenvalues of H .

Lemma 24. Let µ > 0 . Let A be an n -byn Hermitian matrix such that 0 /precedesequal A /precedesequal I and let ˜ V be a (1 , m, µ/ 3) -block-encoding of √ A . Then there exists ˜ U A such that ∣ ∣ ∣ ∣ ∥ ∥ ∥ ( 〈 0 | ⊗ I ) ˜ U A | 0 . . . 0 〉 ∥ ∥ ∥ 2 -Tr( A ) n ∣ ∣ ∣ ∣ ≤ µ that uses 1 query to ˜ V and O (log n ) additional gates.

Proof. Our proof is similar to the proof of [55, Lemma 13]. The idea is to first prepare the maximally entangled state ∑ n i =1 | i 〉| i 〉 / √ n (which requires O (log n ) gates) and then apply the map √ A to the first register. We can assume without loss of generality that µ ≤ 1, otherwise the statement is trivial.

Note that ˜ V 0 := ( 〈 0 | ⊗ m ⊗ I ) ˜ V ( | 0 〉 ⊗ m ⊗ I ) is a µ/ 3-approximation of √ A by definition. We are interested in the probability p of measuring 0 ⊗ m in the first register after applying ˜ V .

It remains to show that p is a good approximation of Tr( A ) /n . For this, we show ˜ V † 0 ˜ V 0 ≈ A . Note that for all matrices B, ˜ B with ‖ B ‖ ≤ 1, we have

Using equation (81) along with equation (82) (letting B = √ A and ˜ B = ˜ V 0 ), we see

Theorem 25. Let U H be an ( α, m, ε ) -block-encoding of a Hermitian matrix H ∈ C n × n , where U H can be implemented using T H elementary gates. Suppose H /follows 0 and let λ min > 0 be a known lower bound on the smallest eigenvalue of H . Then, with probability at least 4 / 5 , we can estimate Tr( H ) to within multiplicative error θ ∈ (0 , 1] in cost

provided that ε ≤ O ( λ 3 min θ 2 α 2 log( α/θλ min ) ) .

Proof. Our proof is similar to the proof of [55, Corollary 10]. Let A := H/α 36 and note that since H/α /followsequal λ min I/α , we have

Let ˜ U A be a (1 , m +2 , θλ min / 216 α )-block-encoding of √ A constructed via Lemma 23. Using Lemma 24, we can construct a unitary circuit ˜ V such that

Therefore we have as well as

Using the amplitude estimation technique of [55, Lemma 9] with p min = λ min / 72 α and µ = θ/ 3 gives us a ˜ p that with probability at least 4/5 satisfies

By combining (86) and (89) and using the triangle inequality, we get

Therefore, 36 αn ˜ p is a multiplicative θ -approximation of Tr( H ).

It remains to show that the complexity statement holds. The construction of ˜ U A via Lemma 23 uses O ( T H · α λ min log( α/θλ min )) elementary gates from the applications of U H , U † H , and controlledU H , and O (( m +1) α λ min log( α/θλ min )) other one- and two-qubit gates. The construction of ˜ V requires an additional O (log n ) gates. The amplitude estimation step requires O ( √ α/λ min /θ ) uses of ˜ V and ˜ V † , and O ( √ α/λ min log n/θ ) additional gates. So the overall complexity is

Noting that T H ≥ m and T H ≥ log n for any non-trivial U H gives the stated bound.

G Quantum gradient estimation

Lemma 26. Given K ∈ S K ( a ) and ‖ K -K ∗ ‖ F > ε , we have

where the constant c = √ 2 µ f νλ min ( R ) /a .

Proof. This lemma follows directly from Lemma 9 and Lemma 10.

Lemma 27 (Block-encoded ∇ f ( K )) . Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. Let K ∈ S K ( a ) be a stabilizing policy stored in a quantum-accessible data structure. Given any ε b > 0 , we can implement a ( γ ∇ , ε b ) -block-encoding of ∇ f ( K ) in cost ˜ O ( a 6 ρ 3 √ κ 11 ε b ) , where γ ∇ ≤ ˜ O ( a 6 ρ 4 κ 6 ) .

we can implement a ( γ X , ε )-block-encoding of X ( K ) in cost ˜ O ( a 2 ρ √ κ 5 ε ) , where γ X ≤ ˜ O ( a 2 ρ 2 κ 3 ). Based on our assumptions on the input procedures and the usage of quantum data structure, we can implement a ( s ‖ K ‖ F , 0)-block-encoding of RK and a ( sγ P , sε )-blockencoding of B /latticetop P ( K ). It follows that a ( γ ∇ , ε b )-block-encoding of ∇ f ( K ) can be implemented in cost ˜ O ( a 3 ρ √ κ 5 ε ) , where

and ε b ≤ O (( a + γ P + γ X γ P ) ε ). To achieve the desired precision, we pass the error parameter ε → ε/ (2( a + γ P + γ X γ P )) to the asymptotic cost.

Definition 9. For a matrix G of size m × n , we could always write is as

where G i is a column vector of size m × 1. Define | G vec 〉 as

Lemma 28 (Matrix Vectorization) . Let U G be a ( γ ∇ , ε ) -block-encoding of ( ∇ f ( K )) /latticetop , and we denote

where r is the number of ancilla qubits required. Then, we can prepare a quantum state | G vec 〉 with Ω(1) success probability using ˜ O ( γ ∇ √ m ‖ G ‖ F ) queries to U G and its inverse.

Proof. First, from Lemma 27 we know that we have U /latticetop G that encodes ∇ f ( K ) /latticetop . Notice that ∇ f ( K ) is of size m × n , so

where r 1 , r 2 , r 3 are the indexes for registers, where the r 1 register has log 2 ( m ) qubits and the r 3 register has log 2 ( n ) qubits(We can always assume m and n are the power of 2). The state |⊥〉 contains the state satisfy

By leveraging amplitude amplification, we can get the state | G vec 〉 using O ( γ ∇ √ m ‖ G ‖ F ) queries to the block-encoding U /latticetop G .

Lemma 29 (Gradient entry estimation) . Denote the state preparation in Lemma 28 for | G vec 〉 as U G vec , we are able to get the classical G with

queries to U G and ˜ O ( γ ∇ ‖ G ‖ F m 3 / 2 n ε r ) additional elementary gates.

Proof. [7, Theorem 2] indicates that we can get G using ˜ O ( mn /epsilon1 r ) queries to the oracle that prepares | G vec 〉 and ˜ O ( mn ε r ) additional elementary gates. Since each U G vec requires

queries, we know the total cost should be

using

Lemma 30 (Gradient norm estimation) . Suppose we have the block-encoding U G defined in Lemma 27, where

then we are able to estimate ‖ G ‖ F up an additive error ε a using

queries to U G and its inverse and also ˜ O ( γ ∇ √ m ‖ G ‖ F ε 2 a +2 ‖ G ‖ F ε a ) elementary gates.

Proof. We still need the computation we did in equation 98, say

According to [55, Lemma 9], we can get the estimation to ( ‖ G ‖ F γ ∇ √ m ) 2 with multiplicative error µ using O ( γ ∇ √ m µ ‖ G ‖ F ) queries to U /latticetop G . Note our estimator to ‖ G ‖ F as a est , the goal we want to achieve is

Consider our estimation to the amplitude as ( a est γ ∇ √ m ) 2 , set µ = /epsilon1 2 a +2 /epsilon1 a ‖ G ‖ F ‖ G ‖ 2 F , we have

This in turn gives us the estimation in 107. And this estimation requires

queries to U G and its inverse. The gate complexity also follows from [55, Lemma 9].

Theorem 31 (Quantum gradient estimation) . Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. Let K ∈ S K ( a ) be a stabilizing policy stored in a quantum-accessible data structure. Provided that ‖ K -K ∗ ‖ > ε , we can compute a θ -robust estimate of ∇ f ( K ) in cost

Proof. We need an estimation to ∇ f ( K ) up to a multiplicative error θ . This requires a few steps as below:

Block-Encoding. From Lemma 27, we know we are able to construct a ( γ ∇ , ε b )-blockencoding to ∇ f ( K ) using

elementary gates. And we also have the estimation that γ ∇ ≤ ˜ O ( a 6 ρ 4 κ 6 ).

Entry Retrieving. The next step is to retrieve the entries in the estimation to ∇ f ( K ). We firstly use the technique introduced in Lemma 28 to get a state of the estimation to | G vec 〉 , and this step requires √

queries. Now we can use Lemma 29 to get the entries in | G vec 〉 up to an additive error /epsilon1 r , using

queries to U G .

Norm Estimation. The final step is to estimate the norm of G . Lemma 30 tells us that we can achieve an additive error ε a to ‖ G ‖ F using

queries to U /latticetop G .

The rest of the theorem is just to assemble all these steps. Note the requirement is

and according to Lemma 26, ‖∇ f ( K ) ‖ F ≥ c/epsilon1 before convergence.

Denote the entries retrieved in the Entry Retrieving step form a matrix G where ‖G‖ F = 1, the norm estimated in the Norm Estimation step as a est , our estimator is then a est G . Choose

we then know that ‖ G -∇ f ( K ) ‖ ≤ cθε 3 , ∇ f ( K ) ≥ cε , then

Thus it is clear to see

And the total gate complexity should be

H Convergence analysis

Lemma 32. Given any K ∈ S K , let G be a θ -robust estimate of ∇ f ( K ) . Then, we have

Proof. It is straightforward to verify that

Lemma 33 (Descent lemma) . Given K ∈ S K ( a ) , and let G be a θ -robust estimate of ∇ f ( K ) . Then, there exists a positive σ m such that for all σ ∈ [0 , σ m ] , we have

where µ , σ m only depends on A,B,R,Q , and a , and µ > σ m .

Proof. This lemma is a direct consequence of Lemma 32 (with θ < 1 / 2) and [45, Proposition 6].

Proposition 34. For any initial stabilizing feedback gain K 0 ∈ S K , we denote a := f ( K 0 ) . Then, there exists constants µ > σ m > 0 , depending only on A,B,Q,R , and a , such that for any fixed σ ∈ [0 , σ m ] , the iterates of Algorithm 1 satisfy

where b is an absolute constant that is independent of k .

Proof. The first part (124a) is a direct corollary of Lemma 33. By Lemma 9 and (124a), we have

where the last step follows from [45, Lamma 2] and b = λ max ( RX ( K 0 )).

Theorem 35 (Quantum policy gradient for LQR) . Assume that we have efficient procedures (as described in Assumption 1) to access the problem data A,B,Q,R in O (poly log( n )) time. Let K 0 ∈ S K be a stabilizing policy. Then, Algorithm 1 outputs an ε -approximate solution to Problem 1 in cost

Proof. Notice that with lemma 33, we need log ( 1 ε ) iterations. By Theorem 31, the quantum gradient estimation subroutine requires ˜ O ( m 1 . 5 n ε 1 . 5 ) elementary gates.

I Extended numerical experiments

I.1 Aircraft Control Problem

We perform another experiment on a practical problem that can be formulated as LQR. Here, we consider the aircraft flight control problem, specifically for pitch angle control. We adopt a linearized model of the aircraft around a steady flight condition. For a small aircraft, the pitch dynamics can be represented by the following state variables: pitch angle θ (rad) and pitch rate q (rad/s). The control input is elevator deflection angle δ (rad). The state-space model can be represented as ˙ x = Ax + Bu , where x = [ θ, q ] /latticetop , u = [ δ ]. We set A = [[0 , 1] , [0 , -0 . 5]] /latticetop , B = [0 , 1] /latticetop , Q = [[10 , 0] , [0 , 1]] /latticetop , and R = [0 . 1]. The plot of our optimization curve is available in the Figure Figure 3. Our method converges faster than the classical method.

(a)

J

-

J

∗

(b)

f

(

K

)

-

f

(

K

∗

)

101

10°

10°

10-1

二

classical

一

classical

10-1

ours

ours

0

20

40

0

20

40

I.2 Relative Errors

We conducted further numerical experiments to understand how the optimality scales with problem size in Figure Figure 4. Here, we scale the number of masses in a spring-mass system from 2 to 4, and the problem dimension scales accordingly from 2 to 8. We measure the optimality by the relative error found in both our method and the classical method. The relative errors are ( J -J ∗ ) /j ∗ and ( f ( K ) -f ( K ∗ )) /f ( K ∗ ). As the dimension scales up, the relative errors increase, while our method consistently outperforms the classical optimization method.

(a) Relative

J

-

J

∗

error

(b) Relative

f

(

K

)

-

f

(

K

∗

) error

10-2

10-2

10-4

classical

10-4

classical

ours

ours

10-6

10-6

2

3

4

2

3

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract accurately reflect the paper's technical contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations and future work of this work are discussed in the last section of the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors are encouraged to create a separate "Limitations" section in their paper.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of technical assumptions is provided in the introduction section (see 'Problem Formulation'). A complete proof of our main theoretical results can be found in the main paper and the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All assumptions should be clearly stated or referenced in the statement of any theorems.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Theorems and Lemmas that the proof relies upon should be properly referenced.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All information needed to reproduce the main experiment results of the paper is disclosed in the 'Numerical Experiments" section. The code for the experiment is attached in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for the experiment is attached in the supplementary material. Guidelines:

The answer NA means that paper does not include experiments requiring code.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

Please see the NeurIPS code and data submission guidelines ( https://nips. cc/public/guides/CodeSubmissionPolicy ) for more details.

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are provided in the 'Numerical Experiments' section. The source code for the experiment is attached in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The full details can be provided either with the code, in appendix, or as supplemental material.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars to reflect the statistical significance of the experiments, see the 'Numerical Experiments' section and Figure 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

The assumptions made should be given (e.g., Normally distributed errors).

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient descriptive information on the computer resources, see the 'Numerical Experiments' section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the societal impacts of the work performed in the 'Conclusion and Future Work' section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper mainly focuses on the theoretical foundation of computer algorithms and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The source code of the numerical experiments is provided in the supplementary material with appropriate documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

The paper should discuss whether and how consent was obtained from people whose asset is used.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces an end-to-end quantum algorithm for linear quadratic control problem. The proposed quantum-assisted differentiable simulator is suitable for large-scale dynamical systems where the dimension of system state is huge. Sample complexity is also provided when apply quantum computation in such problem. Simulated results support its theory.

STRENGTHS:
The quantum application to linear-quadratic problem seems new and provides a lot computation benefit.

WEAKNESSES:
This work only considers linear-quadratic control problem.

QUESTIONS:
What are the optimality plot such as the plots shown in Figure 2(a) and 2(b) for Figure 2(c) when you increase the system dimension?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The paper "Differentiable Quantum Computing for Large-scale Linear Control" introduces a quantum algorithm for linear-quadratic control problems, offering provable speedups. It utilizes a policy gradient method enhanced with a novel quantum subroutine for solving the matrix Lyapunov equation, leading to more accurate and robust gradient estimation than classical methods. The proposed algorithm achieves a super-quadratic speedup, making it the first end-to-end quantum application to linear control problems with demonstrable quantum advantages.

STRENGTHS:
Quantum Speedup: The algorithm achieves a super-quadratic speedup over classical methods, which is a significant advancement in the field of quantum computing for control problems.

Innovative Approach: It introduces a novel quantum-assisted differentiable simulator, enhancing the accuracy and robustness of gradient estimation.

WEAKNESSES:
The experiments are insufficient. The proposed method has only been applied to simple abstract problems, but it would be beneficial to conduct experiments on practical problems in the form of LQR.

QUESTIONS:
1. How does the proposed quantum algorithm handle cases where the sparsity assumptions on matrices A,B,Q,R do not hold?

2. Can the authors provide detailed runtime performance comparisons between the proposed quantum algorithm and state-of-the-art classical algorithms?

3. Comparison with Other Methods: How does the proposed method's stability and convergence rate compare with other existing quantum and classical approaches for solving linear-quadratic control problems?

LIMITATIONS:
The primary limitation of the proposed method is its dependency on the availability of quantum resources and the sparsity assumptions for the matrices involved.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis proposes an end-to-end solution to the quantum-assisted LQR problem. Based on a policy gradient method, the proposed algorithm incorporates a quantum subroutine for solving the matrix Lyapunov equation, achieving a super-quadratic speedup.\n\nSTRENGTHS:\n1. To the best of my knowledge, this is the first end-to-end quantum application to linear control problems with provable quantum advantage.\n\n2. This paper also provides numerical evidence to demonstrate the robustness and favorable convergence behavior of the method.\n\n3. This paper is clearly written.\n\nWEAKNESSES:\n1. While the paper highlights the theoretical advantages of the quantum algorithm, implementing these algorithms on current quantum hardware might pose significant challenges. Today's quantum computers suffer from noise and have a limited number of qubits, which could affect the actual performance and reliability of the algorithm.\n\n2. Although the paper claims a super-quadratic speedup, it's important to verify whether this speedup is practically achievable. Especially in large-scale industrial models, the real-world complexity and scalability of the algorithm are critical issues.\n\n3. This paper is not always well written. For example, in Line 520, \"The evolution of a quantum state can always described by a unitary operator\".\n\nQUESTIONS:\n1. How does the proposed quantum algorithm account for the current limitations of quantum hardware, such as noise and the limited number of qubits? \n2. The paper introduces a novel quantum subroutine for solving the matrix Lyapunov equation. Could you elaborate on the specific technical innovations that this subroutine brings compared to existing quantum algorithms? What are the key theoretical breakthroughs that enable the claimed super-quadratic speedup?\n\nLIMITATIONS:\nThe limitations of this work have been adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of applying quantum computing to linear quadratic regulator (LQR) control. The approach is based on an efficient quantum estimation of the policy gradient. When the dimension n of the state space is large, the proposed approach can achieve orders of magnitude improvement on the time complexity to find the optimal controller compared to existing policy gradient methods.\n\nSTRENGTHS:\nI am not familiar with quantum computing. Since the LQR sample complexity has received much attention recently, this work should be interesting for the learning theory community if the claim about time complexity improvement is correct.\n\nWEAKNESSES:\nSince this work proposes a model-based approach and assumes access to the exact model, including A, B, Q, and R (Algorithm 1), I wonder why the authors compare with the model-free approach in [44], which does not assume knowledge of the model and uses two-point gradient estimation. I don’t think this is a fair comparison.\n\nBesides, I do not understand why the application of the proposed quantum approach is limited to the classic LQR problem. If the real contribution is the first quantum approach to solve the Lyapunov equation (11) efficiently, I think the proposed method can be applied to other important problems like stability analysis. I hope the authors can clarify the most general statement of the main contribution or what limits the application to LQR.\n\nQUESTIONS:\nBesides my comments in the weakness part, I also hope the authors can clarify if the proposed method is robust to estimation errors in A and B if we estimate them from samples (see [Dean, Sarah, et al., 2020]).\n\n[Dean, Sarah, et al., 2020] Dean, Sarah, et al. \"On the sample complexity of the linear quadratic regulator.\" Foundations of Computational Mathematics 20.4 (2020): 633-679.\n\nLIMITATIONS:\nI do not see any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}