Safe Linear-Quadratic Dual Control with Almost Sure Performance Guarantee

1 INTRODUCTION . One of the most fundamental and well-studied problems in optimal control , Linear-Quadratic Regulation ( LQR ) has recently aroused renewed interest in the context of data-driven control and reinforcement learning . Considering it is usually challenging to obtain an exact system model from first principles , and that the system may slowly change over time due to various reasons , e.g. , component wear-out , data-driven regulation of unknown linear systems has become an active research problem in the intersection of machine learning and control , with recent works including e.g. , Dean et al . ( 2019 ) ; Mania et al . ( 2019 ) ; Cohen et al . ( 2019 ) ; Wagenmaker & Jamieson ( 2020 ) . In particular , from the perspective of reinforcement learning theory , the LQR problem has become a standard benchmark for continuous control . In this paper , we focus on the dual control ( Feldbaum , 1960 ) setting , also known as online adaptive control in the literature , where the same policy must be adopted to identify the system parameters and optimize the control objective , leading to the well-known exploration-exploitation dilemma . Recently , it was shown by Simchowitz & Foster ( 2020 ) that the optimal regret for this problem setting scales as Θ̃ ( √ T ) , which can be achieved with probability 1 − δ using a certainty equivalent control strategy , where the learner selects control inputs according to the optimal controller for the current estimate of the system while injecting time-decaying exploration noise . However , the strategy proposed in this work , like those in its predecessors ( Abbasi-Yadkori & Szepesvári , 2011 ; Dean et al. , 2018 ; Mania et al. , 2019 ) , may have a nonzero probability δ of failing . Furthermore , it shall be noticed that δ has been chosen as a fixed design parameter in the aforementioned works , which implies the probability of failing does not converge to zero even if the policy is run indefinitely . The above observation gives rise to the question that we address in this paper : Can we design a learning scheme for LQR dual control , such that the policy adopted in almost every trajectory converges to the optimal policy ? We identify that the above goal can hardly be achieved by a naive certainty equivalent learning scheme . Qualitatively , the system parameters learned from data are always corrupted by random noise , and as a result , the controller proposed in previous works may destabilize the system , albeit with a small probability , causing catastrophic system failure . Based on the above reasoning , we propose a notion of bounded-cost safety for the LQR dual control problem : we recognize a learning scheme to be safe if no destabilizing control policy is applied during the entire learning process . In this paper , we propose a learning scheme that satisfies the above definition of bounded-cost safety , and guarantees both the parameter inference error and the suboptimality gap of controller performance converge to zero almost surely . Our strategy consists of two parts : a safe switched controller and a parameter inference algorithm . The switched controller can be viewed as a safety-augmented version of the certainty equivalent controller : it normally selects control inputs according to the optimal linear feedback for the currently estimated system parameters , but falls back to a conservative controller for several steps when the actual state deviates significantly from the target state . We prove that this switching strategy ensures the bounded-cost safety of the learning process , while only inducing a suboptimality gap that decays exponentially as the switching threshold increases . For the parameter inference part , in contrast to the direct least-squares approach for estimating the matrices A , B widely adopted in the literature , we estimate the Markov parameters , also known as the impulse response of the system , based on the cross-correlation between the exploration noise and the system output , and establish the almost-sure convergence using a law of large numbers for martingales . We prefer this approach for its clarity in physical meaning and simplicity of convergence analysis , but we do not foresee substantial difficulty in replacing our parameter inference module with standard least-squares . We prove that under the above described learning scheme , the parameter inference error scales asO ( T−1/4+ ) , while the suboptimality gap of control performance scales as O ( T−1/2+ ) , where T is the number of time steps , and is an arbitrarily small positive number . Both the above results match the corresponding asymptotic rates in Simchowitz & Foster ( 2020 ) , which provides an rate-optimal algorithm for online LQR in the high-probability regime . The main contributions of this paper are as follows : 1 . We propose a practical notion of safety for the LQR dual control problem that has not been considered in the literature , and provide an instance of safe learning scheme based on a switching strategy . 2 . We prove almost sure convergence rates of parameter inference error and suboptimality gap of control performance for our scheme , which match the corresponding optimal rates in the high-probability regime . To the best of our knowledge , this is the first analysis of the almost sure convergence rate for online LQR . The rest of this paper is organized as follows : Section 2 gives a brief introduction of LQR , formulates the LQR dual control problem , and defines the performance metrics as well as the notion of boundedcost safe learning scheme . Section 3 presents and interprets our algorithm . Section 4 states the main theoretical results and characterizes the convergence rates . Section 5 provides simulation results on an industrial process example to illustrate the effectiveness of our proposed strategy . Section 6 summarizes the related literature . Finally , Section 7 gives concluding remarks and discusses future directions . 2 PROBLEM FORMULATION . We consider the control of the following discrete-time linear system : xk+1 = Axk +Buk + wk , ( 1 ) where xk ∈ Rn is the state vector , uk ∈ Rp is the input vector , and wk ∈ Rn is the process noise . We assume x0 ∼ N ( 0 , X0 ) , wk ∼ N ( 0 , W ) , and that x0 , w0 , w1 , . . . are pairwise independent . We also assume w.l.o.g . that ( A , B ) is controllable . We consider control policies of the form π : Rn × Rq → Rp × Rq , ( uk , ξk+1 ) = π ( xk , ξk ) , ( 2 ) which can be either deterministic or stochastic , where ξk ∈ Rq is the internal state of the policy . Notice that we allow the flexibility of the policy being non-Markovian with the introduction of ξk , and we also use the simplified notation uk = π ( xk ) if π is Markovian . The performance of a policy π can be characterized by the infinite-horizon quadratic cost Jπ = lim sup T→∞ 1 T E [ T−1∑ k=0 x > k Qxk + u > k Ruk ] , ( 3 ) where Q 0 , R 0 are known weight matrices specified by the system operator . We denote the optimal cost and the optimal control law by J∗ = inf π Jπ , π∗ ∈ arg min π Jπ . ( 4 ) It is well-known that the optimal policy is a linear function of the state π∗ ( x ) = K∗x , with associated cost J∗ = tr ( WP ∗ ) , where P ∗ is the solution to the discrete-time algebraic Riccati equation P ∗ = Q+A > P ∗A−A > P ∗B ( R+B > P ∗B ) −1 B > P ∗A , ( 5 ) and the linear feedback control gain K∗ can be determined by K∗ = − ( R+B > PB ) −1 B > P ∗A . ( 6 ) Based on the definitions above , we can use Jπ−J∗ to measure of the suboptimality gap of a specific policy π . In the online LQR setting , the system and input matrices A , B are assumed unknown , and the learning process can be viewed as deploying a sequence of time-varying policies { πk } with the dual objectives of exploring the system parameters and stabilizing the system . To characterize the safety of a learning process , we make the definitions below : Definition 1 . A policy π is destabilizing if Jπ = +∞ . Definition 2 . A learning process applying policies { πk } ∞k=0 is bounded-cost safe , if πk is not destabilizing for any time k and for any realization of the noise process . Notice that Definition 1 is a generalization of the common notion of destabilizing linear feedback gain , i.e. , π ( x ) = Kx is destabilizing when ρ ( A + BK ) ≥ 1 , which is equivalent to Jπ = +∞ . Based on the notion of destabilizing policies , we propose the concept of bounded-cost safety in Definition 2 , which requires that destabilizing policies , or policies with unbounded cost , are never applied . It should be pointed out that bounded-cost safety does not guarantee the stability of trajectories , but is an indicator of the reliability of a learning scheme . We assume the system is open-loop strictly stable , i.e. , ρ ( A ) < 1 . Indeed , if there is a known stabilizing linear feedback gain K0 , then we can apply the dual control scheme on the pre-stabilized system ( A + BK0 , B ) instead of ( A , B ) . Existence of such a known stabilizing linear feedback gain a standard assumption in previous works on online LQR ( Mania et al. , 2019 ; Simchowitz & Foster , 2020 ) , and relatively easy to establish through coarse system identification ( Dean et al. , 2019 ; Faradonbeh et al. , 2018b ) or adaptive stabilization methods ( Faradonbeh et al. , 2018a ; 2019 ) . 3 ALGORITHM . The complete algorithm we propose for LQR dual control is presented in Algorithm 1 . The modules in the algorithm will be described later in this section . 3.1 SAFE CONTROL POLICY . This subsection describes the policy for determining the control input uk . The first n + p steps are a warm-up period where purely random inputs are injected . Afterwards , in each step , we inject a exploitation term ũ plus a polynomial-decaying exploratory noise ( k+ 1 ) −βζ , where the decay rate β ∈ ( 0 , 1/2 ) is a constant . The exploitation term ũ is a modified version of the certainty equivalent control input K̂kxk , and this modification , described in Algorithm 2 , is crucial to the safety of the learning process , which we will detail below . Algorithm 1 Safe LQR dual control Input : State dimension n , input dimension p , exploratory noise decay rate β 1 : for k = 0 , 1 , . . . , n+ p− 1 do 2 : ξk+1 ← 0 3 : Apply control input uk ← ( k + 1 ) −βζk , where ζk ∼ N ( 0 , Ip ) 4 : for k = n+ p , n+ p+ 1 , . . . do 5 : Observe the current state xk 6 : for τ = 0 , 1 , . . . , n+ p− 1 do 7 : Ĥk , τ ← 1 k − τ k∑ i=τ+1 ( i− τ ) β [ xi − τ−1∑ t=0 Ĥk , tũi−t−1 ] ζ > i−τ−1 8 : Reconstruct Âk , B̂k from Ĥk,0 , . . . , Ĥk , n+p−1 using Algorithm 3 9 : Compute certainty equivalent feedback gain K̂k by replacing A , B with Âk , B̂k in ( 5 ) , ( 6 ) 10 : Determine policy πk ( · , · ) ← π ( · , · ; k , K̂k , β ) , where π is described by Algorithm 2 11 : ( uk , ξk+1 ) ← πk ( xk , ξk ) ; record ũk ← ũ , ζk ← ζ , where ũ , ζ are the corresponding vari- ables generated when executing the policy 12 : Apply control input uk Algorithm 2 Safe policy π ( x , ξ ; k , K , β ) Input : Arguments : system state x , policy internal state ξ ; Parameters : step k , linear feedback gain K , exploratory noise decay rate β Output : Control input u and next policy internal state ξ′ , i.e. , ( u , ξ′ ) = π ( x , ξ ; k , K , β ) 1 : if ξ > 0 then 2 : ũ← 0 , ξ′ ← ξ − 1 3 : else 4 : if max { ‖K‖ , ‖x‖ } ≥ log k then 5 : ũ← 0 , ξ′ ← blog kc 6 : else 7 : ũ← Kx , ξ′ ← 0 8 : u← ũ+ ( k + 1 ) −βζ , where ζ ∼ N ( 0 , I ) In short , we stop injecting the exploitation input for blog kc+ 1 consecutive steps , if either the state norm ‖xk‖ or the norm of the feedback gain ‖K̂k‖ exceeds the threshold log k. Recall from ( 2 ) that we use ξk to denote the internal state of the policy , and in Algorithm 1 , ξk is a counter that records how many steps are left in the “ non-action ” period . Essentially , we utilize the innate stability of the system to prevent the state from exploding catastrophically . This “ non-action ” mechanism is a critical feature of our control design , without which the controller learned from data may destabilize the system , albeit with a small probability , causing system failure in practice and forbidding the establishment of almost sure performance guarantees in theory . We provide an ablation study of this “ non-action ” mechanism in Section 5 . We choose both the switching threshold and the length of the “ non-action ” period to be timegrowing . The enlarging threshold corresponds to diminishing degree of conservativeness , which is essential for the policy performance to converge to the optimal performance . Meanwhile , the prolonging “ non-action ” period rules out the potential oscillation of state caused by the frequent switching of the controller ( see Appendix B for an illustrative example of this oscillation phenomenon ) . In particular , it can be shown that the suboptimality gap incurred by the switching strategy scales as O ( tM exp ( −cM2 ) ) , where M is the switching threshold , t is the length of the “ non-action ” period , and c is a system-dependent constant ( see Lemma 10 in Appendix A.1.2 ) . With both M and t growing as O ( log k ) , the contribution of our switching strategy to the overall suboptimality gap is merely Õ ( 1 ) .

Summary: This paper studies an adaptive control for LQR and provides an algorithm to converge to the optimal policy almost surely in an asymptotic sense, where the convergence rate is also provided. The paper assumes that the system is open-loop stable, and switches to zero control input (with exponentially decaying excitation noises) for a while when the state or control gain is larger than O(log(k)), which can steer the state to a smaller neighbourhood of 0 due to the open-loop stability of the original system. Theoretically, the paper shows that any switched policy generated by the algorithm guarantees a finite infinite-horizon-averaged cost, which is called "safe" in this paper. Further, the paper establishes the strong consistency of the Markov parameter estimation in Alg 3 with an error convergence rate. Besides, the paper shows that the learned policies converge to the optimal policy at a rate of O(1/sqrt(k)) in terms of the infinite-horizon-averaged cost. Lastly, the performance is evaluated with different parameter choices of the algorithm.