Safe Linear-Quadratic Dual Control with Almost Sure Performance Guarantee

1 INTRODUCTION . One of the most fundamental and well-studied problems in optimal control , Linear-Quadratic Regulation ( LQR ) has recently aroused renewed interest in the context of data-driven control and reinforcement learning . Considering it is usually challenging to obtain an exact system model from first principles , and that the system may slowly change over time due to various reasons , e.g. , component wear-out , data-driven regulation of unknown linear systems has become an active research problem in the intersection of machine learning and control , with recent works including e.g. , Dean et al . ( 2019 ) ; Mania et al . ( 2019 ) ; Cohen et al . ( 2019 ) ; Wagenmaker & Jamieson ( 2020 ) . In particular , from the perspective of reinforcement learning theory , the LQR problem has become a standard benchmark for continuous control . In this paper , we focus on the dual control ( Feldbaum , 1960 ) setting , also known as online adaptive control in the literature , where the same policy must be adopted to identify the system parameters and optimize the control objective , leading to the well-known exploration-exploitation dilemma . Recently , it was shown by Simchowitz & Foster ( 2020 ) that the optimal regret for this problem setting scales as Θ̃ ( √ T ) , which can be achieved with probability 1 − δ using a certainty equivalent control strategy , where the learner selects control inputs according to the optimal controller for the current estimate of the system while injecting time-decaying exploration noise . However , the strategy proposed in this work , like those in its predecessors ( Abbasi-Yadkori & Szepesvári , 2011 ; Dean et al. , 2018 ; Mania et al. , 2019 ) , may have a nonzero probability δ of failing . Furthermore , it shall be noticed that δ has been chosen as a fixed design parameter in the aforementioned works , which implies the probability of failing does not converge to zero even if the policy is run indefinitely . The above observation gives rise to the question that we address in this paper : Can we design a learning scheme for LQR dual control , such that the policy adopted in almost every trajectory converges to the optimal policy ? We identify that the above goal can hardly be achieved by a naive certainty equivalent learning scheme . Qualitatively , the system parameters learned from data are always corrupted by random noise , and as a result , the controller proposed in previous works may destabilize the system , albeit with a small probability , causing catastrophic system failure . Based on the above reasoning , we propose a notion of bounded-cost safety for the LQR dual control problem : we recognize a learning scheme to be safe if no destabilizing control policy is applied during the entire learning process . In this paper , we propose a learning scheme that satisfies the above definition of bounded-cost safety , and guarantees both the parameter inference error and the suboptimality gap of controller performance converge to zero almost surely . Our strategy consists of two parts : a safe switched controller and a parameter inference algorithm . The switched controller can be viewed as a safety-augmented version of the certainty equivalent controller : it normally selects control inputs according to the optimal linear feedback for the currently estimated system parameters , but falls back to a conservative controller for several steps when the actual state deviates significantly from the target state . We prove that this switching strategy ensures the bounded-cost safety of the learning process , while only inducing a suboptimality gap that decays exponentially as the switching threshold increases . For the parameter inference part , in contrast to the direct least-squares approach for estimating the matrices A , B widely adopted in the literature , we estimate the Markov parameters , also known as the impulse response of the system , based on the cross-correlation between the exploration noise and the system output , and establish the almost-sure convergence using a law of large numbers for martingales . We prefer this approach for its clarity in physical meaning and simplicity of convergence analysis , but we do not foresee substantial difficulty in replacing our parameter inference module with standard least-squares . We prove that under the above described learning scheme , the parameter inference error scales asO ( T−1/4+ ) , while the suboptimality gap of control performance scales as O ( T−1/2+ ) , where T is the number of time steps , and is an arbitrarily small positive number . Both the above results match the corresponding asymptotic rates in Simchowitz & Foster ( 2020 ) , which provides an rate-optimal algorithm for online LQR in the high-probability regime . The main contributions of this paper are as follows : 1 . We propose a practical notion of safety for the LQR dual control problem that has not been considered in the literature , and provide an instance of safe learning scheme based on a switching strategy . 2 . We prove almost sure convergence rates of parameter inference error and suboptimality gap of control performance for our scheme , which match the corresponding optimal rates in the high-probability regime . To the best of our knowledge , this is the first analysis of the almost sure convergence rate for online LQR . The rest of this paper is organized as follows : Section 2 gives a brief introduction of LQR , formulates the LQR dual control problem , and defines the performance metrics as well as the notion of boundedcost safe learning scheme . Section 3 presents and interprets our algorithm . Section 4 states the main theoretical results and characterizes the convergence rates . Section 5 provides simulation results on an industrial process example to illustrate the effectiveness of our proposed strategy . Section 6 summarizes the related literature . Finally , Section 7 gives concluding remarks and discusses future directions . 2 PROBLEM FORMULATION . We consider the control of the following discrete-time linear system : xk+1 = Axk +Buk + wk , ( 1 ) where xk ∈ Rn is the state vector , uk ∈ Rp is the input vector , and wk ∈ Rn is the process noise . We assume x0 ∼ N ( 0 , X0 ) , wk ∼ N ( 0 , W ) , and that x0 , w0 , w1 , . . . are pairwise independent . We also assume w.l.o.g . that ( A , B ) is controllable . We consider control policies of the form π : Rn × Rq → Rp × Rq , ( uk , ξk+1 ) = π ( xk , ξk ) , ( 2 ) which can be either deterministic or stochastic , where ξk ∈ Rq is the internal state of the policy . Notice that we allow the flexibility of the policy being non-Markovian with the introduction of ξk , and we also use the simplified notation uk = π ( xk ) if π is Markovian . The performance of a policy π can be characterized by the infinite-horizon quadratic cost Jπ = lim sup T→∞ 1 T E [ T−1∑ k=0 x > k Qxk + u > k Ruk ] , ( 3 ) where Q 0 , R 0 are known weight matrices specified by the system operator . We denote the optimal cost and the optimal control law by J∗ = inf π Jπ , π∗ ∈ arg min π Jπ . ( 4 ) It is well-known that the optimal policy is a linear function of the state π∗ ( x ) = K∗x , with associated cost J∗ = tr ( WP ∗ ) , where P ∗ is the solution to the discrete-time algebraic Riccati equation P ∗ = Q+A > P ∗A−A > P ∗B ( R+B > P ∗B ) −1 B > P ∗A , ( 5 ) and the linear feedback control gain K∗ can be determined by K∗ = − ( R+B > PB ) −1 B > P ∗A . ( 6 ) Based on the definitions above , we can use Jπ−J∗ to measure of the suboptimality gap of a specific policy π . In the online LQR setting , the system and input matrices A , B are assumed unknown , and the learning process can be viewed as deploying a sequence of time-varying policies { πk } with the dual objectives of exploring the system parameters and stabilizing the system . To characterize the safety of a learning process , we make the definitions below : Definition 1 . A policy π is destabilizing if Jπ = +∞ . Definition 2 . A learning process applying policies { πk } ∞k=0 is bounded-cost safe , if πk is not destabilizing for any time k and for any realization of the noise process . Notice that Definition 1 is a generalization of the common notion of destabilizing linear feedback gain , i.e. , π ( x ) = Kx is destabilizing when ρ ( A + BK ) ≥ 1 , which is equivalent to Jπ = +∞ . Based on the notion of destabilizing policies , we propose the concept of bounded-cost safety in Definition 2 , which requires that destabilizing policies , or policies with unbounded cost , are never applied . It should be pointed out that bounded-cost safety does not guarantee the stability of trajectories , but is an indicator of the reliability of a learning scheme . We assume the system is open-loop strictly stable , i.e. , ρ ( A ) < 1 . Indeed , if there is a known stabilizing linear feedback gain K0 , then we can apply the dual control scheme on the pre-stabilized system ( A + BK0 , B ) instead of ( A , B ) . Existence of such a known stabilizing linear feedback gain a standard assumption in previous works on online LQR ( Mania et al. , 2019 ; Simchowitz & Foster , 2020 ) , and relatively easy to establish through coarse system identification ( Dean et al. , 2019 ; Faradonbeh et al. , 2018b ) or adaptive stabilization methods ( Faradonbeh et al. , 2018a ; 2019 ) . 3 ALGORITHM . The complete algorithm we propose for LQR dual control is presented in Algorithm 1 . The modules in the algorithm will be described later in this section . 3.1 SAFE CONTROL POLICY . This subsection describes the policy for determining the control input uk . The first n + p steps are a warm-up period where purely random inputs are injected . Afterwards , in each step , we inject a exploitation term ũ plus a polynomial-decaying exploratory noise ( k+ 1 ) −βζ , where the decay rate β ∈ ( 0 , 1/2 ) is a constant . The exploitation term ũ is a modified version of the certainty equivalent control input K̂kxk , and this modification , described in Algorithm 2 , is crucial to the safety of the learning process , which we will detail below . Algorithm 1 Safe LQR dual control Input : State dimension n , input dimension p , exploratory noise decay rate β 1 : for k = 0 , 1 , . . . , n+ p− 1 do 2 : ξk+1 ← 0 3 : Apply control input uk ← ( k + 1 ) −βζk , where ζk ∼ N ( 0 , Ip ) 4 : for k = n+ p , n+ p+ 1 , . . . do 5 : Observe the current state xk 6 : for τ = 0 , 1 , . . . , n+ p− 1 do 7 : Ĥk , τ ← 1 k − τ k∑ i=τ+1 ( i− τ ) β [ xi − τ−1∑ t=0 Ĥk , tũi−t−1 ] ζ > i−τ−1 8 : Reconstruct Âk , B̂k from Ĥk,0 , . . . , Ĥk , n+p−1 using Algorithm 3 9 : Compute certainty equivalent feedback gain K̂k by replacing A , B with Âk , B̂k in ( 5 ) , ( 6 ) 10 : Determine policy πk ( · , · ) ← π ( · , · ; k , K̂k , β ) , where π is described by Algorithm 2 11 : ( uk , ξk+1 ) ← πk ( xk , ξk ) ; record ũk ← ũ , ζk ← ζ , where ũ , ζ are the corresponding vari- ables generated when executing the policy 12 : Apply control input uk Algorithm 2 Safe policy π ( x , ξ ; k , K , β ) Input : Arguments : system state x , policy internal state ξ ; Parameters : step k , linear feedback gain K , exploratory noise decay rate β Output : Control input u and next policy internal state ξ′ , i.e. , ( u , ξ′ ) = π ( x , ξ ; k , K , β ) 1 : if ξ > 0 then 2 : ũ← 0 , ξ′ ← ξ − 1 3 : else 4 : if max { ‖K‖ , ‖x‖ } ≥ log k then 5 : ũ← 0 , ξ′ ← blog kc 6 : else 7 : ũ← Kx , ξ′ ← 0 8 : u← ũ+ ( k + 1 ) −βζ , where ζ ∼ N ( 0 , I ) In short , we stop injecting the exploitation input for blog kc+ 1 consecutive steps , if either the state norm ‖xk‖ or the norm of the feedback gain ‖K̂k‖ exceeds the threshold log k. Recall from ( 2 ) that we use ξk to denote the internal state of the policy , and in Algorithm 1 , ξk is a counter that records how many steps are left in the “ non-action ” period . Essentially , we utilize the innate stability of the system to prevent the state from exploding catastrophically . This “ non-action ” mechanism is a critical feature of our control design , without which the controller learned from data may destabilize the system , albeit with a small probability , causing system failure in practice and forbidding the establishment of almost sure performance guarantees in theory . We provide an ablation study of this “ non-action ” mechanism in Section 5 . We choose both the switching threshold and the length of the “ non-action ” period to be timegrowing . The enlarging threshold corresponds to diminishing degree of conservativeness , which is essential for the policy performance to converge to the optimal performance . Meanwhile , the prolonging “ non-action ” period rules out the potential oscillation of state caused by the frequent switching of the controller ( see Appendix B for an illustrative example of this oscillation phenomenon ) . In particular , it can be shown that the suboptimality gap incurred by the switching strategy scales as O ( tM exp ( −cM2 ) ) , where M is the switching threshold , t is the length of the “ non-action ” period , and c is a system-dependent constant ( see Lemma 10 in Appendix A.1.2 ) . With both M and t growing as O ( log k ) , the contribution of our switching strategy to the overall suboptimality gap is merely Õ ( 1 ) .

Summary: This paper addresses the problem of combining system identification and optimal control in an online framework for stable linear systems with full-state sensing. Despite the restrictions of linearity, stability, and full-state sensing, the proposed problem formulation is still challenging and I am unaware of any results which provide the same optimality guarantees provided by the authors in the online framework. The paper itself is unusually well-conceived, well-written, well-structured, is clear, and makes very efficient use of notation.