Continuous Control with Contexts, Provably

( √ T ) regret bound in the online setting where T is the num- ber of environments the agent played . This also implies after playing Õ ( 1/ 2 ) environments , the agent is able to transfer the learned knowledge to obtain an -suboptimal policy for an unseen environment . To our knowledge , this is first provably efficient algorithm to build a decoder in the continuous control setting . While our main focus is theoretical , we also present experiments that demonstrate the effectiveness of our algorithm . 1 INTRODUCTION . Humans are able to solve a new task without any training based on previous experience in similar tasks . Our intelligent agent should be able do the same , learning from previous experience , adapting to the new ones and improving the performance as the agent gains more experience . This is a challenging problem as we need to design an adaptation mechanism which is fundamentally different from classical supervised learning methods . A common approach is to build a decoder so that once the agent sees a description of new task , i.e. , the context of the new task , the decoder turns the context into a succinct representation of the new task , based on which the agent is able to design a policy to solve the task . Note this procedure resembles how a human solves a new task . For example , if a human wants to push an object on a table , the human first sees the object and the table ( context ) . Then , in his/her mind , the context becomes a representation of this task , e.g. , a sense of weight of the object . Based on this representation , the human can easily reason about how much force to exert on the object . This general approach has been applied in practice . For example , Wu et al . ( 2018 ) studied the visual navigation task and built a Bayesian model that takes the context of new environments and outputs the policy that enables the agent to navigate . Killian et al . ( 2016 ) used this approach to develop personalized medicine policies for HIV treatment . While this is a promising approach , currently we only have limited theoretical understanding . The approach can be formulated in Contextual Markov Decision Process ( CMDP ) framework ( Hallak et al. , 2015 ) . Recently , there is a line of work gave provable guarantees for CMDP ( Abbasi-Yadkori & Neu , 2014 ; Hallak et al. , 2015 ; Dann et al. , 2018 ; Modi et al. , 2018 ; Modi & Tewari , 2019 ) . These work all studied tabular MDPs , and use function approximation , e.g. , linear functions , generalized linear models , etc , to model the mapping from the context to the probability transition matrix . A major drawback of these work is that they are restricted to the tabular setting and thus can only deal with discrete environments . Therefore , they can hardly model real-world continuous control tasks , like the task of pushsing an object as we described above . A natural question arises : Can we design a provably efficient decoder for continuous control problems ? In this paper , we make an important step towards answering this question . We study the fundamental setting in continuous control , linear quadratic regulator ( LQR ) . LQR is arguably the most widely used framework in continuous control , as LQR easily models real world physical phenomena , e.g. , the pushing object task we described earlier . We propose a new algorithm that builds a decoder , so that for a new LQR task , the decoder takes LQR ’ s context and outputs a representation based on which the agent can easily infer a near-optimal policy for new continuous control tasks . In the training phase , we build the decoder via a sequence of LQRs ( in an online fashion ) with unknown parameters . For each new task , we first use the current decoder to build the representation of this task , infer a policy based on this representation and use this policy to do control for this episode . There are two crucial components in our algorithm . First , after each episode , we will refine the estimate of the decoder based on the observations from this episode . Second , it is crucial to use a upper confidence bound ( UCB ) estimator of the decoder to build the representation so that the agent can perform a near-optimal trade-off between exploration and exploitation . In this way , we provably show the decoder improves the performance as it experiences more training tasks . Formally , we show our algorithm enjoys Õ ( √ T ) regret ( the difference between the cumulative rewards of our algorithm and the unknown optimal policy on every seen environment ) bound in the online setting . Moreover , the algorithm is able to obtain an -suboptimal policy for an unseen LQR environment after playing Õ ( −2 ) environments . To our knowledge , this is the first provably efficient algorithm that builds a decoder for continuous control environments . Empirically , we simulate several physical environments to illustrate the effectiveness of our algorithm . Organization This paper is organized as follows . In Section 2 , we discuss related work . In Section 3 , we formally describe the problem setup . In Section 4 , we present our algorithm and its theoretical guarantees . In Section 5 , we use simulation on physical environments to demonstrate the effectiveness of our approach . We conclude in Section 6 and defer most technical proofs to the appendix . 2 RELATED WORK . Recently there is a large body of literature focusing on learning for control in LQR systems . The first work we are aware of is Fiechter ( 1997 ) which studies the sample complexity of LQR in the offline setting . For the online setting , where the agent can only obtain the next state starting from the present state , the first near-optimal regret bound ( Õ ( √ T ) ) is due to Abbasi-Yadkori & Szepesvári ( 2011 ) , which studies the learning problem in the infinite-horizon average-case cost setting . Later on , a sequence of papers ( Tu & Recht , 2017 ; Dean et al. , 2017 ; 2018 ; Tu & Recht , 2018 ; AbbasiYadkori et al. , 2018 ; Cohen et al. , 2019 ) studied this problem in similar settings , improved efficiency of the algorithms and characterized the gap between model-free and model-based approaches . Building an agent that quickly adapts to new environment has received increasing interest in the machine learning community . Taylor & Stone ( 2009 ) gave a summary for the literature status before 2009 . More recently , a sequence of theory papers Lehnert & Littman ( 2018 ) ; Spector & Belongie ( 2018 ) ; Abel et al . ( 2018 ) ; Lehnert et al . ( 2019 ) studied the transferability of reward knowledge , state-abstraction , and model features for Markov decision processes . Please also refer to references in paper cited above for more details . There are also some experimental works , e.g. , Santara et al . ( 2019 ) ; Yu et al . ( 2018 ) ; Wu et al . ( 2018 ) ; Gamrian & Goldberg ( 2018 ) , studying how to transfer knowledge from seen tasks to unseen tasks . Nevertheless , we are not aware of any study on how to provably perform continuous control with contexts . 3 PRELIMINARIES . Notations . We begin by introducing necessary notations . We write [ h ] to denote the set { 1 , . . . , h } . We use Id ∈ Rd×d to denote the d-dimensional identity matrix . We use 0d×d′ to represent the allzero matrix in Rd×d′ . If it is clear from the context , we omit the subscript d × d′ . Let ‖·‖2 denote the Euclidean norm of a vector in Rd . For a symmetric matrix A , let ‖A‖op denote its operator norm and λi ( A ) denote its i-th eigenvalue . Throughout the paper , all sets are multisets , i.e. , a single element can appear multiple times . Finite Horizon Linear Quadratic Regulator . We now formally define the finite horizon Linear Quadratic Regulator ( LQR ) problem . In the LQR problem , there is a state space X ⊂ Rd and a closed action space U ⊂ Rd′ . Suppose we always start from the initial state x1 = xinit ∈ X and play for H steps . Then at a state xh ∈ X , if an action uh ∈ U is played , the next state is given by xh+1 = Axh +Buh + wh+1 , ( 1 ) where A , B are matrices of proper dimension and wh+1 is a zero-mean random vector . Here A , B can be viewed as the succinct representation of this LQR because as will be explained below , given A , B , we can easily infer the optimal policy for this LQR . For simplicity , we additionally denote M = [ A , B ] , and yh = [ x > h , u > h ] > ∈ Rd+d′ . Now the state transition can be rewritten as xh+1 = Myh + wh+1 . For the ease of presentation , we assume that the covariance matrix of noise vectorwh+1 is E ( wh+1w > h+1 ) = Id . Our analysis follows similarly if the covariance matrix is not Id ( see e.g . Remark 3 of Abbasi-Yadkori & Szepesvári ( 2011 ) ) . After each step , the player receives an immediate cost x > hQhxh +u > hRhuh , where Qh , Rh are positive definite ( PD ) matrices of proper dimensions . Throughout the paper , we assume the agent knows Qh and Rh for all h. At a terminal state xH , there is no action to be played , and the player receives a terminal cost x > HQHxH , where QH is a PD matrix of proper dimension . The goal of the player is to find a policy π : ( X ×U ) ∗×X → U , which is a function that maps the trajectory { ( xi , ui ) } h−1i=1 ∪ { xh } to the next action uh , such that the following objectives are minimized : { Jπh ( M , x ) : = E [ ( H−1∑ h′=h x > hQhxh + u > hRhuh ) + x > HQfxH ∣∣∣∣ xh = x ] } h∈ [ H ] , where the action uh is given by uh = π [ ( x1 , u1 ) , ( x2 , u2 ) , . . . , ( xh−1 , uh−1 ) , xh ] , and the expectation is over the randomness of wh and π . It is well-known that the optimal policy π∗ is Markovian Puterman ( 2014 ) , i.e. , it only depends on the present state . For an unconstrained action space U , we have ∀x ∈ X , h ∈ [ H − 1 ] : π∗h ( M , x ) : = Kh ( M ) x where M = [ A , B ] and Kh ( M ) is a matrix that will be defined shortly . It is also known ( see e.g . Bertsekas ( 1996 ) ) that the optimal cost function J∗h ( x ) : = J π∗ h ( x ) is given by J∗h ( M , x ) : = x > Ph ( M ) x+ Ch ( M ) = inf π Jπh ( M , x ) ( 2 ) where Ph ( M ) = { Qh +A > Ph+1 ( M ) A−A > Ph+1B ( Rh +B > Ph+1 ( M ) B ) −1B > Ph+1 ( M ) A h < H QH h = H ( 3 ) and Ch ( M ) = { Ch+1 ( M ) + Ewh+1 [ w > h+1Ph+1 ( M ) wh+1 ] h < H 0 h = H . We now define Kh ( M ) as Kh ( M ) : = − ( Rh +B > Ph+1 ( M ) B ) −1B > Ph+1 ( M ) A . ( 4 ) Note that the optimal value Equation ( 2 ) satisfies Bellman equations , ∀h ∈ [ H − 1 ] : J∗h ( M , x ) = x > Qhx+ π∗ ( x ) > Rhπ∗ ( x ) + E [ J∗h+1 ( Ax+Bπ ∗ ( x ) + w ) ] and ∀h ∈ [ H − 1 ] : J∗h ( M , x ) = x > Qhx+ min u E [ u > Rhu+ J∗h+1 ( Ax+Bu+ w ) ] . Now we have shown that if we are given A and B , then we can obtain the optimal policy directly . In this paper , we deal with setting where A and B are unknown and we need to use decoder to decode A and B from the contexts of the current LQR , as specified below . Learning to Control LQR with Contexts In the continuous control with contexts setting , in each episode we observe a context ( C , D ) ∼ µ , where µ is a distribution on Rp×d × Rp′×d′ . The context [ C , D ] encodes the information of the environment . Formally , the representation ( [ A , B ] ) of this environment can be decoded from the context via a decoding matrix Θ∗ ∈ Rd× ( p+p ′ ) : [ A , B ] = Θ∗ · [ C 0p×d′ 0p′×d D ] . ( 5 ) From now on , to emphasize that the representation of LQR can be decoded from Θ∗ , we write MΘ∗ , C , D : = Θ∗ · [ C 0p×d′ 0p′×d D ] = [ A , B ] . ( 6 ) If it is clear from the context , we ignore [ C , D ] for notational simplicity . Note the optimal decoder Θ∗ is unknown to the agent and the goal is to learn Θ∗ from contexts and interactions with the environment . Below we formally define the problem that we study . Definition 3.1 ( Contextual Transfer Learning Problem ) . Build an agent that plays onK LQR games ( one trajectory per game ) with context pairs { ( C ( 1 ) , D ( 1 ) ) , ( C ( 2 ) , D ( 2 ) ) , . . . , ( C ( K ) , D ( K ) ) } ∼ µ , for some integer K ≥ 0 such that for another new context pair ( C , D ) ∼ µ , the agent outputs a policy π based on ( C , D ) which satisfies E [ Jπh ( MΘ∗ , C , D , x1 ) − J∗h ( MΘ∗ , C , D , x1 ) ] ≤ for some given target accuracy > 0 . Here K is the sample complexity which ideally scales polynomially with and problem-dependent parameters . The performance of the agent can also be measured by regret , as defined below . Regret ( KH ) : = K∑ k=1 J π̃ ( k ) 1 ( MΘ∗ , C ( k ) , D ( k ) , x1 ) − J∗1 ( MΘ∗ , C ( k ) , D ( k ) , x1 ) , ( 7 ) where π̃ ( k ) is the policy played at episode k by the agent . This quantity measurse the sub-optimality of policies the agent played in the first K episodes . Remark 3.1 . We consider matrix-type linear maps from context to the representation only for sake of presentation . Our algorithm and analysis can be readily extended to other linear maps , e.g. , [ A∗ ( C ) , B∗ ( D ) ] : = f ( C , D ) for some unknown linear function f .

Summary: In order to generalize the RL agent to unseen environment, in this work the authors studied the theoretical learning problem of building a decoder on top of linear continuous control using linear quadratic regulator (LQR). They presented a simple, UCB-based algorithm that refines the estimates of the encoder while doing LQR and balances  the exploration-exploitation trade-off. In the online setting, the proposed algorithm has a O(\sqrt{T}) regret bound, where T is the number of environments the agent played. This also implies after certain exploration, the agent is able to transfer the learned knowledge to obtain a near-optimal policy to an unseen environment. To justify their theoretical bounds the authors also present experiments that demonstrate the effectiveness of the algorithm.