Lipschitz Lifelong Reinforcement Learning

1 INTRODUCTION . Lifelong Reinforcement Learning ( RL ) is an online problem where an agent faces a series of RL tasks , drawn sequentially . Transferring the knowledge of prior experience while solving new tasks is a key question in that setting ( Lazaric , 2012 ; Taylor and Stone , 2009 ) . We elaborate on the intuitive idea that similar tasks should allow a large amount of transfer . An agent able to compute online a similarity measure between source tasks and the current target task should be able to perform transfer . By measuring the amount of inter-task similarity , we design a novel method for value transfer , practically deployable in the online Lifelong RL setting . Specifically , we introduce a metric between MDPs and prove that the optimal Q-value function is Lipschitz continuous with respect to MDPs . This property allows to compute a provable upper-bound on the optimal value function of a target task , given the learned optimal value function of a source task . Knowing this upper bound allows to accelerate the convergence of an RMax algorithm ( Brafman and Tennenholtz , 2002 ) . This transfer method is non-negative ( it can not cause performance degradation ) as the computed upper bound does not underestimate the optimal Q-value function . Our contributions are as follows . First , we study theoretically the Lipschitz continuity of the optimal value function in the task space ( Section 3 ) . Then , we use this continuity property to propose a value-transfer method based on a local distance between MDPs ( Section 4 ) . Full knowledge of both MDPs is not required and the transfer is non-negative , which makes the method both practical and safe . In Section 4.2 , we build a PAC-MDP algorithm called Lipschitz RMax , applying this transfer method online in the Lifelong RL setting . We provide sample and computational complexity bounds and showcase the algorithm in Lifelong RL experiments ( Section 5 ) . 2 BACKGROUND AND RELATED WORK . Reinforcement Learning ( RL ) ( Sutton and Barto , 1998 ) is a framework for sequential decision making . The problem is typically modeled as a Markov Decision Process ( MDP ) ( Puterman , 2014 ) consisting in a 4-tuple 〈S , A , R , T 〉 where S is a state space , A an action space , Ras is the expected reward of taking action a in state s and T ass′ is the transition probability of reaching state s ′ when taking action a in state s. Without loss of generality , we assume Ras ∈ [ 0 , 1 ] . Given a discount factor γ ∈ [ 0 , 1 ) , the expected cumulative return ∑ t γ tRatst obtained along a trajectory starting with state s and action a is noted Q ( s , a ) and called the Q-function . The optimal Q-function Q∗ is the highest attainable expected return from s , a and V ∗ ( s ) = maxa∈AQ∗ ( s , a ) is the optimal value function in s. Lifelong RL ( Silver et al. , 2013 ; Brunskill and Li , 2014 ) is the problem of experiencing online a series of MDPs drawn from an unknown distribution . Each time an MDP is sampled , a classical RL problem takes place where the agent is able to interact with the environment to maximize its expected return . In this setting , it is reasonable to think that knowledge gained on previous MDPs could be re-used to improve the performance in new MDPs . In this paper , we provide a novel method for such transfer by characterizing the way the optimal Q-function can evolve across tasks . We restrict the scope of the study to the case where sampled MDPs share the same state-action space S ×A . For brevity , we will refer indifferently to MDPs , models or tasks , and write them M = 〈R , T 〉 . Using a metric between MDPs has the appealing characteristic of quantifying the amount of similarity between tasks , which intuitively should be linked to the amount of transfer achievable . Song et al . ( 2016 ) define a metric based on the bi-simulation metric introduced by Ferns et al . ( 2004 ) and the Wasserstein metric ( Villani , 2008 ) . Value transfer is performed between states with low bi-simulation distances . However , this metric requires knowing both MDPs completely and is thus unusable in the Lifelong RL setting where we expect to perform transfer before having learned the current MDP . Further , the transfer technique they propose does allow negative transfer ( see Appendix , Section A ) . Carroll and Seppi ( 2005 ) also define a value-transfer method based on a measure of similarity between tasks . However , this measure is not computable online and thus not applicable to the Lifelong RL setting . Mahmud et al . ( 2013 ) and Brunskill and Li ( 2013 ) propose MDP clustering methods respectively using a metric quantifying the regret of running the optimal policy of one MDP in the other MDP and the L1 norm between the MDP models . An advantage of clustering is to prune the set of possible source tasks . They use their approach for policy transfer , which differs from the value-transfer method proposed in this paper . Ammar et al . ( 2014 ) learn the model of a source MDP and view the prediction error on a target MDP as a dissimilarity measure in the task space . Their method makes use of samples from both tasks and is not readily applicable to the online setting considered in this paper . Lazaric et al . ( 2008 ) provide a practical method for sample transfer , computing a similarity metric reflecting the probability of the models to be identical . Their approach is applicable in a batch RL setting as opposed to our online setting . The approach developed by Sorg and Singh ( 2009 ) is very similar to ours in the sense that they prove bounds on the optimal Q-function for new tasks , assuming that both MDPs are known and that a soft homomorphism exists between the state spaces . Brunskill and Li ( 2013 ) also provide a method that can be used for Q-function bounding in multi-task RL . Abel et al . ( 2018 ) present the MaxQInit algorithm , providing transferred bounds on the Q-function with high probability while preserving PAC-MDP guarantees . Given a set of solved tasks , they derive the probability that the maximum over the Q-values of previous MDPs is an upper bound on the current task ’ s optimal Q-function . This results in a method for non-negative transfer with high probability once enough tasks have been sampled . 3 LIPSCHITZ CONTINUITY OF Q-FUNCTIONS . The intuition we build on is that similar MDPs should have similar optimal Q-functions . Formally , this insight can be translated into a continuity property of the optimal Q-functions over the MDP spaceM . The remainder of this section mathematically formalizes this intuition that will be used in the next Section to derive a practical method for value transfer . To that end , we introduce a local pseudo-metric characterizing the distance between the models of two MDPs at a particular state-action pair . A reminder and a detailed discussion on the metrics ( and related objects ) used herein can be found in the Appendix , Section B . Definition 1 . Given two tasks M = 〈R , T 〉 and M̄ = 〈R̄ , T̄ 〉 , and a function f : S → R+ , we define the pseudo-metric between models at ( s , a ) ∈ S ×A w.r.t . f as : DMM̄f ( s , a ) , |Ras − R̄as |+ ∑ s′∈S f ( s′ ) |T ass′ − T̄ ass′ | . ( 1 ) This pseudo-metric is relative to a positive function f . We implicitly cast this definition in the context of discrete state spaces . The extension to continuous spaces is straightforward but beyond the scope of this paper . Let Q∗M denote the optimal Q-function of MDP M ∈M . Proposition 1 ( Local pseudo-Lipschitz continuity ) . For two MDPs M , M̄ , for all ( s , a ) ∈ S ×A , |Q∗M ( s , a ) −Q∗M̄ ( s , a ) | ≤ ∆ MM̄ ( s , a ) , ( 2 ) with the MDPs local pseudo-metric ∆MM̄ ( s , a ) , min { dM̄M ( s , a ) , d M M̄ ( s , a ) } , and the local MDP dissimilarity dM̄M : S ×A → R is the unique solution to the following fixed-point equation for d : d ( s , a ) = DMM̄γV ∗ M̄ ( s , a ) + γ ∑ s′∈S T ass′ max a′∈A d ( s′ , a′ ) . ( 3 ) All the proofs of the paper can be found in the Appendix . This result establishes that the distance between the optimal Q-functions of two MDPs at ( s , a ) ∈ S ×A is controlled by a local dissimilarity between the MDPs . The latter follows a fixed-point equation ( Equation 3 ) , which can be solved by Dynamic Programming ( DP ) ( Bellman , 1957 ) . Note that , although the local MDP dissimilarity dM̄M is asymmetric , ∆MM̄ ( s , a ) is a pseudo-metric , hence the name pseudo-Lipschitz continuity . Similar results for the value function of a fixed policy and the optimal value function V ∗M can easily be derived , as well as a global pseudo-Lipschitz continuity property ( Appendix , Sections D and E ) . Thus , overall , the optimal Q-functions of two close MDPs ( in the sense of Equation 1 ) are themselves close to each other . A direct consequence in Lifelong RL is that previously solved tasks can help bound the value function of a new task . Even a partially learned Q-function can be used for that purpose if error bounds are known or if it provably overestimates the true Q∗ . The next section exploits this property to build a PAC-MDP algorithm that performs provably non-negative transfer between successive tasks and accelerates learning . 4 TRANSFER USING THE LIPSCHITZ CONTINUITY . A purpose of value transfer , when interacting online with a new MDP , is to initialize the value function and drive the exploration to accelerate learning . We aim to exploit value transfer in a method guaranteeing three conditions : C1 . the resulting algorithm is PAC-MDP ( Strehl et al. , 2009 ) ; C2 . the transfer accelerates learning ; C3 . the transfer is non-negative . From Proposition 1 , one can naturally define a local upper bound on the optimal Q-function of an MDP given the optimal Q-function of another MDP . Definition 2 . Given two tasks M and M̄ , for all ( s , a ) ∈ S ×A , the Lipschitz upper bound on Q∗M induced by Q∗ M̄ is defined as UM̄ ( s , a ) ≥ Q∗M ( s , a ) with : UM̄ ( s , a ) , Q ∗ M̄ ( s , a ) + ∆ MM̄ ( s , a ) . ( 4 ) The optimism in the face of uncertainty principle leads to consider that the long-term expected return from any state is the 11−γ maximum return , unless proven otherwise . The RMax algorithm ( Brafman and Tennenholtz , 2002 ) in particular explores an MDP so as to shrink this upper bound . RMax is a model-based , online RL algorithm with PAC-MDP guarantees ( Strehl et al. , 2009 ) which means that convergence to near-optimal policy is guaranteed in a polynomial number of steps . It relies on an optimistic model initialization that yields an optimistic upper bound U on the optimal Q-function , then acts greedily w.r.t . U . By default , it takes the maximum value U ( s , a ) = 11−γ but any tighter upper bound is admissible . Thus , shrinking U with Equation 4 is expected to improve the learning speed for new tasks in Lifelong RL . In RMax , during the resolution of a task M , S × A is split into a subset of known state-action pairs K and its complement Kc of unknown pairs . A state-action pair is known if the number of collected reward and transition samples allows estimating an -accurate model in L1-norm with probability higher than 1− δ . We refer to and δ as the RMax precision parameters . This translates into a threshold nknown on the number of visits n ( s , a ) to a pair s , a that are necessary to reach this precision . Given the experience of a set of m MDPs M̄ = { M̄1 , . . . , M̄m } , we define the total bound as the minimum over all the Lipschitz bounds induced by each previous MDP . Proposition 2 . Given a partially known task M = 〈R , T 〉 , the set of known state-action pairs K , and the set of Lipschitz bounds on Q∗M induced by previous tasks { UM̄1 , . . . , UM̄m } , the function Q defined below is an upper bound on Q∗M for all s , a ∈ S ×A . Q ( s , a ) , { Ras + γ ∑ s′∈S T ass′ max a′ Q ( s′ , a′ ) if ( s , a ) ∈ K , U ( s , a ) otherwise , ( 5 ) with U ( s , a ) = min { 1 1−γ , UM̄1 ( s , a ) , . . . , UM̄m ( s , a ) } . Traditionally in RMax , Equation 5 is solved to a precision Q via Value Iteration . This yields a function Q that is a valid heuristic ( provable upper bound on Q∗M ) for the exploration of MDP M .

Summary: This paper studies the problem of reusing prior experience in Reinforcement learning. The main contribution of this work is to introduce a novel metric between MDPs. They use the value functions Lipschitz continuity in the task space w.r.t the metric and show the connection. The theoretical results of this paper show the value transfer and they can derive results for the same. They demonstrate the benefit of their method in experiments.