Provably efficient multi-task reinforcement learning with model transfer

1 Introduction . In many real-world applications , reinforcement learning ( RL ) agents can be deployed as a group to complete similar tasks at the same time . For example , in healthcare robotics , robots are paired with people with dementia to perform personalized cognitive training activities by learning their preferences [ 42 , 21 ] ; in autonomous driving , a set of autonomous vehicles learn how to navigate and avoid obstacles in various environments [ 27 ] . In these settings , each learning agent alone may only be able to acquire a limited amount of data , while the agents as a group have the potential to collectively learn faster through sharing knowledge among themselves . Multi-task learning [ 7 ] is a practical framework that can be used to model such settings , where a set of learning agents share/transfer knowledge to improve their collective performance . Despite many empirical successes of multi-task RL ( see , e.g. , [ 51 , 28 , 27 ] ) and transfer learning for RL ( see , e.g. , [ 26 , 39 ] ) , a theoretical understanding of when and how information sharing or knowledge transfer can provide benefits remains limited . Exceptions include [ 16 , 6 , 11 , 17 , 32 , 25 ] , which study multi-task learning from parameter or representation transfer perspectives . However , these works still do not provide a completely satisfying answer : for example , in many application scenarios , the reward structures and the environment dynamics are only slightly different for each task—this is , however , not captured by representation transfer [ 11 , 17 ] or existing works on clustering-based parameter transfer [ 16 , 6 ] . In such settings , is it possible to design provably efficient multi-task RL algorithms that have guarantees never worse than agents learning individually , while outperforming the individual agents in favorable situations ? In this work , we formulate an online multi-task RL problem that is applicable to the aforementioned settings . Specifically , inspired by a recent study on multi-task multi-armed bandits [ 43 ] , we formulate the �-Multi-Player Episodic Reinforcement Learning ( abbreviated as �-MPERL ) problem , in which all tasks share the same state and action spaces , and the tasks are assumed to be similar—i.e. , the dissimilarities between the environments of different tasks ( specifically , the reward distributions and transition dynamics associated with the players/tasks ) are bounded in terms of a dissimilarity parameter � ≥ 0 . This problem not only models concurrent RL [ 34 , 16 ] as a special case by taking � = 0 , but also captures richer multi-task RL settings when � is nonzero . We study regret minimization for the �-MPERL problem , specifically : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 . We identify a problem complexity notion named subpar state-action pairs , which captures the amenability to information sharing among tasks in �-MPERL problem instances . As shown in the multi-task bandits literature ( e.g. , [ 43 ] ) , inter-task information sharing is not always helpful to reduce the players ’ collective regret . Subpar state-action pairs , intuitively speaking , are clearly suboptimal for all tasks , for which we can robustly take advantage of ( possibly biased ) data collected for other tasks to achieve a lower regret in a certain task . 2 . In the setting where the dissimilarity parameter � is known , we design a model-based algorithm MULTI-TASK-EULER ( Algorithm 1 ) , which is built upon state-of-the-art algorithms for learning single-task Markov decision processes ( MDPs ) [ 3 , 46 , 36 ] , as well as algorithmic ideas of model transfer in RL [ 39 ] . MULTI-TASK-EULER crucially utilizes the dissimilarity assumption to robustly take advantage of information sharing among tasks , and achieves regret upper bounds in terms of subpar state-action pairs , in both ( value function suboptimality ) gap-dependent and gap-independent fashions . Specifically , compared with a baseline algorithm that does not utilize information sharing , MULTI-TASK-EULER has a regret guarantee that : ( 1 ) is never worse , i.e. , it avoids negative transfer [ 33 ] ; ( 2 ) can be much superior when there are a large number of subpar state-action pairs . 3 . We also present gap-dependent and gap-independent regret lower bounds for the �-MPERL problem in terms of subpar state-action pairs . These lower bounds nearly match the upper bounds when the episode length of the MDP is a constant . Together , the upper and lower bounds characterize the intrinsic complexity of the �-MPERL problem . 2 Preliminaries . Throughout this paper , we denote by [ n ] : = { 1 , . . . , n } . For a set A in a universe U , we use AC = U \ A to denote its complement . Denote by Δ ( X ) the set of probability distributions over X . For functions f , g , we use f � g or f = O ( g ) ( resp . f � g or f = Ω ( g ) ) to denote that there exists some constant c > 0 , such that f ≤ cg ( resp . f ≥ cg ) , and use f � g to denote f � g and f � g simultaneously . Define a ∨ b : = max ( a , b ) , and a ∧ b : = min ( a , b ) . We use E to denote the expectation operator , and use var to denote the variance operator . Throughout , we use Õ ( · ) and Ω̃ ( · ) notation to hide polylogarithmic factors . Multi-task RL in episodic MDPs . We have a set of M MDPs � Mp = ( H , S , A , p0 , Pp , rp ) �M p=1 , each associated with a player p ∈ [ M ] . Each MDP Mp is regarded as a task . The MDPs share the same episode length H ∈ N+ , finite state space S , finite action space A , and initial state distribution p0 ∈ Δ ( S ) . Let ⊥ be a default terminal state that is not contained in S . The transition probabilities Pp : S × A → Δ ( S ∪ { ⊥ } ) and reward distributions rp : S × A → Δ ( [ 0 , 1 ] ) of the players are not necessarily identical . We assume that the MDPs are layered1 , in that the state space S can be partitioned into disjoint subsets ( Sh ) Hh=1 , where p0 is supported on S1 , and for every p ∈ [ M ] , h ∈ [ H ] , and every s ∈ Sh , a ∈ A , Pp ( · | s , a ) is supported on Sh+1 ; here , we define SH+1 = { ⊥ } . We denote by S : = |S| the size of the state space , and A : = |A| the size of the action space . Interaction process . The interaction process between the players and the environment is as follows : at the beginning , both ( rp ) Mp=1 and ( Pp ) Mp=1 are unknown to the players . For each episode k ∈ [ K ] , conditioned on the interaction history up to episode k − 1 , each player p ∈ [ M ] independently interacts with its respective MDP Mp ; specifically , player p starts with state sk1 , p ∼ p0 , and at every step ( layer ) h ∈ [ H ] , it chooses action akh , p , transitions to next state skh+1 , p ∼ Pp ( · | skh , p , akh , p ) and receives a stochastic immediate reward rkh , p ∼ rp ( · | skh , p , akh , p ) ; after all players have finished their k-th episode , they can communicate and share information . The goal of the players is to maximize their expected collective reward E ��K k=1 �M p=1 �H h=1 r k h , p � . Policy and value functions . A deterministic , history-independent policy π is a mapping from S to A , which can be used by a player to make decisions in its respective MDP . For player p and step h , we 1This is a standard assumption ( see , e.g. , [ 44 ] ) . It is worth noting that any episodic MDP ( with possibly nonstationary transition and reward ) can be converted to a layered MDP with stationary transition and reward , with the state space size being H times the size of the original state space . use V πh , p : Sh → [ 0 , H ] and Qπh , p : Sh ×A → [ 0 , H ] to denote its respective value and action-value functions , respectively . They satisfy the following recurrence known as the Bellman equation : ∀h ∈ [ H ] : V πh , p ( s ) = Qπh , p ( s , π ( s ) ) , Qπh , p ( s , a ) = Rp ( s , a ) + ( PpV πh+1 , p ) ( s , a ) , where we use the convention that V πH+1 , p ( ⊥ ) = 0 , and for f : Sh+1 → R , ( Ppf ) ( s , a ) : =� s�∈Sh+1 Pp ( s � | s , a ) f ( s� ) , and Rp ( s , a ) : = Er̂∼rp ( ·|s , a ) [ r̂ ] is the expected immediate reward of player p. For player p and policy π , denote by V π0 , p = Es1∼p0 � V π1 , p ( s1 ) � its expected reward . For player p , we also define its optimal value function V �h , p : Sh → [ 0 , H ] and the optimal actionvalue function Q�h , p : Sh ×A → [ 0 , H ] using the Bellman optimality equation : ∀h ∈ [ H ] : V �h , p ( s ) = max a∈A Q�h , p ( s , a ) , Q � h , p ( s , a ) = Rp ( s , a ) + ( PpV �h+1 , p ) ( s , a ) , ( 1 ) where we again use the convention that V �H+1 , p ( ⊥ ) = 0 . For player p , denote by V �0 , p = Es1∼p0 � V �1 , p ( s1 ) � its optimal expected reward . Given a policy π , as V πh , p for different h ’ s are only defined in the respective layer Sh , we “ collate ” the value functions ( V πh , p ) H h=1 and obtain a single value function V π p : S ∪ { ⊥ } → R. Formally , for every h ∈ [ H + 1 ] and s ∈ Sh , V πp ( s ) : = V π h , p ( s ) . We define Qπp , V � p , Q � p similarly . For player p , given its optimal action value function Q � p , any of its greedy policies π�p ( s ) ∈ argmaxa∈A Q�p ( s , a ) is optimal with respect to Mp . Suboptimality gap . For player p , we define the suboptimality gap of state-action pair ( s , a ) as gapp ( s , a ) = V � p ( s ) −Q�p ( s , a ) . We define the mininum suboptimality gap of player p as gapp , min = min ( s , a ) : gapp ( s , a ) > 0 gapp ( s , a ) , and the minimum suboptimality gap over all players as gapmin = minp∈ [ M ] gapp , min . For player p ∈ [ M ] , define Zp , opt : = � ( s , a ) : gapp ( s , a ) = 0 � as the set of optimal state-action pairs with respect to p. Performance metric . We measure the performance of the players using their collective regret , i.e. , over a total of K episodes , how much extra reward they would have collected in expectation if they were executing their respective optimal policies from the beginning . Formally , suppose for each episode k , player p executes policy πk ( p ) , then the collective regret of the players is defined as : Reg ( K ) = M� p=1 K� k=1 � V �0 , p − V π k ( p ) 0 , p � . Baseline : individual STRONG-EULER . A naive baseline for multi-task RL is to let each player run a separate RL algorithm without communication . For concreteness , we choose to let each player run the state of the art STRONG-EULER algorithm [ 36 ] ( see also its precursor EULER [ 46 ] ) , which enjoys minimax gap-independent [ 3 , 8 ] and gap-dependent regret guarantees , and refer to this strategy as individual STRONG-EULER . Specifically , as it is known that STRONG-EULER has a regret of Õ ( √ H2SAK + H4S2A ) , individual STRONG-EULER has a collective regret of Õ ( M √ H2SAK +MH4S2A ) . In addition , by a union bound and summing up the gap-dependent regret guarantees of STRONG-EULER for the M MDPs altogether , it can be checked that with probability 1− δ , individual STRONG-EULER has a collective regret of order2 ln � MSAK δ � � p∈ [ M ] � ( s , a ) ∈Zp , opt H3 gapp , min + � ( s , a ) ∈ZCp , opt H3 gapp ( s , a ) +MH4S2A ln SA gapmin . ( 2 ) 2The originally-stated gap-dependent regret bound of STRONG-EULER ( [ 36 ] , Corollary 2.1 ) uses a slightly different notion of suboptimality gap , which takes an extra minimum over all steps . A close examination of their proof shows that STRONG-EULER has regret bound ( 2 ) in layered MDPs . See also Remark 21 in Appendix C.4 . Our goal is to design multi-task RL algorithms that can achieve collective regret strictly lower than this baseline in both gap-dependent and gap-independent fashions when the tasks are similar . Notion of similarity . Throughout this paper , we will consider the following notion of similarity between MDPs in the multi-task episodic RL setting . Definition 1 . A collection of MDPs ( Mp ) Mp=1 is said to be �-dissimilar , if for all p , q ∈ [ M ] , and ( s , a ) ∈ S ×A , ��Rp ( s , a ) −Rq ( s , a ) �� ≤ � , �Pp ( · | s , a ) − Pq ( · | s , a ) �1 ≤ � H . If this happens , we call ( Mp ) Mp=1 an �-Multi-Player Episodic Reinforcement Learning ( abbrev . �-MPERL ) problem instance . If the MDPs in ( Mp ) Mp=1 are 0-dissimilar , then they are identical by definition , and our interaction protocol degenerates to the concurrent RL protocol [ 34 ] . Our dissimilarity notion is complementary to those of [ 6 , 16 ] : they require the MDPs to be either identical , or have well-separated parameters for at least one state-action pair ; in contrast , our dissimilarity notion allows the MDPs to be nonidentical and arbitrarily close . We have the following intuitive lemma that shows the closeness of optimal value functions of different MDPs , in terms of the dissimilarity parameter � : Lemma 2 . If ( Mp ) Mp=1 are �-dissimilar , then for every p , q ∈ [ M ] , and ( s , a ) ∈ S × A , ���Q�p ( s , a ) −Q�q ( s , a ) ��� ≤ 2H� ; consequently , ���gapp ( s , a ) − gapq ( s , a ) ��� ≤ 4H� .

Summary: This paper is about multi-task learning in episodic tabular MDPs. M learner's interact with M tabular MDPs with a shared state/action space but (potentially) different transition and reward functions. The authors assume that the mean rewards are epsilon close over all state/action pairs and that the transition kernels are epsilon/H close in L1 over all state/action pairs where H is the episode length. The learner's have unlimited communication at the end of every episode.  The main contribution is a new algorithm and analysis showing that the information sharing can help when epsilon is small and the number of state/action pairs with a large V^*(s) - Q*(s,a) gap is large. The basic idea is to maintain empirical estimates of transitions and rewards for each learner individually and also an aggregate of all estimates. Of course the latter is somehow an estimate of some average reward/transitions, so the algorithm adds to the corresponding confidence bounds a term that depends on epsilon. Besides the modified bonus terms, the learner uses essentially an instance of the Euler algorithm for each MDP. That's quite a natural idea, and the resulting bounds make sense.   The authors also provide lower bounds. These are of a minimax-ish flavour and show that there /exist/ MDPs where the upper bounds are tight. I find this to be only quite weak evidence that the upper bounds are expressed using the "right" quantities, if such things even exist.