Provably efficient multi-task reinforcement learning with model transfer

1 Introduction . In many real-world applications , reinforcement learning ( RL ) agents can be deployed as a group to complete similar tasks at the same time . For example , in healthcare robotics , robots are paired with people with dementia to perform personalized cognitive training activities by learning their preferences [ 42 , 21 ] ; in autonomous driving , a set of autonomous vehicles learn how to navigate and avoid obstacles in various environments [ 27 ] . In these settings , each learning agent alone may only be able to acquire a limited amount of data , while the agents as a group have the potential to collectively learn faster through sharing knowledge among themselves . Multi-task learning [ 7 ] is a practical framework that can be used to model such settings , where a set of learning agents share/transfer knowledge to improve their collective performance . Despite many empirical successes of multi-task RL ( see , e.g. , [ 51 , 28 , 27 ] ) and transfer learning for RL ( see , e.g. , [ 26 , 39 ] ) , a theoretical understanding of when and how information sharing or knowledge transfer can provide benefits remains limited . Exceptions include [ 16 , 6 , 11 , 17 , 32 , 25 ] , which study multi-task learning from parameter or representation transfer perspectives . However , these works still do not provide a completely satisfying answer : for example , in many application scenarios , the reward structures and the environment dynamics are only slightly different for each task—this is , however , not captured by representation transfer [ 11 , 17 ] or existing works on clustering-based parameter transfer [ 16 , 6 ] . In such settings , is it possible to design provably efficient multi-task RL algorithms that have guarantees never worse than agents learning individually , while outperforming the individual agents in favorable situations ? In this work , we formulate an online multi-task RL problem that is applicable to the aforementioned settings . Specifically , inspired by a recent study on multi-task multi-armed bandits [ 43 ] , we formulate the �-Multi-Player Episodic Reinforcement Learning ( abbreviated as �-MPERL ) problem , in which all tasks share the same state and action spaces , and the tasks are assumed to be similar—i.e. , the dissimilarities between the environments of different tasks ( specifically , the reward distributions and transition dynamics associated with the players/tasks ) are bounded in terms of a dissimilarity parameter � ≥ 0 . This problem not only models concurrent RL [ 34 , 16 ] as a special case by taking � = 0 , but also captures richer multi-task RL settings when � is nonzero . We study regret minimization for the �-MPERL problem , specifically : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 . We identify a problem complexity notion named subpar state-action pairs , which captures the amenability to information sharing among tasks in �-MPERL problem instances . As shown in the multi-task bandits literature ( e.g. , [ 43 ] ) , inter-task information sharing is not always helpful to reduce the players ’ collective regret . Subpar state-action pairs , intuitively speaking , are clearly suboptimal for all tasks , for which we can robustly take advantage of ( possibly biased ) data collected for other tasks to achieve a lower regret in a certain task . 2 . In the setting where the dissimilarity parameter � is known , we design a model-based algorithm MULTI-TASK-EULER ( Algorithm 1 ) , which is built upon state-of-the-art algorithms for learning single-task Markov decision processes ( MDPs ) [ 3 , 46 , 36 ] , as well as algorithmic ideas of model transfer in RL [ 39 ] . MULTI-TASK-EULER crucially utilizes the dissimilarity assumption to robustly take advantage of information sharing among tasks , and achieves regret upper bounds in terms of subpar state-action pairs , in both ( value function suboptimality ) gap-dependent and gap-independent fashions . Specifically , compared with a baseline algorithm that does not utilize information sharing , MULTI-TASK-EULER has a regret guarantee that : ( 1 ) is never worse , i.e. , it avoids negative transfer [ 33 ] ; ( 2 ) can be much superior when there are a large number of subpar state-action pairs . 3 . We also present gap-dependent and gap-independent regret lower bounds for the �-MPERL problem in terms of subpar state-action pairs . These lower bounds nearly match the upper bounds when the episode length of the MDP is a constant . Together , the upper and lower bounds characterize the intrinsic complexity of the �-MPERL problem . 2 Preliminaries . Throughout this paper , we denote by [ n ] : = { 1 , . . . , n } . For a set A in a universe U , we use AC = U \ A to denote its complement . Denote by Δ ( X ) the set of probability distributions over X . For functions f , g , we use f � g or f = O ( g ) ( resp . f � g or f = Ω ( g ) ) to denote that there exists some constant c > 0 , such that f ≤ cg ( resp . f ≥ cg ) , and use f � g to denote f � g and f � g simultaneously . Define a ∨ b : = max ( a , b ) , and a ∧ b : = min ( a , b ) . We use E to denote the expectation operator , and use var to denote the variance operator . Throughout , we use Õ ( · ) and Ω̃ ( · ) notation to hide polylogarithmic factors . Multi-task RL in episodic MDPs . We have a set of M MDPs � Mp = ( H , S , A , p0 , Pp , rp ) �M p=1 , each associated with a player p ∈ [ M ] . Each MDP Mp is regarded as a task . The MDPs share the same episode length H ∈ N+ , finite state space S , finite action space A , and initial state distribution p0 ∈ Δ ( S ) . Let ⊥ be a default terminal state that is not contained in S . The transition probabilities Pp : S × A → Δ ( S ∪ { ⊥ } ) and reward distributions rp : S × A → Δ ( [ 0 , 1 ] ) of the players are not necessarily identical . We assume that the MDPs are layered1 , in that the state space S can be partitioned into disjoint subsets ( Sh ) Hh=1 , where p0 is supported on S1 , and for every p ∈ [ M ] , h ∈ [ H ] , and every s ∈ Sh , a ∈ A , Pp ( · | s , a ) is supported on Sh+1 ; here , we define SH+1 = { ⊥ } . We denote by S : = |S| the size of the state space , and A : = |A| the size of the action space . Interaction process . The interaction process between the players and the environment is as follows : at the beginning , both ( rp ) Mp=1 and ( Pp ) Mp=1 are unknown to the players . For each episode k ∈ [ K ] , conditioned on the interaction history up to episode k − 1 , each player p ∈ [ M ] independently interacts with its respective MDP Mp ; specifically , player p starts with state sk1 , p ∼ p0 , and at every step ( layer ) h ∈ [ H ] , it chooses action akh , p , transitions to next state skh+1 , p ∼ Pp ( · | skh , p , akh , p ) and receives a stochastic immediate reward rkh , p ∼ rp ( · | skh , p , akh , p ) ; after all players have finished their k-th episode , they can communicate and share information . The goal of the players is to maximize their expected collective reward E ��K k=1 �M p=1 �H h=1 r k h , p � . Policy and value functions . A deterministic , history-independent policy π is a mapping from S to A , which can be used by a player to make decisions in its respective MDP . For player p and step h , we 1This is a standard assumption ( see , e.g. , [ 44 ] ) . It is worth noting that any episodic MDP ( with possibly nonstationary transition and reward ) can be converted to a layered MDP with stationary transition and reward , with the state space size being H times the size of the original state space . use V πh , p : Sh → [ 0 , H ] and Qπh , p : Sh ×A → [ 0 , H ] to denote its respective value and action-value functions , respectively . They satisfy the following recurrence known as the Bellman equation : ∀h ∈ [ H ] : V πh , p ( s ) = Qπh , p ( s , π ( s ) ) , Qπh , p ( s , a ) = Rp ( s , a ) + ( PpV πh+1 , p ) ( s , a ) , where we use the convention that V πH+1 , p ( ⊥ ) = 0 , and for f : Sh+1 → R , ( Ppf ) ( s , a ) : =� s�∈Sh+1 Pp ( s � | s , a ) f ( s� ) , and Rp ( s , a ) : = Er̂∼rp ( ·|s , a ) [ r̂ ] is the expected immediate reward of player p. For player p and policy π , denote by V π0 , p = Es1∼p0 � V π1 , p ( s1 ) � its expected reward . For player p , we also define its optimal value function V �h , p : Sh → [ 0 , H ] and the optimal actionvalue function Q�h , p : Sh ×A → [ 0 , H ] using the Bellman optimality equation : ∀h ∈ [ H ] : V �h , p ( s ) = max a∈A Q�h , p ( s , a ) , Q � h , p ( s , a ) = Rp ( s , a ) + ( PpV �h+1 , p ) ( s , a ) , ( 1 ) where we again use the convention that V �H+1 , p ( ⊥ ) = 0 . For player p , denote by V �0 , p = Es1∼p0 � V �1 , p ( s1 ) � its optimal expected reward . Given a policy π , as V πh , p for different h ’ s are only defined in the respective layer Sh , we “ collate ” the value functions ( V πh , p ) H h=1 and obtain a single value function V π p : S ∪ { ⊥ } → R. Formally , for every h ∈ [ H + 1 ] and s ∈ Sh , V πp ( s ) : = V π h , p ( s ) . We define Qπp , V � p , Q � p similarly . For player p , given its optimal action value function Q � p , any of its greedy policies π�p ( s ) ∈ argmaxa∈A Q�p ( s , a ) is optimal with respect to Mp . Suboptimality gap . For player p , we define the suboptimality gap of state-action pair ( s , a ) as gapp ( s , a ) = V � p ( s ) −Q�p ( s , a ) . We define the mininum suboptimality gap of player p as gapp , min = min ( s , a ) : gapp ( s , a ) > 0 gapp ( s , a ) , and the minimum suboptimality gap over all players as gapmin = minp∈ [ M ] gapp , min . For player p ∈ [ M ] , define Zp , opt : = � ( s , a ) : gapp ( s , a ) = 0 � as the set of optimal state-action pairs with respect to p. Performance metric . We measure the performance of the players using their collective regret , i.e. , over a total of K episodes , how much extra reward they would have collected in expectation if they were executing their respective optimal policies from the beginning . Formally , suppose for each episode k , player p executes policy πk ( p ) , then the collective regret of the players is defined as : Reg ( K ) = M� p=1 K� k=1 � V �0 , p − V π k ( p ) 0 , p � . Baseline : individual STRONG-EULER . A naive baseline for multi-task RL is to let each player run a separate RL algorithm without communication . For concreteness , we choose to let each player run the state of the art STRONG-EULER algorithm [ 36 ] ( see also its precursor EULER [ 46 ] ) , which enjoys minimax gap-independent [ 3 , 8 ] and gap-dependent regret guarantees , and refer to this strategy as individual STRONG-EULER . Specifically , as it is known that STRONG-EULER has a regret of Õ ( √ H2SAK + H4S2A ) , individual STRONG-EULER has a collective regret of Õ ( M √ H2SAK +MH4S2A ) . In addition , by a union bound and summing up the gap-dependent regret guarantees of STRONG-EULER for the M MDPs altogether , it can be checked that with probability 1− δ , individual STRONG-EULER has a collective regret of order2 ln � MSAK δ � � p∈ [ M ] � ( s , a ) ∈Zp , opt H3 gapp , min + � ( s , a ) ∈ZCp , opt H3 gapp ( s , a ) +MH4S2A ln SA gapmin . ( 2 ) 2The originally-stated gap-dependent regret bound of STRONG-EULER ( [ 36 ] , Corollary 2.1 ) uses a slightly different notion of suboptimality gap , which takes an extra minimum over all steps . A close examination of their proof shows that STRONG-EULER has regret bound ( 2 ) in layered MDPs . See also Remark 21 in Appendix C.4 . Our goal is to design multi-task RL algorithms that can achieve collective regret strictly lower than this baseline in both gap-dependent and gap-independent fashions when the tasks are similar . Notion of similarity . Throughout this paper , we will consider the following notion of similarity between MDPs in the multi-task episodic RL setting . Definition 1 . A collection of MDPs ( Mp ) Mp=1 is said to be �-dissimilar , if for all p , q ∈ [ M ] , and ( s , a ) ∈ S ×A , ��Rp ( s , a ) −Rq ( s , a ) �� ≤ � , �Pp ( · | s , a ) − Pq ( · | s , a ) �1 ≤ � H . If this happens , we call ( Mp ) Mp=1 an �-Multi-Player Episodic Reinforcement Learning ( abbrev . �-MPERL ) problem instance . If the MDPs in ( Mp ) Mp=1 are 0-dissimilar , then they are identical by definition , and our interaction protocol degenerates to the concurrent RL protocol [ 34 ] . Our dissimilarity notion is complementary to those of [ 6 , 16 ] : they require the MDPs to be either identical , or have well-separated parameters for at least one state-action pair ; in contrast , our dissimilarity notion allows the MDPs to be nonidentical and arbitrarily close . We have the following intuitive lemma that shows the closeness of optimal value functions of different MDPs , in terms of the dissimilarity parameter � : Lemma 2 . If ( Mp ) Mp=1 are �-dissimilar , then for every p , q ∈ [ M ] , and ( s , a ) ∈ S × A , ���Q�p ( s , a ) −Q�q ( s , a ) ��� ≤ 2H� ; consequently , ���gapp ( s , a ) − gapq ( s , a ) ��� ≤ 4H� .

Summary: The submission investigates the problem of multi-task reinforcement learning, which is where a collection of agents simultaneous learn similar tasks, ideally while exploiting the similarities in the task structure. The submission (i) proposes a formalism for this setting and (ii) designs and analyzes a model-based algorithm (Multi-task Euler) for the aforementioned formalism. The main engine Multi-task Euler is the exploitation of "subpar state-action pairs" --- i.e., state-action pairs which are suboptimal across all tasks. When there are many subpar state-action pairs, Multi-task Euler strictly outperforms Euler without information sharing. In any case, Multi-task Euler is never outperformed by Euler without information sharing.