Sharing Knowledge in Multi-Task Deep Reinforcement Learning

1 INTRODUCTION . Multi-Task Learning ( MTL ) ambitiously aims to learn multiple tasks jointly instead of learning them separately , leveraging the assumption that the considered tasks have common properties which can be exploited by Machine Learning ( ML ) models to generalize the learning of each of them . For instance , the features extracted in the hidden layers of a neural network trained on multiple tasks have the advantage of being a general representation of structures common to each other . This translates into an effective way of learning multiple tasks at the same time , but it can also improve the learning of each individual task compared to learning them separately ( Caruana , 1997 ) . Furthermore , the learned representation can be used to perform Transfer Learning ( TL ) , i.e . using it as a preliminary knowledge to learn a new similar task resulting in a more effective and faster learning than learning the new task from scratch ( Baxter , 2000 ; Thrun & Pratt , 2012 ) . The same benefits of extraction and exploitation of common features among the tasks achieved in MTL , can be obtained in Multi-Task Reinforcement Learning ( MTRL ) when training a single agent on multiple Reinforcement Learning ( RL ) problems with common structures ( Taylor & Stone , 2009 ; Lazaric , 2012 ) . In particular , in MTRL an agent can be trained on multiple tasks in the same domain , e.g . riding a bicycle or cycling while going towards a goal , or on different but similar domains , e.g . balancing a pendulum or balancing a double pendulum1 . Considering recent advances in Deep Reinforcement Learning ( DRL ) and the resulting increase in the complexity of experimental benchmarks , the use of Deep Learning ( DL ) models , e.g . deep neural networks , has become a popular and effective way to extract common features among tasks in MTRL algorithms ( Rusu et al. , 2015 ; Liu et al. , 2016 ; Higgins et al. , 2017 ) . However , despite the high representational capacity of DL models , the extraction of good features remains challenging . For instance , the performance of the learning process can degrade when unrelated tasks are used together ( Caruana , 1997 ; Baxter , 2000 ) ; another detrimental issue may occur when the training of a single model is not balanced properly among multiple tasks ( Hessel et al. , 2018 ) . Recent developments in MTRL achieve significant results in feature extraction by means of algorithms specifically developed to address these issues . While some of these works rely on a single deep neural network to model the multi-task agent ( Liu et al. , 2016 ; Yang et al. , 2017 ; Hessel et al. , 2018 ; Wulfmeier et al. , 2019 ) , others use multiple deep neural networks , e.g . one for each task and another for the multi-task agent ( Rusu et al. , 2015 ; Parisotto et al. , 2015 ; Higgins et al. , 2017 ; Teh et al. , 2017 ) . Intuitively , achieving good results in MTRL with a single deep neural network is more desirable than using many of them , since the training time is likely much less and the whole architecture is easier to implement . In this paper we study the benefits of shared representations among tasks . We theoretically motivate the intuitive effectiveness of our method , deriving theoretical guarantees that exploit the theoretical framework provided by Maurer et al . ( 2016 ) , in which the authors present upper bounds on the quality of learning in MTL when extracting features for multiple tasks in a single shared representation . The significancy of this result is that the cost of learning the shared representation decreases with a factor O ( 1/√T ) , where T is the number of tasks for many function approximator hypothesis classes . The main contribution of this work is twofold . 1 . We derive upper confidence bounds for Approximate Value-Iteration ( AVI ) and Approximate Policy-Iteration ( API ) 2 ( Farahmand , 2011 ) in the MTRL setting , and we extend the approximation error bounds in Maurer et al . ( 2016 ) to the case of multiple tasks with different dimensionalities . Then , we show how to combine these results resulting in , to the best of our knowledge , the first proposed extension of the finite-time bounds of AVI/API to MTRL . Despite being an extension of previous works , we derive these results to justify our approach showing how the error propagation in AVI/API can theoretically benefit from learning multiple tasks jointly . 2 . We leverage these results proposing a neural network architecture , for which these bounds hold with minor assumptions , that allow us to learn multiple tasks with a single regressor extracting a common representation . We show an empirical evidence of the consequence of our bounds by means of a variant of FittedQ-Iteration ( FQI ) ( Ernst et al. , 2005 ) , based on our shared network and for which our bounds apply , that we call Multi FittedQ-Iteration ( MFQI ) . Then , we perform an empirical evaluation in challenging RL problems proposing multitask variants of the Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) and Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) algorithms . These algorithms are practical implementations of the more general AVI/API framework , designed to solve complex problems . In this case , the bounds apply to these algorithms only with some assumptions , e.g . stationary sampling distribution . The outcome of the empirical analysis joins the theoretical results , showing significant performance improvements compared to the singletask version of the algorithms in various RL problems , including several MuJoCo ( Todorov et al. , 2012 ) domains . 2 PRELIMINARIES . Let B ( X ) be the space of bounded measurable functions w.r.t . the σ-algebra σX , and similarly B ( X , L ) be the same bounded by L < ∞ . A Markov Decision Process ( MDP ) is defined as a 5-tupleM = < S , A , P , R , γ > , where S is the state space , A is the action space , P : S × A → S is the transition distribution where P ( s′|s , a ) 1For simplicity , in this paper we refer to the concepts of task and domain interchangeably . 2All proofs and the theorem for API are in Appendix A.2 . is the probability of reaching state s′ when performing action a in state s , R : S × A × S → R is the reward function , and γ ∈ ( 0 , 1 ] is the discount factor . A deterministic policy π maps , for each state , the action to perform : π : S → A . Given a policy π , the value of an action a in a state s represents the expected discounted cumulative reward obtained by performing a in s and following π thereafter : Qπ ( s , a ) , E [ ∑∞ k=0 γ kri+k+1|si = s , ai = a , π ] , where ri+1 is the reward obtained after the i-th transition . The expected discounted cumulative reward is maximized by following the optimal policy π∗ which is the one that determines the optimal action values , i.e. , the ones that satisfy the Bellman optimality equation ( Bellman , 1954 ) : Q∗ ( s , a ) , ∫ S P ( s ′|s , a ) [ R ( s , a , s′ ) + γmaxa′ Q∗ ( s′ , a′ ) ] ds′ . The solution of the Bellman optimality equation is the fixed point of the optimal Bellman operator T ∗ : B ( S × A ) → B ( S × A ) defined as ( T ∗Q ) ( s , a ) , ∫ S P ( s ′|s , a ) [ R ( s , a , s′ ) + γmaxa′ Q ( s′ , a′ ) ] ds′ . In the MTRL setting , there are multiple MDPsM ( t ) = < S ( t ) , A ( t ) , P ( t ) , R ( t ) , γ ( t ) > where t ∈ { 1 , . . . , T } and T is the number of MDPs . For each MDPM ( t ) , a deterministic policy πt : S ( t ) → A ( t ) induces an action-value function Qπtt ( s ( t ) , a ( t ) ) = E [ ∑∞ k=0 γ kr ( t ) i+k+1|si = s ( t ) , ai = a ( t ) , πt ] . In this setting , the goal is to maximize the sum of the expected cumulative discounted reward of each task . In our theoretical analysis of the MTRL problem , the complexity of representation plays a central role . As done in Maurer et al . ( 2016 ) , we consider the Gaussian complexity , a variant of the well-known Rademacher complexity , to measure the complexity of the representation . Given a set X̄ ∈ X Tn of n input samples for each task t ∈ { 1 , . . . , T } , and a classH composed of k ∈ { 1 , . . . , K } functions , the Gaussian complexity of a random set H ( X̄ ) = { ( hk ( Xti ) ) : h ∈ H } ⊆ RKTn is defined as follows : G ( H ( X̄ ) ) = E [ sup h∈H ∑ tki γtkihk ( Xti ) ∣∣∣∣∣Xti ] , ( 1 ) where γtki are independent standard normal variables . We also need to define the following quantity , taken from Maurer ( 2016 ) : let γ be a vector of m random standard normal variables , and f ∈ F : Y → Rm , with Y ⊆ Rn , we define O ( F ) = sup y , y′∈Y , y 6=y′ E [ sup f∈F 〈γ , f ( y ) − f ( y′ ) 〉 ‖y − y′‖ ] . ( 2 ) Equation 2 can be viewed as a Gaussian average of Lipschitz quotients , and appears in the bounds provided in this work . Finally , we define L ( F ) as the upper bound of the Lipschitz constant of all the functions f in the function class F . 3 THEORETICAL ANALYSIS . The following theoretical study starts from the derivation of theoretical guarantees for MTRL in the AVI framework , extending the results of Farahmand ( 2011 ) in the MTRL scenario . Then , to bound the approximation error term in the AVI bound , we extend the result described in Maurer ( 2006 ) to MTRL . As we discuss , the resulting bounds described in this section clearly show the benefit of sharing representation in MTRL . To the best of our knowledge , this is the first general result for MTRL ; previous works have focused on finite MDPs ( Brunskill & Li , 2013 ) or linear models ( Lazaric & Restelli , 2011 ) . 3.1 MULTI-TASK REPRESENTATION LEARNING . The multi-task representation learning problem consists in learning simultaneously a set of T tasks µt , modeled as probability measures over the space of the possible input-output pairs ( x , y ) , with x ∈ X and y ∈ R , being X the input space . Let w ∈ W : X → RJ , h ∈ H : RJ → RK and f ∈ F : RK → R be functions chosen from their respective hypothesis classes . The functions in the hypothesis classes must be Lipschitz continuous functions . Let Z̄ = ( Z1 , . . . , ZT ) be the multi-sample over the set of tasks µ = ( µ1 , . . . , µT ) , where Zt = ( Zt1 , . . . , Ztn ) ∼ µnt and Zti = ( Xti , Yti ) ∼ µt . We can formalize our regression problem as the following minimization problem : min { 1 nT T∑ t=1 N∑ i=1 ` ( ft ( h ( wt ( Xti ) ) ) , Yti ) : f ∈ FT , h ∈ H , w ∈ WT } , ( 3 ) where we use f = ( f1 , . . . , fT ) , w = ( w1 , . . . , wT ) , and define the minimizers of Equation ( 3 ) as ŵ , ĥ , and f̂ . We assume that the loss function ` : R× R→ [ 0 , 1 ] is 1-Lipschitz in the first argument for every value of the second argument . While this assumption may seem restrictive , the result obtained can be easily scaled to the general case . To use the principal result of this section , for a generic loss function ` ′ , it is possible to use ` ( · ) = ` ′ ( · ) / max , where max is the maximum value of ` ′ . The expected loss over the tasks , given w , h and f is the task-averaged risk : εavg ( w , h , f ) = 1 T T∑ t=1 E [ ` ( ft ( h ( wt ( X ) ) ) , Y ) ] ( 4 ) The minimum task-averaged risk , given the set of tasks µ and the hypothesis classesW , H and F is ε∗avg , and the corresponding minimizers are w ∗ , h∗ and f∗ .

Summary: This paper provides a theoretical justification for the benefit of multi-task deep RL (MTRL) with shared representations. By extending prior work (Farahmand (2011) and Maurer et al. (2016)), the authors demonstrate that the bound of MTRL can be improved if the cost of learning the shared representation at each AVI iteration can be reduced, which is mitigated as we increase the number of tasks. The author also empirically verify their theoretical results in a tabular Q-Fitted Iteration domain and also in challenging RL domains such as Mujoco. The results show that MTRL with shared representation can outperform their single task counterparts to some degree.