MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

1 INTRODUCTION . We envision a future scenario where a variety of robotic systems , which are each trained or manually engineered to solve a similar task , provide their policies for a new robot to learn a relevant task quickly . For example , imagine various pick-and-place robots working in factories all over the world . Depending on the manufacturer , these robots will differ in their kinematics ( e.g. , link length , joint orientations ) and dynamics ( e.g. , link mass , joint damping , friction , inertia ) . They could provide their policies to a new robot ( Devin et al. , 2017 ) , even though their dynamics factors , on which the policies are implicitly conditioned , are not typically available ( Chen et al. , 2018 ) . Moreover , we can not rely on a history of their individual experiences , as they may be unavailable due to a lack of communication between factories or prohibitively large dataset sizes . In such scenarios , we argue that a key technique to develop is the ability to transfer knowledge from a collection of robots to a new robot quickly only by exploiting their policies while being agnostic to their different kinematics and dynamics , rather than collecting a vast amount of samples to train the new robot from scratch . The scenario illustrated above poses a new challenge in the transfer learning for reinforcement learning ( RL ) domains . Formally , consider multiple instances of a single environment that differ in their state transition dynamics , e.g. , independent ant robots with different leg designs in Figure 1 , which reach different locations by executing the same walking actions . These source agents interacting with one of the environment instances provide their deterministic policy to a new target agent in another environment instance . Then , our problem is : can we efficiently learn the policy of a target agent given only the collection of source policies ? Note that information about source environmental dynamics , such as the exact state transition distribu- tions and the history of environmental states , will not be visible to the target agent as mentioned above . Also , the source policies are neither trained nor hand-engineered for the target environment instance , and therefore not guaranteed to work optimally and may even fail ( Chen et al. , 2018 ) . These conditions prevent us from adopting existing work on transfer RL between different environmental dynamics , as they require access to source environment instances or their dynamics for training a target policy ( e.g. , Lazaric et al . ( 2008 ) ; Chen et al . ( 2018 ) ; Yu et al . ( 2019 ) ; Tirinzoni et al . ( 2018 ) ) . Similarly , meta-learning approaches ( Vanschoren , 2018 ; Sæmundsson et al. , 2018 ; Clavera et al. , 2019 ) can not be used here because they typically train an agent on a diverse set of tasks ( i.e. , environment instances ) . Also , existing techniques that utilize a collection of source policies , e.g. , policy reuse frameworks ( Fernández & Veloso , 2006 ; Rosman et al. , 2016 ; Zheng et al. , 2018 ) and option frameworks ( Sutton et al. , 1999 ; Bacon et al. , 2017 ; Mankowitz et al. , 2018 ) , are not a promising solution because , to our knowledge , they assume source policies have the same environmental dynamics but have different goals . As a solution to the problem , we propose a new transfer RL approach named MULTI-source POLicy AggRegation ( MULTIPOLAR ) . As shown in Figure 2 , our key idea is twofold ; 1 ) In a target policy , we adaptively aggregate the deterministic actions produced by a collection of source policies . By learning aggregation parameters to maximize the expected return at a target environment instance , we can better adapt the aggregated actions to unseen environmental dynamics of the target instance without knowing source environmental dynamics nor source policy performances . 2 ) We also train an auxiliary network that predicts a residual around the aggregated actions , which is crucial for ensuring the expressiveness of the target policy even when some source policies are not useful . As another notable advantage , the proposed MULTIPOLAR can be used for both continuous and discrete action spaces with few modifications while allowing a target policy to be trained in a principled fashion . Similar to Ammar et al . ( 2014 ) ; Song et al . ( 2016 ) ; Chen et al . ( 2018 ) ; Tirinzoni et al . ( 2018 ) ; Yu et al . ( 2019 ) , our method assumes that the environment structure ( state/action space ) is identical between the source and target environments , while dynamics/kinematics parameters are different . This assumption holds in many real-world applications such as in sim-to-real tasks ( Tan et al. , 2018 ) , industrial insertion tasks ( Schoettler et al. , 2019 ) ( different dynamics comes from the differences in parts ) , and wearable robots ( Zhang et al. , 2017 ) ( with users as dynamics ) . We evaluate MULTIPOLAR in a variety of environments ranging from classic control problems to challenging robotics simulations . Our experimental results demonstrate the significant improvement of sample efficiency with the proposed approach , compared to baselines that trained a target policy from scratch or from a single source policy . We also conducted a detailed analysis of our approach and found it works well even when some of the source policies performed poorly in their original environment instance . Main contributions : ( 1 ) a new transfer RL problem that leverages multiple source policies collected under diverse environmental dynamics to train a target policy in another dynamics , and ( 2 ) MULTIPOLAR , a simple yet principled and effective solution verified in our extensive experiments . 2 PRELIMINARIES . Reinforcement Learning We formulate our problem under the standard RL framework ( Sutton & Barto , 1998 ) , where an agent interacts with its environment modeled by a Markov decision process ( MDP ) . An MDP is represented by the tupleM = ( ρ0 , γ , S , A , R , T ) where ρ0 is the initial state distribution and γ is a discount factor . At each timestep t , given the current state st ∈ S , the agent executes an action at ∈ A based on its policy π ( at | st ; θ ) that is parameterized by θ . The environment returns a reward R ( st , at ) ∈ R and transitions to the next state st+1 based on the state transition distribution T ( st+1 | st , at ) . In this framework , RL aims to maximize the expected return with respect to the policy parameters θ . Environment Instances In this work , we considerK instances of the same environment that differ only in their state transition dynamics . We model each environment instance by an indexed MDP : Mi = ( ρ0 , γ , S , A , R , Ti ) where no two state transition distributions Ti , Tj ; i 6= j are identical . We also assume that each Ti is unknown when training a target policy , i.e. , agents can not access the exact form of Ti nor a collection of states sampled from Ti . Source Policies For each of the K environment instances , we are given a deterministic source policy µi : S → A that only maps states to actions . Each source policy µi can be either parameterized ( e.g. , learned from interacting with the environment modeled byMi ) or non-parameterized ( e.g. , heuristically designed by humans ) . Either way , we assume no prior knowledge about µi is available for a target agent , such as their representations or original performances , except that they were acquired inMi with an unknown Ti . Problem Statement Given the set of source policies L = { µ1 , . . . , µK } , our goal is to train a new target agent ’ s policy πtarget ( at | st ; L , θ ) in a sample efficient fashion , where the target agent interacts with another environment instance Mtarget = ( ρ0 , S , A , R , Ttarget ) and Ttarget is not necessarily identical to Ti ( i = 1 . . . , K ) . 3 MULTI-SOURCE POLICY AGGREGATION . As shown in Figure 2 , with the Multi-Source Policy Aggregation ( MULTIPOLAR ) , we formulate a target policy πtarget using a ) the adaptive aggregation of deterministic actions from the set of source policies L , and b ) the auxiliary network predicting residuals around the aggregated actions . We first present our method for the continuous action space , and then extend it to the discrete space . Adaptive Aggregation of Source Policies Let us denote by a ( i ) t = µi ( st ) the action predicted deterministically by source policy µi given the current state st. For the continuous action space , a ( i ) t ∈ RD is a D-dimensional real-valued vector representing D actions performed jointly in each timestep . For the collection of source policies L , we derive the matrix of their deterministic actions : At = [ ( a ( 1 ) t ) > , . . . , ( a ( K ) t ) > ] ∈ RK×D . ( 1 ) The key idea of this work is to aggregateAt adaptively in an RL loop , i.e. , to maximize the expected return . This adaptive aggregation gives us a “ baseline ” action that could introduce a strong inductive bias in the training of a target policy , without knowing source environmental dynamics Ti . More specifically , we define the adaptive aggregation function Fagg : S → A that produces the baseline action based on the current state st as follows : Fagg ( st ; L , θagg ) = 1 K 1K ( θagg At ) , ( 2 ) where θagg ∈ RK×D is a matrix of trainable parameters , is the element-wise multiplication , and 1K is the all-ones vector of length K. θagg is neither normalized nor regularized , and can scale each action of each policy independently . This means that we do not merely adaptively interpolate action spaces , but more flexibly emphasize informative source actions while suppressing irrelevant ones . Predicting Residuals around Aggregated Actions Moreover , we learn auxiliary network Faux : S → A jointly with Fagg , to predict residuals around the aggregated actions . Faux is used to improve the target policy training in two ways . 1 ) If the aggregated actions from Fagg are already useful in the target environment instance , Faux will correct them for a higher expected return . 2 ) Otherwise , Faux learns the target task while leveraging Fagg as a prior to have a guided exploration process . Any network could be used for Faux as long as it is parameterized and fully differentiable . Finally , the MULTIPOLAR function is formulated as : F ( st ; L , θagg , θaux ) = Fagg ( st ; L , θagg ) + Faux ( st ; θaux ) , ( 3 ) where θaux denotes a set of trainable parameters for Faux . Note that the idea of predicting residuals for a source policy has also been presented by Silver et al . ( 2018 ) ; Johannink et al . ( 2019 ) ; Rana et al . ( 2019 ) . The main difference here is that , while these works just add raw action outputs provided from a single hand-engineered source policy , we adaptively aggregate actions from multiple source policies in order to obtain a more flexible and canonical representation . Target Policy Target policy πtarget can be modeled by reparameterizing the MULTIPOLAR function as a Gaussian distribution , i.e. , N ( F ( st ; L , θagg , θaux ) , Σ ) , where Σ is a covariance matrix estimated based on what the used RL algorithm requires . Since we regard µi ∈ L as fixed functions mapping states to actions , this Gaussian policy πtarget is differentiable with respect to θagg and θaux , and hence could be trained with any RL algorithm that explicitly updates policy parameters . Unlike Silver et al . ( 2018 ) ; Johannink et al . ( 2019 ) ; Rana et al . ( 2019 ) , we can formulate the target policy in a principled fashion for actions in a discrete space . Specifically , instead of aD-dimensional real-valued vector , here we have a D-dimensional one-hot vector a ( i ) t ∈ { 0 , 1 } D , ∑ j ( a ( i ) t ) j = 1 as outputs of µi , where ( a ( i ) t ) j = 1 indicates that the j-th action is to be executed . Following Eqs . ( 2 ) and ( 3 ) , the output of F ( st ; L , θagg , θaux ) can be viewed as D-dimensional un-normalized action scores , from which we can sample a discrete action after normalizing it by the softmax function .

Summary: In this paper authors propose a method for transfer reinforcement learning (RL). Specifically they are claiming that RL agents can transfer knowledge between each other about the environment dynamics. In order to showcase their approach they have come up with a new transfer RL task that makes use of some source policies trained under a diverse set of environment dynamics. Their key contributions to solve the task involve a decision aggregation framework that is able to build on top of relevant policies while suppressing irrelevant ones and an auxiliary network that predicts the residuals around the aggregated actions.