Transfer RL across Observation Feature Spaces via Model-Based Regularization

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has the potential to be used in many large-scale applications such as robotics , gaming and automotive . In these real-life scenarios , it is an essential ability for agents to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks , which is known as Transfer RL ( TRL ) . Most existing TRL works ( Taylor & Stone , 2009 ; Zhu et al. , 2020 ) focus on tasks with the same state-action space but different dynamics/reward . However , these approaches do not apply to the case where the observation space changes significantly . Observation change is common in practice as in the following scenarios . ( 1 ) Incremental environment development . RL is used to train non-player characters ( NPC ) in games ( Juliani et al. , 2018 ) , which may be frequently updated . When there are new scenes , characters , or obstacles added to the game , the agent ’ s observation space will change accordingly . ( 2 ) Hardware upgrade/replacement . For robots with sensory observations ( Bohez et al. , 2017 ) , the observation space could change ( e.g . from text to audio , from lidar to camera ) as the sensor changes . ( 3 ) Restricted data access . In some RL applications ( Ganesh et al. , 2019 ) , agent observation contains sensitive data ( e.g . inventory ) which may become unavailable in the future due to data restrictions . In these cases , the learner may have to discard the old policy and train a new policy from scratch , as the policy has a significantly different input space , even though the underlying dynamics are similar . But training an RL policy from scratch can be expensive and unstable . Therefore , there is a crucial need for a technique that transfers knowledge across tasks with similar dynamics but different observation spaces . Besides these existing common applications , there are more benefits of across-observation transfer . For example , observations in real-world environments are usually rich and redundant , so that directly learning a policy is hard and expensive . If we can transfer knowledge from low-dimensional and informative vector observations ( usually available in a simulator ) to richer observations , the learning efficiency can be significantly improved . Therefore , an effective transfer learning method enables many novel and interesting applications , such as curriculum learning via observation design . In this paper , we aim to fill the gap and propose a new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a ( drastically ) different observation space . In order to meet more practical needs , we focus on the challenging setting where the observation change is : ( 1 ) unpredictable ( there is no prior knowledge about how the observations change ) , ( 2 ) drastic ( the source and target tasks have significantly different observation feature spaces , e.g. , vector to image ) , and ( 3 ) irretrievable ( once the change happens , it is impossible to query the source task , so that the agent can not interact with both environments simultaneously ) . Note that different from many prior works ( Taylor et al. , 2007 ; Mann & Choe , 2013 ) , we do not assume the knowledge of any inter-task mapping . That is , the agent does not know which new observation feature is corresponding the which old observation feature . To remedy the above challenges and achieve knowledge transfer , we make a key observation that , if only the observation features change , the source and target tasks share the same latent space and dynamics ( e.g . in Figure 1 , O ( S ) and O ( T ) can be associated to the same latent state ) . Therefore , we first disentangle representation learning from policy learning , and then accelerate the target-task agent by regularizing the representation learning process with the latent dynamics model learned in the source task . We show by theoretical analysis and empirical evaluation that the target task can be learned more efficiently with our proposed transfer learning method than from scratch . Summary of Contributions . ( 1 ) To the best of our knowledge , we are the first to discuss the transfer problem where the source and target tasks have drastically different observation feature spaces , and there is no prior knowledge of an inter-task mapping . ( 2 ) We theoretically characterize what constitutes a “ good representation ” and analyze the sufficient conditions the representation should satisfy . ( 3 ) Theoretical analysis shows that a model-based regularizer enables efficient representation learning in the target task . Based on this , we propose a novel algorithm that automatically transfers knowledge across observation representations . ( 4 ) Experiments in 7 environments show that our proposed algorithm significantly improves the learning performance of RL agents in the target task . 2 PRELIMINARIES AND BACKGROUND . Basic RL Notations . An RL task can be modeled by a Markov Decision Process ( MDP ) ( Puterman , 2014 ) , defined as a tuple M = 〈O , A , P , R , γ〉 , where O is the state/observation space , A is the action space , P is the transition kernel , R is the reward function and γ is the discount factor . At timestep t , the agent observes state ot , takes action at based on its policy π : O → ∆ ( A ) ( where ∆ ( · ) denotes the space of probability distributions ) , and receives reward rt = R ( ot , at ) . The environment then proceeds to the next state ot+1 ∼ P ( ·|ot , at ) . The goal of an RL agent is to find a policy π in the policy space Π with the highest cumulative reward , which is characterized by the value functions . The value of a policy π for a state o ∈ O is defined as V π ( o ) = Eπ , P [ ∑∞ t=0 γ trt|o0 = o ] . The Q value of a policy π for a state-action pair ( o , a ) ∈ O × A is defined as Qπ ( o , a ) = Eπ , P [ ∑∞ t=0 γ trt|o0 = o , a0 = a ] . Appendix A provides more background of RL . Representation Learning in RL . Real-world applications usually have large observation spaces for which function approximation is needed to learn the value or the policy . However , directly learning a policy over the entire observation space could be difficult , as there is usually redundant information in the observation inputs . A common solution is to map the large-scale observation into a smaller representation space via a non-linear encoder ( also called a representation mapping ) φ : O → Rd , where d is the representation dimension , and then learn the policy/value function over the representation space φ ( O ) . In DRL , the encoder and the policy/value are usually jointly learned . 3 PROBLEM SETUP : TRANSFER ACROSS DIFFERENT OBSERVATION SPACES . We aim to transfer knowledge learned from a source MDP to a target MDP , whose observation spaces are different while dynamics are structurally similar . Denote the source MDP as M ( S ) = 〈O ( S ) , A , P ( S ) , R ( S ) , γ〉 , and the target MDP asM ( T ) = 〈O ( T ) , A , P ( T ) , R ( T ) , γ〉 . Note that O ( S ) and O ( T ) can be significantly different , such asO ( S ) being a low-dimensional vector space andO ( T ) being a high-dimensional pixel space , which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally . In this work , as motivated in the Introduction , we focus on the setting wherein the dynamics ( ( P ( S ) , R ( S ) ) and ( P ( T ) , R ( T ) ) ) of the two MDPs between which we transfer knowledge are defined on different observation spaces but share structural similarities . Specifically , we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task . We formalize this in Assumption 1 : Assumption 1 . There exists a function f : O ( T ) → O ( S ) such that ∀o ( T ) i , o ( T ) j ∈ O ( T ) , ∀a ∈ A , P ( T ) ( o ( T ) j |o ( T ) i , a ) = P ( S ) ( f ( o ( T ) j ) |f ( o ( T ) i ) , a ) , R ( T ) ( o ( T ) i , a ) = R ( S ) ( f ( o ( T ) i ) , a ) . Remarks . ( 1 ) Assumption 1 is mild as many real-world scenarios fall under this assumption . For instance , when upgrading the cameras of a patrol robot to have higher resolutions , such a mapping f can be a down-sampling function . ( 2 ) Our algorithm does not assume any prior knowledge of the mapping f , although Assumption 1 assumes the existence of such an underlying mapping . ( 3 ) f is a general function without extra restrictions . f can be a many-to-one mapping , i.e. , more than one target observations can be related to the same observation in the source task . f can be non-surjective , i.e. , there could exist source observations that do not correspond to any target observation . Many prior works ( Mann & Choe , 2013 ; Brys et al. , 2015 ) have similar assumptions , but require prior knowledge of such an inter-task mapping to achieve knowledge transfer . However , such a mapping might not be available in practice . As an alternative , in the next section , we propose to learn a latent representation of the observations and a dynamics model in this latent space . The dynamics model is transferred to speed up learning in the target task , where we do not learn the mapping f explicitly . 4 METHODOLOGY : TRANSFER WITH REGULARIZED REPRESENTATION . The main idea of our proposed transfer method is to learn a latent dynamics model in the source task , and use the dynamics model to regularize representation learning in the target task . This section is organized as follows . First , we formally characterize “ what a good representation is for RL ” in Section 4.1 , which is crucial to motivate our algorithm . Then , we introduce our proposed algorithm for across-observation transfer in Section 4.2 , where we learn a latent dynamics model as an auxiliary task in the source task , and transfer the dynamics model to the target task to regularize representation learning . Theoretical analysis for the proposed algorithm is provided in Section 4.3 , which explains why the proposed method works and provides insights for the architecture design .

Summary: The paper considers the problem of RL environments where the observations space can change dramatically, e.g. from proprioceptive to pixel observations. The authors introduce various definitions for this setting and provide proofs regarding convergence (in the sense of Q-values converging to true values). Over 5 environments in continuous control, the authors demonstrate that their method enables transfer learning from proprioceptive to pixel observations and that such transfer learning can be beneficial.