Transfer RL across Observation Feature Spaces via Model-Based Regularization

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has the potential to be used in many large-scale applications such as robotics , gaming and automotive . In these real-life scenarios , it is an essential ability for agents to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks , which is known as Transfer RL ( TRL ) . Most existing TRL works ( Taylor & Stone , 2009 ; Zhu et al. , 2020 ) focus on tasks with the same state-action space but different dynamics/reward . However , these approaches do not apply to the case where the observation space changes significantly . Observation change is common in practice as in the following scenarios . ( 1 ) Incremental environment development . RL is used to train non-player characters ( NPC ) in games ( Juliani et al. , 2018 ) , which may be frequently updated . When there are new scenes , characters , or obstacles added to the game , the agent ’ s observation space will change accordingly . ( 2 ) Hardware upgrade/replacement . For robots with sensory observations ( Bohez et al. , 2017 ) , the observation space could change ( e.g . from text to audio , from lidar to camera ) as the sensor changes . ( 3 ) Restricted data access . In some RL applications ( Ganesh et al. , 2019 ) , agent observation contains sensitive data ( e.g . inventory ) which may become unavailable in the future due to data restrictions . In these cases , the learner may have to discard the old policy and train a new policy from scratch , as the policy has a significantly different input space , even though the underlying dynamics are similar . But training an RL policy from scratch can be expensive and unstable . Therefore , there is a crucial need for a technique that transfers knowledge across tasks with similar dynamics but different observation spaces . Besides these existing common applications , there are more benefits of across-observation transfer . For example , observations in real-world environments are usually rich and redundant , so that directly learning a policy is hard and expensive . If we can transfer knowledge from low-dimensional and informative vector observations ( usually available in a simulator ) to richer observations , the learning efficiency can be significantly improved . Therefore , an effective transfer learning method enables many novel and interesting applications , such as curriculum learning via observation design . In this paper , we aim to fill the gap and propose a new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a ( drastically ) different observation space . In order to meet more practical needs , we focus on the challenging setting where the observation change is : ( 1 ) unpredictable ( there is no prior knowledge about how the observations change ) , ( 2 ) drastic ( the source and target tasks have significantly different observation feature spaces , e.g. , vector to image ) , and ( 3 ) irretrievable ( once the change happens , it is impossible to query the source task , so that the agent can not interact with both environments simultaneously ) . Note that different from many prior works ( Taylor et al. , 2007 ; Mann & Choe , 2013 ) , we do not assume the knowledge of any inter-task mapping . That is , the agent does not know which new observation feature is corresponding the which old observation feature . To remedy the above challenges and achieve knowledge transfer , we make a key observation that , if only the observation features change , the source and target tasks share the same latent space and dynamics ( e.g . in Figure 1 , O ( S ) and O ( T ) can be associated to the same latent state ) . Therefore , we first disentangle representation learning from policy learning , and then accelerate the target-task agent by regularizing the representation learning process with the latent dynamics model learned in the source task . We show by theoretical analysis and empirical evaluation that the target task can be learned more efficiently with our proposed transfer learning method than from scratch . Summary of Contributions . ( 1 ) To the best of our knowledge , we are the first to discuss the transfer problem where the source and target tasks have drastically different observation feature spaces , and there is no prior knowledge of an inter-task mapping . ( 2 ) We theoretically characterize what constitutes a “ good representation ” and analyze the sufficient conditions the representation should satisfy . ( 3 ) Theoretical analysis shows that a model-based regularizer enables efficient representation learning in the target task . Based on this , we propose a novel algorithm that automatically transfers knowledge across observation representations . ( 4 ) Experiments in 7 environments show that our proposed algorithm significantly improves the learning performance of RL agents in the target task . 2 PRELIMINARIES AND BACKGROUND . Basic RL Notations . An RL task can be modeled by a Markov Decision Process ( MDP ) ( Puterman , 2014 ) , defined as a tuple M = 〈O , A , P , R , γ〉 , where O is the state/observation space , A is the action space , P is the transition kernel , R is the reward function and γ is the discount factor . At timestep t , the agent observes state ot , takes action at based on its policy π : O → ∆ ( A ) ( where ∆ ( · ) denotes the space of probability distributions ) , and receives reward rt = R ( ot , at ) . The environment then proceeds to the next state ot+1 ∼ P ( ·|ot , at ) . The goal of an RL agent is to find a policy π in the policy space Π with the highest cumulative reward , which is characterized by the value functions . The value of a policy π for a state o ∈ O is defined as V π ( o ) = Eπ , P [ ∑∞ t=0 γ trt|o0 = o ] . The Q value of a policy π for a state-action pair ( o , a ) ∈ O × A is defined as Qπ ( o , a ) = Eπ , P [ ∑∞ t=0 γ trt|o0 = o , a0 = a ] . Appendix A provides more background of RL . Representation Learning in RL . Real-world applications usually have large observation spaces for which function approximation is needed to learn the value or the policy . However , directly learning a policy over the entire observation space could be difficult , as there is usually redundant information in the observation inputs . A common solution is to map the large-scale observation into a smaller representation space via a non-linear encoder ( also called a representation mapping ) φ : O → Rd , where d is the representation dimension , and then learn the policy/value function over the representation space φ ( O ) . In DRL , the encoder and the policy/value are usually jointly learned . 3 PROBLEM SETUP : TRANSFER ACROSS DIFFERENT OBSERVATION SPACES . We aim to transfer knowledge learned from a source MDP to a target MDP , whose observation spaces are different while dynamics are structurally similar . Denote the source MDP as M ( S ) = 〈O ( S ) , A , P ( S ) , R ( S ) , γ〉 , and the target MDP asM ( T ) = 〈O ( T ) , A , P ( T ) , R ( T ) , γ〉 . Note that O ( S ) and O ( T ) can be significantly different , such asO ( S ) being a low-dimensional vector space andO ( T ) being a high-dimensional pixel space , which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally . In this work , as motivated in the Introduction , we focus on the setting wherein the dynamics ( ( P ( S ) , R ( S ) ) and ( P ( T ) , R ( T ) ) ) of the two MDPs between which we transfer knowledge are defined on different observation spaces but share structural similarities . Specifically , we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task . We formalize this in Assumption 1 : Assumption 1 . There exists a function f : O ( T ) → O ( S ) such that ∀o ( T ) i , o ( T ) j ∈ O ( T ) , ∀a ∈ A , P ( T ) ( o ( T ) j |o ( T ) i , a ) = P ( S ) ( f ( o ( T ) j ) |f ( o ( T ) i ) , a ) , R ( T ) ( o ( T ) i , a ) = R ( S ) ( f ( o ( T ) i ) , a ) . Remarks . ( 1 ) Assumption 1 is mild as many real-world scenarios fall under this assumption . For instance , when upgrading the cameras of a patrol robot to have higher resolutions , such a mapping f can be a down-sampling function . ( 2 ) Our algorithm does not assume any prior knowledge of the mapping f , although Assumption 1 assumes the existence of such an underlying mapping . ( 3 ) f is a general function without extra restrictions . f can be a many-to-one mapping , i.e. , more than one target observations can be related to the same observation in the source task . f can be non-surjective , i.e. , there could exist source observations that do not correspond to any target observation . Many prior works ( Mann & Choe , 2013 ; Brys et al. , 2015 ) have similar assumptions , but require prior knowledge of such an inter-task mapping to achieve knowledge transfer . However , such a mapping might not be available in practice . As an alternative , in the next section , we propose to learn a latent representation of the observations and a dynamics model in this latent space . The dynamics model is transferred to speed up learning in the target task , where we do not learn the mapping f explicitly . 4 METHODOLOGY : TRANSFER WITH REGULARIZED REPRESENTATION . The main idea of our proposed transfer method is to learn a latent dynamics model in the source task , and use the dynamics model to regularize representation learning in the target task . This section is organized as follows . First , we formally characterize “ what a good representation is for RL ” in Section 4.1 , which is crucial to motivate our algorithm . Then , we introduce our proposed algorithm for across-observation transfer in Section 4.2 , where we learn a latent dynamics model as an auxiliary task in the source task , and transfer the dynamics model to the target task to regularize representation learning . Theoretical analysis for the proposed algorithm is provided in Section 4.3 , which explains why the proposed method works and provides insights for the architecture design .

Summary: The paper addresses the problem of adapting a policy to a novel representation of the environment. Different from most previous work the paper focuses on the problem of a change in the representation of the observation provided by the environment, i.e., assuming that the underlying dynamics P and R remains mostly similar. The idea is to learn source and target models of the environment jointly with the policy. This serves as a model-based regularization that guides policy and representation towards the learned models. This effectively leads to a learned representation that is shared for the policy for the source and the target environment representation.