Year: 2021

Abstract:
In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task.

heading: 1 INTRODUCTION
text: Deep Reinforcement Learning (DRL) has the potential to be used in many large-scale applications such as robotics, gaming and automotive. In these real-life scenarios, it is an essential ability for agents to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks, which is known as Transfer RL (TRL). Most existing TRL works (Taylor & Stone, 2009; Zhu et al., 2020) focus on tasks with the same state-action space but different dynamics/reward. However, these approaches do not apply to the case where the observation space changes significantly.
Observation change is common in practice as in the following scenarios. (1) Incremental environment development. RL is used to train non-player characters (NPC) in games (Juliani et al., 2018), which may be frequently updated. When there are new scenes, characters, or obstacles added to the game, the agent’s observation space will change accordingly. (2) Hardware upgrade/replacement. For robots with sensory observations (Bohez et al., 2017), the observation space could change (e.g. from text to audio, from lidar to camera) as the sensor changes. (3) Restricted data access. In some RL applications (Ganesh et al., 2019), agent observation contains sensitive data (e.g. inventory) which may become unavailable in the future due to data restrictions. In these cases, the learner may have to discard the old policy and train a new policy from scratch, as the policy has a significantly different input space, even though the underlying dynamics are similar. But training an RL policy from scratch can be expensive and unstable. Therefore, there is a crucial need for a technique that transfers knowledge across tasks with similar dynamics but different observation spaces.
Besides these existing common applications, there are more benefits of across-observation transfer. For example, observations in real-world environments are usually rich and redundant, so that directly learning a policy is hard and expensive. If we can transfer knowledge from low-dimensional and informative vector observations (usually available in a simulator) to richer observations, the learning efficiency can be significantly improved. Therefore, an effective transfer learning method enables many novel and interesting applications, such as curriculum learning via observation design.
In this paper, we aim to fill the gap and propose a new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a (drastically) different observation space. In order to meet more practical needs, we focus on the challenging setting where the observation change is: (1) unpredictable (there is no prior knowledge about how the observations change), (2) drastic (the source and target tasks have significantly different
observation feature spaces, e.g., vector to image), and (3) irretrievable (once the change happens, it is impossible to query the source task, so that the agent can not interact with both environments simultaneously). Note that different from many prior works (Taylor et al., 2007; Mann & Choe, 2013), we do not assume the knowledge of any inter-task mapping. That is, the agent does not know which new observation feature is corresponding the which old observation feature.
To remedy the above challenges and achieve knowledge transfer, we make a key observation that, if only the observation features change, the source and target tasks share the same latent space and dynamics (e.g. in Figure 1, O(S) and O(T ) can be associated to the same latent state). Therefore, we first disentangle representation learning from policy learning, and then accelerate the target-task agent by regularizing the representation learning process with the latent dynamics model learned in the source task. We show by theoretical analysis and empirical evaluation that the target task can be learned more efficiently with our proposed transfer learning method than from scratch.
Summary of Contributions. (1) To the best of our knowledge, we are the first to discuss the transfer problem where the source and target tasks have drastically different observation feature spaces, and there is no prior knowledge of an inter-task mapping. (2) We theoretically characterize what constitutes a “good representation” and analyze the sufficient conditions the representation should satisfy. (3) Theoretical analysis shows that a model-based regularizer enables efficient representation learning in the target task. Based on this, we propose a novel algorithm that automatically transfers knowledge across observation representations. (4) Experiments in 7 environments show that our proposed algorithm significantly improves the learning performance of RL agents in the target task.

heading: 2 PRELIMINARIES AND BACKGROUND
text: Basic RL Notations. An RL task can be modeled by a Markov Decision Process (MDP) (Puterman, 2014), defined as a tuple M = 〈O,A, P,R, γ〉, where O is the state/observation space, A is the action space, P is the transition kernel, R is the reward function and γ is the discount factor. At timestep t, the agent observes state ot, takes action at based on its policy π : O → ∆(A)(where ∆(·) denotes the space of probability distributions), and receives reward rt = R(ot, at). The environment then proceeds to the next state ot+1 ∼ P (·|ot, at). The goal of an RL agent is to find a policy π in the policy space Π with the highest cumulative reward, which is characterized by the value functions. The value of a policy π for a state o ∈ O is defined as V π(o) = Eπ,P [ ∑∞ t=0 γ
trt|o0 = o]. The Q value of a policy π for a state-action pair (o, a) ∈ O × A is defined as Qπ(o, a) = Eπ,P [ ∑∞ t=0 γ
trt|o0 = o, a0 = a]. Appendix A provides more background of RL. Representation Learning in RL. Real-world applications usually have large observation spaces for which function approximation is needed to learn the value or the policy. However, directly learning a policy over the entire observation space could be difficult, as there is usually redundant information in the observation inputs. A common solution is to map the large-scale observation into a smaller representation space via a non-linear encoder (also called a representation mapping) φ : O → Rd, where d is the representation dimension, and then learn the policy/value function over the representation space φ(O). In DRL, the encoder and the policy/value are usually jointly learned.

heading: 3 PROBLEM SETUP: TRANSFER ACROSS DIFFERENT OBSERVATION SPACES
text: We aim to transfer knowledge learned from a source MDP to a target MDP, whose observation spaces are different while dynamics are structurally similar. Denote the source MDP as M(S) = 〈O(S),A, P (S), R(S), γ〉, and the target MDP asM(T ) = 〈O(T ),A, P (T ), R(T ), γ〉. Note that O(S) and O(T ) can be significantly different, such asO(S) being a low-dimensional vector space andO(T ) being a high-dimensional pixel space, which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally.
In this work, as motivated in the Introduction, we focus on the setting wherein the dynamics ((P (S), R(S)) and (P (T ), R(T ))) of the two MDPs between which we transfer knowledge are defined on different observation spaces but share structural similarities. Specifically, we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task. We formalize this in Assumption 1: Assumption 1. There exists a function f : O(T ) → O(S) such that ∀o(T )i , o (T ) j ∈ O(T ),∀a ∈ A,
P (T )(o(T )j |o (T ) i , a) = P (S)(f(o(T )j )|f(o (T ) i ), a), R (T )(o(T )i , a) = R (S)(f(o(T )i ), a).
Remarks. (1) Assumption 1 is mild as many real-world scenarios fall under this assumption. For instance, when upgrading the cameras of a patrol robot to have higher resolutions, such a mapping f can be a down-sampling function. (2) Our algorithm does not assume any prior knowledge of the mapping f , although Assumption 1 assumes the existence of such an underlying mapping. (3) f is a general function without extra restrictions. f can be a many-to-one mapping, i.e., more than one target observations can be related to the same observation in the source task. f can be non-surjective, i.e., there could exist source observations that do not correspond to any target observation.
Many prior works (Mann & Choe, 2013; Brys et al., 2015) have similar assumptions, but require prior knowledge of such an inter-task mapping to achieve knowledge transfer. However, such a mapping might not be available in practice. As an alternative, in the next section, we propose to learn a latent representation of the observations and a dynamics model in this latent space. The dynamics model is transferred to speed up learning in the target task, where we do not learn the mapping f explicitly.

heading: 4 METHODOLOGY: TRANSFER WITH REGULARIZED REPRESENTATION
text: The main idea of our proposed transfer method is to learn a latent dynamics model in the source task, and use the dynamics model to regularize representation learning in the target task. This section is organized as follows. First, we formally characterize “what a good representation is for RL” in Section 4.1, which is crucial to motivate our algorithm. Then, we introduce our proposed algorithm for across-observation transfer in Section 4.2, where we learn a latent dynamics model as an auxiliary task in the source task, and transfer the dynamics model to the target task to regularize representation learning. Theoretical analysis for the proposed algorithm is provided in Section 4.3, which explains why the proposed method works and provides insights for the architecture design.

heading: 4.1 CHARACTERIZING CONDITIONS FOR GOOD REPRESENTATIONS
text: As discussed in Section 2, real-world applications usually have rich and redundant observations, where learning a good representation (Jaderberg et al., 2016; Dabney et al., 2020) is essential for efficiently finding an optimal policy. However, the properties that constitute a good representation for an RL task are still an open question. Some prior works (Bellemare et al., 2019; Dabney et al., 2020; Gelada et al., 2019) have discussed the representation quality in DRL, but we take a different perspective and focus on characterizing the sufficient properties of representation for learning a task.
Given a representation mapping φ, the Q value of any (o, a) ∈ O ×A can be approximately represented by a function of φ(o), i.e., Q̂(o, a) = f(φ(o); θa), where θa is a set of learnable parameters. For example, if linear approximation is used, the Q value can be represented as Q̂(o, a) = φ(o)θa, where θa ∈ Rd is a vector associated with action a. To study the relation between representation quality and approximation quality, we define an approximation operator Hφ, which approximates the Q value with hypothesis class H. Formally, let Θ denote the parameter space of functions in H,
then HφQ(o) := h(φ(o); θ∗), where θ∗ = argminθ∈ΘE[‖h(φ(o); θ)−Q(φ, ·)‖]. Such an approximation can be realized by neural networks as universal function approximators (Hornik et al., 1989). Therefore, the approximation error ‖Q− Q̂‖ only depends on the selection of the representation φ. The quality of the encoder φ is crucial for learning an accurate value function or learning a good policy. The ideal encoder φ should discard irrelevant information in the raw observation but keep essential information. In supervised or self-supervised representation learning (Chen et al., 2020; Achille & Soatto, 2018), it is believed that a good representation φ(X) of input X should contain minimal information of X which maintaining sufficient information for predicting the label Y . However, in RL, it is difficult to identify whether a representation is sufficient, since there is no label corresponding to each input. The focus of an agent is to estimate the value of each input o ∈ O, which is associated with some policy. Therefore, we point out that the representation quality in RL is policy-dependent. Below, we formally characterize the sufficiency of a representation mapping in terms of a fixed policy and learning a task.
Sufficiency for A Fixed Policy. If the agent is executing a fixed policy, and its goal is to estimate the expected future return from the environment, then a representation is sufficient for the policy as long as it can encode the policy value Vπ . A formal definition is provided by Definition 9 in Appendix B.
Sufficiency for Learning A Task. The goal of RL is to find an optimal policy. Therefore, it is not adequate for the representation to only fit one policy. Intuitively, a representation mapping is sufficient for learning if we are able to find an optimal policy over the representation space φ(O), which requires multiple iterations of policy evaluation and policy improvement. Definition 2 below defines a set of “important” policies for learning with φ(O). Definition 2 (Encoded Deterministic Policies). For a given representation mapping φ(·), define an encoded deterministic policy set ΠDφ as the set of policies that are deterministic and take the same actions for observations with the same representations. Formally, ΠDφ := {π ∈ Π : ∃π̃ : φ(O) → A, s.t. ∀o ∈ O, π(o) = π̃(φ(o))}.
A policy π is in ΠDφ if it does not distinguish o1 and o2 when φ(o1) = φ(o2). Therefore, Π D φ can be regarded as deterministic policies that make decisions for encoded observations. Now, we define the concept of sufficient representation for learning in an MDP. Definition 3 (Sufficient Representation for Learning). A representation mapping φ is sufficient for a task M w.r.t. approximation operator Hφ if HφQπ = Qπ , for all π ∈ ΠDφ . (φ is -sufficient for learning if ‖HφQπ −Qπ‖ ≤ and linearly-sufficient ifHφ is linear transformation.)
Definition 3 suggests that the representation is sufficient for learning a task as long as it is sufficient for policies in ΠDφ . Then, the lemma below justifies that a nearly sufficient representation can ensure that approximate policy iteration converges to a near-optimal solution. Lemma 4 (Error Bound for Approximate Policy Iteration). If φ is -sufficient for task M (with `∞ norm), then the approximated policy iteration with approximation operator Hφ starting from any initial policy that is encoded by φ (π0 ∈ ΠDφ ) satisfies
lim sup k→∞
‖Q∗ −Qπk‖∞ ≤ 2γ2
(1− γ)2 , (1)
where πk is the policy in the k-th iteration.
Lemma 4, proved in Appendix C, is extended from the error bound provided by Bertsekas & Tsitsiklis (1996). For simplicity, we consider the bound in `∞, but tighter bounds can be derived with other norms (Munos, 2005), although a tighter bound is not the focus of this paper.
How Can We Learn A Sufficient Representation? So far we have provided a principle to define whether a given representation is sufficient for learning. In DRL, the representation is learned together with the policy or value function using neural networks, but the quality of the representation may be poor (Dabney et al., 2020), which makes it hard for the agent to find an optimal policy. Based on Definition 3, a natural method to learn a good representation is to let the representation fit as many policy values as possible as auxiliary tasks, which matches the ideas in other works. For example, Bellemare et al. (2019) propose to fit a set of representative policies (called adversarial value functions). Dabney et al. (2020) choose to fit the values of all past policies (along the value improvement path), which requires less computational resource. Different from these works that
Algorithm 1 Source Task Learning Require: Regularization weight λ; update frequency m for stable encoder.
1: Initialize encoder φ(S), stable encoder φ̂(S), policy π(S), transition prediction network P̂ and reward prediction network R̂. 2: for t = 0, 1, · · · do 3: Take action at ∼ π(S)(φ(S)(o(S)t )), get next observation o (S)
t+1 and reward rt, store to buffer. 4: Sample a mini-batch {oi, ai, ri, o′i}Ni=1 from the buffer. 5: Update P̂ and R̂ using one-step gradient descent with ∇P̂LP (φ̂(S); P̂ ) and ∇R̂LR(φ̂(S); R̂),
where LP and LR are defined in Equation (2). 6: Update encoder and policy by minπ(S),φ(S) Lbase(φ(S), π(S))+λ ( LP (φ (S); P̂ )+LR(φ (S); R̂) ) . 7: if t | m then Update the stable encoder φ̂(S) ← φ(S).
Algorithm 2 Target Task Learning with Transferred Dynamics Models
Require: Regularization weight λ; dynamics models P̂ and R̂ learned in the source task. 1: Initialize encoder φ(T ), policy π(T ) 2: for t = 0, 1, · · · do 3: Take action at ∼ π(T )(φ(T )(o(T )t )), get next observation o (T )
t+1 and reward rt, store to buffer. 4: Sample a mini-batch {oi, ai, ri, o′i}Ni=1 from the buffer. 5: Update encoder and policy by minφ(T ),π(T ) Lbase(φ(T ), π(T ))+λ ( LP (φ (T ); P̂ )+LR(φ (T ); R̂)).
directly fit the value functions of multiple policies, in Section 4.2, we propose a computationally efficient way to realize sufficient representation for learning and knowledge transferring: to fit and transfer an auxiliary policy-independent dynamics model.

heading: 4.2 ALGORITHM: LEARNING AND TRANSFERRING MODEL-BASED REGULARIZER
text: Our goal is to use the knowledge learned in the source task to learn a good representation in the target task, such that the agent learns the target task more easily than learning from scratch. Since we focus on developing a generic transfer mechanism, the base learner can be any DRL algorithms. We use Lbase to denote the loss function of the base learner.
As motivated in Section 4.1, we propose to learn policy-independent dynamics models for producing high-quality representations: (1) P̂ which predicts the representation of the next state for a given state action pair, and (2) R̂ which predicts the immediate reward for a given state-action pair. For a batch of N transition samples {oi, ai, o′i, ri}Ni=1, define the transition loss and the reward loss as:
LP (φ, P̂ ) = 1
N N∑ i=1 (P̂ (φ(oi), ai)− φ̄(o′i))2, LR(φ, R̂) = 1 N N∑ i=1 (R̂(φ(oi), ai)− ri)2 (2)
where φ̄(o′i) denotes the representation of the next state o ′ i with stop gradients. In order to fit a more diverse state distribution, transition samples are drawn from an off-policy buffer, which stores shuffled past trajectories. Note that a similar model loss is commonly used in model-based RL (van der Pol et al., 2020) and bisimulation works Zhang et al. (2020b), but our focus is to use the learned model as an auxiliary task and to transfer the model to the target task.
The learning procedures for the source task and the target task are illustrated in Algorithm 1 and Algorithm 2, respectively. Figure 2 depicts the architecture of the learning model for both source and target tasks. z = φ(o) and z′ = φ̄(o′) are the encoded observation and next observation. Given the current encoding z and the action a, the dynamics models P̂ and R̂ return the predicted next encoding ẑ′ = P̂ (z, a) and predicted reward r̂ = R̂(z, a). Then the transition loss is the mean squared error (MSE) between z′ and ẑ′ in a batch; the reward loss is the MSE between r and r̂ in a batch.
In the source task (Algorithm 1): dynamics models P̂ and R̂ are learned by minimizing LP and LR, which are computed based on a recent copy of encoder called stable encoder φ̂(S) (Line 5). The computation of the stable encoder is to help the dynamics models converge, as the actual encoder φ(S) changes at every step. Note that a stable copy of the network is widely used in many DRL algorithms (e.g. the target network in DQN), which can be directly regarded as φ̂(S) without maintaining an extra network. The actual encoder φ(S) is regularized by the auxiliary dynamics models P̂ and R̂ (Line 6).
In the target task (Algorithm 2): dynamics model P̂ and R̂ are transferred from the source task and fixed during learning. Therefore, the learning of φ(T ) is regularized by static dynamics models, which leads to faster and more stable convergence than naively learning an auxiliary task.
Relation and Difference with Model-based RL and Bisimulation Metrics. The goal of modelbased RL (Kipf et al., 2019; Grimm et al., 2020) is to learn an accurate world model and use the model for planning. The dynamics model could be learned on either raw observations or representations. In our framework, we also learn a dynamics model, but the model serves as an auxiliary task, and learning is still performed by the model-free base learner with Lbase. Bisimulation metrics (Castro, 2020; Zhang et al., 2020b) aim to approximate the distances among states by learning dynamics models, whereas we do not explicitly measure the distance among states. Note that we also do not require a reconstruction loss that is common in literature (Lee et al., 2019).

heading: 4.3 THEORETICAL ANALYSIS: BENEFITS OF TRANSFERABLE DYNAMICS MODEL
text: The algorithms introduced in Section 4.2 consist of two designs: learning a latent dynamics model as an auxiliary task, and transferring the dynamics model to the target task. In this section, we show theoretical justifications and practical advantages of our proposed method. We aim to answer the following two questions: (1) How does learning an auxiliary dynamics model help with representation learning? (2) Is the auxiliary dynamics model transferable?
For notational simplicity, let Pa and Ra denote the transition and reward functions associated with action a ∈ A. Note that Pa and Ra are independent of any policy. We then define the sufficiency of a representation mapping w.r.t. dynamics models as below. Definition 5 (Policy-independent Model Sufficiency). For an MDP M , a representation mapping φ is sufficient for its dynamics (Pa, Ra)a∈A if ∀a ∈ A, there exists functions P̂a : Rd → Rd and R̂a : Rd → R such that ∀o ∈ O, P̂a(φ(o)) = Eo′∼Pa(o)[φ(o′)], R̂a(φ(o)) = Ra(o).
Remarks. (1) Since we focus on learning a deterministic transition prediction model P̂ , φ is exactly sufficient for dynamics (Pa, Ra)a∈A only when the transition function P is deterministic. (2) If P is stochastic, but we have maxo,a ‖Eo′∼Pa(o)[φ(o′)] − P̂a(φ(o))‖ ≤ P and maxo,a |Ra(o) − R̂a(φ(o))| ≤ R, then φ is ( P , R)-sufficient for the dynamics of M . Next we show by Proposition 6 and Theorem 7 that learning sufficiency can be achieved via ensuring model sufficiency. Proposition 6 (Learning Sufficiency Induced by Policy-independent Model Sufficiency). Consider an MDP M with deterministic transition function P and reward function R. If φ is sufficient for (Pa, Ra)a∈A, then it is sufficient (but not necessarily linearly sufficient) for learning in M .
Proposition 6 shows that, if the transition is deterministic and the model errors LP , LR are zero, then φ is exactly sufficient for learning. More generally, if the transition function P is not deterministic, and model fitting is not perfect, the learned representation can still be nearly sufficient for learning as characterized by Theorem 7 below, which is extended from a variant of the value difference bound derived by Gelada et al. (2019). Proposition 6 and Theorem 7 justify that learning the latent dynamics model as an auxiliary task encourages the representation to be sufficient for learning. The model error LP and LR defined in Section 4.2 can indicate how good the representation is. Theorem 7. For an MDP M , if representation mapping φ is ( P , R)-sufficient for the dynamics of M , then approximate policy iteration with approximation operator Hφ starting from any initial policy π0 ∈ ΠDφ satisfies
lim sup k→∞
‖Q∗ −Qπk‖∞ ≤ 2γ2
(1− γ)3 ( R + γ PKφ,V ). (3)
where Kφ,V is an upper bound of the value Lipschitz constant as defined in Appendix B.
Transferring Model to Get Better Representation in Target. Although Proposition 6 shows that learning auxiliary dynamics models benefits representation learning, finding the optimal solution is non-trivial since one still has to learn P̂ and R̂. Therefore, the main idea of our algorithm is to transfer the dynamics models P̂ , R̂ from the source task to the target task, to significantly ease the learning in the target task. Theorem 8 below guarantees that transferring the dynamics models is feasible. Our experimental result in Section 6 verifies that learning with transferred and fixed dynamics models outperforms learning with randomly initialized dynamics models. Theorem 8 (Transferable Dynamics Models). Consider a source task M (S) and a target task M (T ) with deterministic transition functions. Suppose φ(S) is sufficient for (P (S)a , R (S) a )a∈A with functions P̂a, R̂a, then there exists a representation φ(T ) satisfying P̂a(φ(o)) = Eo′∼P (T )a [φ(o ′)], R̂a(φ(o)) = R(T )a (o), for all o ∈ O(T ), and φ(T ) is sufficient for learning in M (T ).
Theorem 8 shows that the learned latent dynamics models P̂ , R̂ are transferable from the source task to the target task. For simplicity, Theorem 8 focuses on exact sufficiency as in Proposition 6, but it can be easily extended to -sufficiency if combined with Theorem 7. Proofs for Proposition 6, Theorem 7 and Theorem 8 are all provided in Appendix C.
Trade-off between Approximation Complexity and Representation Complexity. As suggested by Proposition 6, fitting policy-independent dynamics encourages the representation to be sufficient for learning, but not necessarily linearly sufficient. Therefore, we suggest using a non-linear policy/value head following the representation to reduce the approximation error. Linear sufficiency can be achieved if φ is made linearly sufficient for Pπ and Rπ for all π ∈ ΠDφ , where Pπ and Rπ are transition and reward functions induced by policy π (Proposition 10, Appendix B). However, using this method for transfer learning is expensive in terms of both computation and memory, as it requires to learn Pπ and Rπ for many different π’s and store these models for transferring to the target task. Therefore, there is a trade-off between approximation complexity and representation complexity. Learning a linearly sufficient representation reduces the complexity of the approximation operator. But it requires more complexity in the representation itself as it has to satisfy much more constraints. To develop a practical and efficient transfer method, we use a slightly more complex approximation operator (non-linear policy head) while keeping the auxiliary task simple and easy to transfer across tasks. Please see Appendix B for more detailed discussion about linear sufficiency.

heading: 5 RELATED WORK
text: Transfer RL across Observation Feature Spaces. Transferring knowledge between tasks with different observation spaces has been studied for years. Many existing approaches(Taylor et al., 2007; Mann & Choe, 2013; Brys et al., 2015) require an explicit mapping between the source and target observation spaces, which may be hard to obtain in practice. Raiman et al. (2019) introduce network surgery that deals with the change in the input features by determining which components of a neural network model should be transferred and which require retraining. However, it requires knowledge of the input feature maps, and is not designed for drastic changes, e.g. vector to pixel. Sun et al. (2020) propose a provably sample-efficient transfer learning algorithm that works for different observation spaces without knowing any inter-task mapping, but the algorithm is mainly designed for tabular RL and model-based RL which uses the model to plan for a policy, different from our setting. Gupta et al. (2017) achieve transfer learning between two different tasks by learning an invariant feature space, with a key time-based alignment assumption. We empirically compared this method with our proposed transfer algorithm in Section 6. Our work is also related to state abstraction in block MDPs, as studied by Zhang et al. (2020a). But the problem studied in Zhang et al. (2020a) is a multi-task setting where the agent aims to learn generalizable abstract states from a series of tasks. Another related topic is domain adaptation in RL (Higgins et al., 2017; Eysenbach et al., 2020; Zhang et al., 2020b), where the target observation space (e.g. real world) is different from the source observation (e.g. simulator). However, domain adaptation does not assume drastic observation changes (e.g. changed dimension). Moreover, the aim of domain adaptation is usually zero-shot generalization to new observations, thus prior knowledge or a few samples of the target domain is often needed (Eysenbach et al., 2020).
Representation Learning in RL. In environments with rich observations, representation learning is crucial for the efficiency of RL methods. Learning unsupervised auxiliary tasks (Jaderberg et al., 2016) is shown to be effective for learning a good representation. The relationship between learning
policy-dependent auxiliary tasks and learning good representations has been studied in some prior works (Bellemare et al., 2019; Dabney et al., 2020; Lyle et al., 2021), while our focus is to learn policy-independent auxiliary tasks to facilitate transfer learning. Using latent prediction models to regularize representation has been shown to be effective for various types of rich observations (Guo et al., 2020; Lee et al., 2019). Gelada et al. (2019) theoretically justify that learning latent dynamics model guarantees the quality of the learned representation, while we further characterize the relationship between representation and learning performance, and we utilize dynamics models to improve transfer learning. Zhang et al. (2020b) use a bisimulation metric to learn latent representations that are invariant to task-irrelevant details in observation. As pointed out by Achille & Soatto (2018), invariant and sufficient representation is indeed minimal sufficient, so it is an interesting future direction to combine our method with bisimulation metric to learn minimal sufficient representations. There is also a line of work using contrastive learning to train an encoder for pixel observations (Srinivas et al., 2020; Yarats et al., 2021; Stooke et al., 2021), which usually pre-train an encoder based on image samples using self-supervised learning. However, environment dynamics are usually not considered during pre-training. Our algorithm can be combined with these contrastive learning approaches to further improve learning performance in the target task.

heading: 6 EXPERIMENTAL EVALUATION
text: We empirically evaluate our transfer learning algorithm in various environments and multiple observation-change scenarios. Detailed experiment setup and hyperparameters are in Appendix E.
Baselines. To verify the effectiveness of our proposed transfer learning method, we compare our transfer learning algorithm with 4 baselines: (1) Single: a single-task base learner. (2) Auxiliary: learns auxiliary models from scratch to regularize representation. (3) Fine-tune: loads and freezes the source policy head, and retrains an encoder in the target task. (4) Time-aligned (Gupta et al., 2017): supposes the target task and the source task proceed to the same latent state given the same action sequence, and pre-trains a target-task encoder with saved source-task trajectories. More details of baseline implementations are in Appendix E.1.1.
Scenarios. As motivated in Section 1, there are many scenarios where one can benefit from transfer learning across observation feature spaces. We evaluate our proposed transfer algorithm in 7 environments that fit various scenarios, to simulate real-world applications: (1) Vec-to-pixel: a novel and challenging scenario, where the source task has low-dimensional vector observations and the target task has pixel observations. We use 3 vector-input environments CartPole, Acrobot and Cheetah-Run as source tasks, and use the rendered image in the target task. (2) More-sensor: another challenging scenario where the target task has a lot more sensors than the source task. We use 3 MuJoCo environments: HalfCheetah, Hopper and Walker2d, whose original observation dimensions are 17, 11 and 17, respectively. We add mass-based inertia and velocity (provided by MuJoCo’s API), resulting in 145, 91, 145 dimensions in the corresponding target tasks. (3) Broken-sensor: we use an existing game 3DBall contained in the Unity ML-Agents Toolkit (Juliani et al., 2018), which has two different observation specifications that naturally fit our transfer setting: the source observation has 8 features containing the velocity of the ball; the target observation does not have the ball’s velocity, thus the agent has to stack the past 9 frames to infer the velocity. Please see Appendix E.1.2 for more detailed descriptions of all the 7 environments.
Base DRL Learners. What we propose is a transfer learning mechanism that can be combined with any existing DRL methods. For environments with discrete action spaces (CartPole, Acrobot), we use the DQN algorithm (Mnih et al., 2015), while for environments with continuous action spaces (Cheetah-Run, HalfCheetah, Hopper, Walker2d, 3DBall), we use the SAC algorithm (Haarnoja et al., 2018). To ensure a fair comparison, we use the same base DRL learner with the same hyperparameter settings for all tested methods, as detailed in Appendix E.1.3. As is common in prior works, our implementation of the RL algorithms is mostly a proof of concept, thus many advanced training techniques are not included (e.g. Rainbow DQN).
Results. Experimental results on all tested environments are shown in Figure 3. We can see that our proposed transfer method learns significantly better than the single-task learner, and also outperforms all baselines in the challenging target tasks. Our transfer method outperforms Auxiliary since it transfers dynamics model from the source task instead of learning it from scratch, and outperforms Fine-tine since it regularizes the challenging encoder learning with a model-based regularizer.
The Time-aligned method, although requires additional pre-training that is not shown in the figures, does not work better than Single in most environments, because the time-based alignment assumption may not hold as discussed in Appendix E.1.1. In some environments (e.g. Hopper, Walker2d, 3DBall), our transfer algorithm even achieves better asymptotic performance than the source-task policy, which suggests that our method can be used for improving the policy with incremental observation design. To the best of our knowledge, we are the first to achieve effective knowledge transfer from a vector-input environment to a pixel-input environment without any pre-defined mappings.
Ablation Study and Hyper-parameter Test. To verify the effectiveness of proposed transfer method, we conduct ablation study and compare our method with its two variants: only transferring the transition model P̂ and only transferring the reward model R̂. Figure 4 shows the comparison in HalfCheetah, and Appendix E.2 demonstrates more results. We find that all the variants of our method can make some improvements, which suggests that transferring P̂ and R̂ are both effective designs for accelerating the target task learning. Figure 6 in Appendix E.2 shows another ablation study where we investigate different selections of model regularizers and policy heads. In Algorithm 2, a hyper-parameter λ is needed to con-
trol the weight of the transferred model-based regularizer. Figure 7 in Appendix E.2 shows that, for a wide range of λ’s, the agent consistently outperforms the single-task learner.
Potential Limitations and Solutions. As Figure 3 shows, in some environments such as HalfCheetah, our transfer algorithm significantly outperforms baselines without transfer. But in Walker2d, the improvement is less significant, although transferring is still better than not transferring. This phenomenon is common in model-based learning (Nagabandi et al., 2018), as state predicting in Walker2d is harder than that in HalfCheetah due to the complexity of the dynamics. Therefore, we suggest using our method to transfer when the learned models (P̂ , R̂) in the source task are relatively good (error is low). More techniques of improving model-based learning, such as bisimulation (Zhang et al., 2020b; Castro, 2020), can be applied to further improve the transfer performance.

heading: 7 CONCLUSION
text: In this paper, we identify and propose a solution to an important but rarely studied problem: transferring knowledge between tasks with drastically different observation spaces where inter-task mappings are not available. We propose to learn a latent dynamics model in the source task and transfer the model to the target task to facilitate representation learning. Theoretical analysis and empirical study justify the effectiveness of the proposed algorithm.

heading: ETHICS STATEMENT
text: Transfer learning aims to apply previously learned experience to new tasks to improve learning efficiency, which is becoming more and more important nowadays for training intelligent agents in complex systems. This paper focuses on a practical but rarely studied transfer learning scenario, where the observation feature space of an RL environment is subject to drastic changes. Driven by theoretical analysis on representation learning and its relation to latent dynamics learning, we propose a novel algorithm that transfers knowledge between tasks with totally different observation spaces, without any prior knowledge of an inter-task mapping.
This work can benefit many applications as suggested by the examples below. (1) In many real-life environments where deep RL is used (e.g. navigating in a building), the underlying dynamics (e.g. the structure of the building) are usually fixed, but what features the observation space has is designed by human developers (e.g. what sensors are installed) and thus may change frequently during the development. When the agent gets equipped with better sensors, our algorithm makes it possible to reuse previously learned models when learning with the new sensors. (2) An agent usually learns better with a compact observation space (e.g. a low-dimensional vector space containing its location and the goal’s location) than a rich/noisy observation space (e.g. an image containing the goal). However, a compact observation is usually more difficult to construct in practice as it may require expert knowledge and human work. In this case, one can extract compact observations in a few samples and pre-train a policy with our Algorithm 1, then train the agent in the real environment with rich observations with our Algorithm 2 using the learned dynamics models. Our experiment in Figure 3 shows that the learning efficiency in the rich-observation environment can be significantly improved with our proposed transfer method.

heading: REPRODUCIBILITY STATEMENT
text: For theoretical results, we provide concrete proofs in Appendix C and Appendix D. For empirical results, we illustrate implementation details in Appendix E. The source code and running instructions are provided in the supplementary materials.

heading: A ADDITIONAL PRELIMINARY KNOWLEDGE
text: For any policy π, its Q value Qπ is the unique fixed point of the Bellman operator
(T πQ)(o, a) = Eo′∼P (o,a),a′∼π(o′)[R(o, a) + γQ(o′, a′)] (4)
The optimal policy can be found by policy iteration (Howard, 1960), where one starts from an initial policy π0 and repeats policy evaluation and policy improvement. More specifically, at iteration k, the algorithm evaluates Qπk via Equation (4), then improves the policy by πk+1(o) := argmaxa∈AQπk(o, a),∀o ∈ O. It is well-known that the policy iteration algorithm converges to the optimal policy under mild conditions (Puterman, 2014). When the dynamics P and R are unknown, reinforcement learning algorithms use interaction samples from the environment to approximately solve Q̂πk . Prior works (Bertsekas & Tsitsiklis, 1996; Munos, 2005) have shown that if the approximation error is bounded by a small constant, the performance of πk as k → ∞ is guaranteed to be close to the optimal policy value Qπ∗ , which we also denote as Q∗.

heading: B ADDITIONAL DISCUSSION FOR REPRESENTATION SUFFICIENCY
text: Good Representation for A Fixed Policy The following definition characterizes the sufficiency of a representation mapping in terms of evaluating a fixed policy.
Definition 9 (Sufficient Representation for A Fixed Policy). A representation mapping φ is sufficient for a policy π w.r.t. approximation operator Hφ iff HφV π = V π . More generally, for a constant ≥ 0, Φ is -sufficient for π iff ‖HφV π − V π‖ ≤ .
Remarks. (1) If o1, o2 ∈ O have different values under π, a good representation should be able to distinguish them, i.e., φ∗(o1) 6= φ∗(o2). (2) The approximation operator Hφ is linear if HφV = ProjΦ(V ) where ProjΦ(·) denotes the orthogonal projection to the subspace spanned by the basis functions of 〈φ1, φ2, · · · , φd〉. Model Sufficiency over Policies Induces Linear Learning Sufficiency It can be found from Definition 3 that φ is sufficient as long as it representsQπ for all π ∈ ΠDφ . Fitting various value functions to improve representation quality is proposed by some prior works Bellemare et al. (2019); Dabney et al. (2020) and shown to be effective. However, learning and fitting many policy values could be computationally expensive, and is not easy to be applied to transfer learning between tasks with different observation spaces. Can we regularize the representation with the latent dynamics instead of policy values? Proposition 10 below shows that if Φ is linearly sufficient for all dynamics pairs (Pπ, Rπ) induced by policies in ΠDφ and the dynamics pairs associated with all actions, then Φ is linearly sufficient for learning.
For notation simplicity, assume the state space is finite. Then, let (Pπ, Rπ) be the transition matrix and reward vector induced by policy π, i.e., Pπ[i, j] = Ea∼π(oi)[P (oj |oi, a)], Rπ[i] = Ea∼π(oi)[R(oi, a)]. Similarly, let Pa[i, j] = P (oj |oi, a), Ra[i] = R(oi, a). We let Φ denote the representation matrix, where the i-th row of Φ refers to the feature of the i-the observation.
Proposition 10 (Linear Sufficiency Induced by Policy-based Model Sufficiency). For representation Φ, if for all π ∈ ΠDφ , a ∈ A, there exist P̂π, R̂π, P̂a, R̂a such that ΦP̂π = PπΦ, ΦR̂π = Rπ , ΦP̂a = PaΦ,ΦR̂a = Ra, i.e. Φ is linearly sufficient both policy-based dynamics and policyindependent dynamics models, i.e., then Φ is linearly sufficient for a task M w.r.t. approximation operatorHφ.
Proposition 10 proven in Appendix C.3 suggests that we can let representation fit (Pπ, Rπ) for many different π’s. However, it could be computationally intractable since the policy space is large. More importantly, it is hard to store and transfer a large number of dynamics models for all (P̂π, R̂π). In our Proposition 6, we show that learning sufficiency can be induced by policy-independent model
sufficiency, which is much simpler as there is no need to learn and store (P̂π, R̂π) for many policies. As a trade-off, the policy-independent model induces non-linear sufficiency instead of linear sufficiency, requiring a more expressive approximation operator.
Latent MDP induced by Representation We follow the analysis by Gelada et al. (2019) and define a new MDP under the representation mapping φ: M̃ = 〈Õ,A, P̃ , R̃, γ〉, where for all o ∈ O, φ(o) ∈ Õ, P̃ (φ(o), a) = P̂aφ(o), R̃(φ(o), a) = R̂aφ(o). Let Ṽ denote the value function in M̃ , and let π̃ denote a policy in M̃ . We make the following mild assumption
Assumption 11. There exists a constant Kφ,V such that |Ṽπ̃(φ(o1))− Ṽπ̃(φ(o2))| ≤ Kφ,V ‖φ(o1)− φ(o2)‖,∀π̃ : Õ → A, o1, o2 ∈ O.
This assumption is mild as any MDP with bounded reward has bounded value functions.

heading: C TECHNICAL PROOFS

heading: C.1 PROOF OF LEMMA 4
text: Proof of Lemma 4. We first show that policy iteration with approximation operatorHφ starting from a policy π0 ∈ ΠDφ generates a sequence of policies that are in ΠDφ . That is, for all πk, and any o1, o2 ∈ O, πk(o1) = πk(o2) if φ(o1) = φ(o2). We prove this claim by induction. Base case: when k = 0, π0 ∈ ΠDφ .
Inductive step: assume πk ∈ ΠDφ for k ≥ 0, then for iteration k + 1, we know that πk+1(o) := argmaxaQk(o, a) (5) where Qk = HφQπk . Based on the definition of Hφ, Qk(o, a) = f(φ(o); θa) for some θa. Hence, if o1 and o2 have the same representation, Qk(o1, ·) and Qk(o2, ·) are equal. As a result, πk+1(o1) and πk+1(o2) are equal, so πk+1 ∈ ΠDφ .
Next we prove the error bound in Lemma 4. We start by restating the error bounds for approximate policy iteration by Bertsekas & Tsitsiklis (1996):
limsupk→∞‖V ∗ − V πk‖∞ ≤ 2γ
(1− γ)2 supk‖Vk − V πk‖∞ (6)
where πk is the policy in the k-th iteration. Then we extend the above result to the action value Q.
For any πk during the policy iteration (as proven above, πk ∈ ΠDφ ), if φ is -sufficient for M as defined in Definition 3 with `∞ norm, then we have ‖Qk − Qπk‖∞ ≤ . That is, ∀o ∈ O, a ∈ A, |Qk(o, a)−Qπk(o, a)| ≤ . Therefore, ∀o ∈ O,
|Vk(o)− Vπk(o)| = | ∑ a∈A π(a|o)(Qk(o, a)−Qπk(o, a))| ≤ (7)
On the other hand, we can derive ‖Q∗ −Qπk‖∞ = maxo,a |Q ∗(o, a)−Qπk(o, a)| (8)
= max o,a |R(o, a) + γ ∑ o′∈O P (o′|o, a)V ∗(o′)−R(o, a)− γ ∑ o′∈O P (o′|o, a)Vπk(o′)| (9) = γ‖V ∗ − Vπk‖∞ (10)
Combining Equation (7) and (10), we obtain
limsupk→∞‖Q∗ −Qπk‖∞ ≤ 2γ2
(1− γ)2 . (11)

heading: C.2 PROOF OF PROPOSITION 6
text: Proof of Proposition 6. Given that P is deterministic for o ∈ O, a ∈ A, we slightly abuse notation and let o′ = P (o, a) = Pa(o) if P (o′|o, a) = 1. If φ is sufficient for the dynamics models, i.e. ∀o ∈ O, a ∈ A, P̂aφ(o) = φ(Pa(o)), R̂aφ(o) = Ra(o). Then, we can define a new MDP M̃ = 〈Õ,A, P̃ , R̃, γ〉, where for all o ∈ O, φ(o) ∈ Õ, and P̃ (φ(o), a) = P̂aφ(o) = φ(P (o, a)), R̃(φ(o), a) = R̂aφ(o) = R(o, a).
Any policy π ∈ ΠDφ , based on the definition of ΠDφ , can be written as π̃◦φ, where π̃ is a deterministic policy in M̃ . Next, we show that for all o ∈ O, a ∈ A, Qπ(o) = Q̃π̃(φ(o)). By definition of the Q value, we know
Qπ(o, a) = Eπ,P [ ∞∑ t=0 γtR(ot, at)|o0 = o, a0 = a] (12)
Q̃π̃(φ(o)) = Eπ̃,P̃ [ ∞∑ t=0 γtR̃(õt, ãt)|õ0 = φ(o), ã0 = a] (13)
We claim that in the above equations, õt = φ(ot), ãt = at, for all t ≥ 0. We prove the claim by induction.
When t = 0, the claim holds as õ0 = φ(o0) = φ(o), ã0 = a0 = a.
Then, with inductive hypothesis that õt = φ(ot), ãt = at, we show the claim holds for t+ 1:
Action: at+1 = π(ot+1) = π̃(φ(ot+1)) = ãt+1.
State: õt+1 = P̃ (õt, at) = P̃ (φ(ot), at) = φ(P (ot, at)) = φ(ot+1).
Hence, we have shown õt = φ(ot), ãt = at, for all t ≥ 0, then the reward in the t-th step of Equation (12) and (13) are the same, as R̃(õt, ãt) = R̃(φ(ot), at) = R(ot, at). Therefore, Qπ(o) = Q̃π̃(φ(o)).
Therefore, for any π ∈ ΠDφ , its action value can be represented by Q̃π̃ ◦ φ. Therefore, φ is sufficient for learning in M .
Next, we see whether φ is linearly sufficient for learning the task.
Consider an arbitrary policy π ∈ ΠDφ . Without loss of generality, suppose Rπ is linearly represented by φ, i.e. Ra(o, a) = φ(o)>R̂a, then we have
Rπ(o) = ∑ a∈A π(a|φ(o))Ra(o, a)
= ∑ a∈A π(a|φ(o))φ(o)>R̂a
= 〈π(φ(o)), R̂>φ(o)〉 = 〈R̂π(φ(o)), φ(o)〉
where R̂ := [R̂a1 ; R̂a1 ; · · · ; R̂a|A| ]. We can find that unless π always takes the same action for all input states, φ is not guaranteed to linearly encode Rπ .
Similarly, for Pπ , suppose Pa(·|o, a) = φ(o)>P̂a we have φ(Pπ(o)) = ∑ a∈A π(a|φ(o))Pa(·|o, a)
= ∑ a∈A π(a|φ(o))φ(o)>P̂a
= P̂ (π(φ(o)), φ(o), I)
where P̂ := [P̂a1 ; P̂a1 ; · · · ; P̂a|A| ] is an |A| × d × d tensor, and P̂ (·, ·, ·) denotes the multi-linear operation. Hence, if π takes different actions in different states, φ may not linearly encode the transition, either.
Therefore, φ is not guaranteed to linearly encode Rπ and Pπ , and thus is not guaranteed to linearly encode Vπ and Qπ .

heading: C.3 PROOF OF PROPOSITION 10
text: Proof of Proposition 10. We first show that for any π ∈ Π, if φ is linearly sufficient for (Pπ, Rπ), then there exists a vector ω ∈ Rk such that Vπ = V̂π = Φω. Since Φ is linearly sufficient for Pπ and Rπ , we have ΦP̂π = PπΦ and ΦR̂π = Rπ for some P̂π and R̂π . Let ω = (I − γP̂π)†R̂π , then the Bellman error of V̂π = Φω can be computed as
Rπ + γPπV̂π − V̂π = Rπ + γPπΦω − Φω = ΦR̂π + γΦP̂πω − Φω = Φ(R̂π − (I − γP̂π)(I − γP̂π)†R̂π) = Φ(R̂π − R̂π) = 0
Therefore, V̂π is a fixed point of the Bellman operator T π , and thus equal to Vπ .
Next, as we know that Qπ(·, a) = Ra + γ〈Pa, Vπ〉, and ΦP̂a = PaΦ,ΦR̂a = Ra, we can obtain Qπ(·, a) = Ra + γ〈Pa, Vπ〉
= ΦR̂a + PaΦω
= ΦR̂a + ΦP̂aω
= Φ(R̂a + P̂aω)
Therefore, for any π ∈ ΠDφ , Qπ can be represented by a linear approximation operator on Φ, and thus Φ is linearly sufficient for learning.

heading: C.4 PROOF OF THEOREM 7
text: Proof of Theorem 7. Lemma 2 in Gelada et al. (2019) is based on one policy in the induced MDP and bounded model errors. We can replace the Wasserstein distanceW(φP (·|o, a), P̃ (·|φ(o), a)) by the Euclidean distance ‖φP (o, a), P̃ (φ(o), a)‖ as we focus on deterministic transitions. For any policy π ∈ ΠDφ that can be written as π̃ ◦ φ, we have
|Qπ(o, a)− Q̃π̃(φ(o), a)| ≤ R + γKφ,V P
1− γ (14)
Therefore, φ is (1− γ)−1( R + γKφ,V P )-sufficient for learning M . Combined with Lemma 4, we can obtain the bound in Theorem 7.

heading: C.5 PROOF OF THEOREM 8
text: Proof of Theorem 8. First of all, if there exists φ(T ) satisfying P̂a(φ(oi)) = P (T )a [i]Φ (T ), R̂a(φ(oi)) = R(T )a [i], ∀oi ∈ O(T ), then it is sufficient for the dynamics of the target task, and thus sufficient for
learning the target task as stated in Proposition 6. Therefore, our focus is to show the existence of such a representation.
As φ(S) is sufficient for P (S)a and R (S) a for all a ∈ A, we have
P̂a(φ (S)(o(S))) = P (S)(o(S), a)Φ(S) = φ(S)(P (S)(o(S), a)) (15)
R̂a(φ (S)(o(S))) = R(S)(o(S), a) (16)
where we let P (S)(o(S), a) denote the next state of (o(S), a), given that P is deterministic.
Based on Assumption 1, we know that there exists a function f such that ∀o(T ) ∈ O(T ), f(o(T )) ∈ O(S), and f(P (T )(o(T ), a)) = P (S)(f(o(T )), a), R(T )(o(T ), a) = R(S)(f(o(T )), a). Hence, we can obtain
P̂a(φ (S)(f(o(T )))) = φ(S)(P (S)(f(o(T )), a)) = φ(S)(f(P (T )(o(T ), a))) (17)
R̂a(φ (S)(f(o(T )))) = R(S)(f(o(T )), a) = R(T )(o(T ), a) (18)
Let φ̂(T ) := φ(S) ◦ f , then we get P̂a(φ̂ (T )(o(T ))) = φ̂(T )(P (T )(o(T ), a)) (19)
R̂a(φ̂ (T )(o(T ))) = R(T )(o(T ), a) (20)
Therefore, φ̂(T ) is a feasible solution satisfying model sufficiency in the target task, and thus is sufficient for learning.
Theorem 8 holds since we have shown (1) all feasible solutions to Φ(T )P̂a = P (T )a Φ and Φ (T )R̂a = R(T )a are sufficient for learning in M (T ), and (2) there exists at least one feasible solution to Φ(T )P̂a =

heading: P (T )a Φ and Φ
text: (T )R̂a = R (T ) a .

heading: D REPRESENTATION LEARNING FOR APPROXIMATED VALUE ITERATION
text: Now we illustrate how our transfer algorithm and the proposed model-based regularization work for approximated value iteration. We focus on the case where the reward function R(o, a) ≥ 0 for all o ∈ O and a ∈ A. Preliminaries The bases of value iteration is the Bellman optimality operator T ∗. For the value function, we have
T ∗V (o) = max a∈A [R(o, a) + γ ∑ o∈O P (o′|o, a)V (o′)] (21)
For the Q function, we have T ∗Q(o, a) = R(o, a) + γ ∑ o∈O P (o′|o, a) max a′∈A Q(o′, a′), (22) where we slightly abuse notation and use T ∗ for both the value function and the Q function when there is no ambiguity.
Starting from some initial Q0 (or V0) and iteratively applying T ∗, i.e., Qk+1 = T ∗Qk, Qk (or Vk) can finally converge to Q∗ or V ∗ when k → ∞, which is known as value iteration. When an approximation operatorH is used, the process is called approximate value iteration (AVI): Qk+1 = HφT ∗Qk. A prior work has shown the following asymptotic result: Lemma 12 (Approximate Value Iteration for Q). For a reinforcement learning algorithm based on value iteration with approximation operator H, if ‖HT ∗Qk − T ∗Qk‖∞ ≤ , for all Qk along the value iteration path, then we have
lim sup k→∞
‖V ∗ − Vπk‖ ≤ 2
(1− γ)2 . (23)
Value Iteration Learning with Given Representation Given a representation mapping φ, we aim to learn an approximation function h : φ(O) → R|A| such that Q̂(o, ·) = h(φ(o)) ≈ Q(o, ·). For notation simplicity, we further use h(φ(o), a) to denote the approximated Q value Q̂(o, a) for a ∈ A. We start from an initial h0 that has a uniform value c for all inputs, where c > 0 can be randomly selected. The initial approximation for Q value is Q̂0 = h0 ◦ φ. Then, at iteration
k > 0, we solve Q̂k = HφT ∗Q̂k−1 = hk ◦ φ, where hk := argminh‖h ◦ φ− T ∗Q̂k−1‖∞. We use a neural network (universal function approximator) to parameterize h, so the approximation error ‖h ◦ φ− T ∗Q̂k−1‖∞ depends on the representation quality of φ. Therefore, a representation mapping φ is -sufficient for learning with value iteration if ‖HφT ∗Qk− T ∗Qk‖∞ ≤ . Next, we identify the relationship between policy-independent model sufficiency and the learning sufficiency with value iteration methods.
Guaranteed Learning with Model-regularized Representation We first make the following assumption for the learned approximation function h. Assumption 13 (Lipschitz Value Approximation). There exists a constant Kφ,h, such that ∀k ≥ 0, o1, o2 ∈ O, a ∈ A, |hk(φ(o1), a)− hk(φ(o2), a)| ≤ Kφ,h‖φ(o1)− φ(o2)‖, (24) where hk is the approximation function in the k-th iteration.
Then, the following Theorem holds, which justifies that learning with model-regularized representation helps with value iteration learning.
Theorem 14. For an MDP M , if encoder φ satisfies maxo∈O,a∈A |R(o, a)− R̂a(φ(o))| ≤ R and maxo∈O,a∈A ‖Eo′∼P (·|o,a)φ(o′) − P̂a(φ(o))‖2 ≤ P for dynamics models (P̂a, R̂a)a∈A, then the approximated value iteration with approximation operatorHφ under Assumption 13 satisfies
lim sup k→∞
‖V ∗ − V πk‖∞ ≤ 2
(1− γ)2 ( R + γ PKφ,h). (25)
Proof of Theorem 14. Let hk be the approximation function in the k-th iteration. That is, the approximated Q function in the k-th iteration is Q̂k = hk ◦ φ. As the rewards of all state-action pairs are non-negative, we have hk(φ(o), a) ≥ 0.
Define a function ĥk+1 as ĥk+1(φ(o), a) = R̂a(φ(o)) + γ max
a′∈A hk(P̂a(φ(o)), a
′) (26)
Given that T ∗Q̂k(o, a) = R(o, a) + γEo′∼P (·|o,a)[max
a′∈A Q̂k(o
′, a′)] (27)
= R(o, a) + γEo′∼P (·|o,a)[max a′∈A
hk(φ(o ′), a′)], (28)
we have that for any o ∈ O, a ∈ A,∣∣∣ĥk+1(φ(o), a)− T ∗Q̂k(o, a)∣∣∣ (29) = ∣∣∣∣R̂a(φ(o)) + γ maxa′∈Ahk(P̂a(φ(o)), a′)− (R(o, a) + γEo′∼P (·|o,a)[maxa′∈Ahk(φ(o′), a′)]) ∣∣∣∣ (30)
≤ ∣∣∣R̂a(φ(o))−R(o, a)∣∣∣+ γ ∣∣∣∣maxa′∈Ahk(P̂a(φ(o)), a′)− Eo′∼P (·|o,a)[maxa′∈Ahk(φ(o′), a′)] ∣∣∣∣ (31) ≤ R + γEo′∼P (·|o,a)[ ∣∣∣∣maxa′∈Ahk(P̂a(φ(o)), a′)−maxa′∈Ahk(φ(o′), a′) ∣∣∣∣] (32)
≤ R + γEo′∼P (·|o,a) max a′∈A ∣∣∣hk(P̂a(φ(o)), a′)− hk(φ(o′), a′)∣∣∣ (33) ≤ R + γEo′∼P (·|o,a) max
a′∈A Kφ,h ∥∥∥P̂a(φ(o)− φ(o′)∥∥∥ (34) ≤ R + γ PKφ,h, (35)
where (33) is due to the non-negativity of hk, and (34) is due to Assumption 13.
Now we have shown that the constructed ĥk+1 satisfies
‖ĥk+1 ◦ φ− T ∗Q̂k‖∞ ≤ R + γ PKφ,h. (36)
According to the definition ofHφ, we obtain ‖hk+1 ◦ φ− T ∗Q̂k‖∞ ≤ ‖ĥk+1 ◦ φ− T ∗Q̂k‖∞ ≤ R + γ PKφ,h (37)
sinceHφ finds a hk+1 that minimizes the approximation error. Therefore, Theorem 14 follows by combining Inequality (37) and Lemma 12.

heading: E EXPERIMENT DETAILS AND ADDITIONAL RESULTS

heading: E.1 EXPERIMENT SETTING DETAILS

heading: E.1.1 BASELINES
text: • Single: A DQN or SAC learner on the target domain without any auxiliary tasks. • Auxiliary: On the target domain, the encoder φ(T ) is optimized based on the loss Lbase(φ (T ), π(T )) + λ [ LP (φ (T ); P̂ (T )) + LR(φ (T ); R̂(T )) ] . Compared with our transfer algo-
rithms which transfer the learned dynamics from source domain to the target domain, it learns the dynamics model (P̂ (T ), R̂(T )) on the target domain from scratch. Here we set λ to be the same as our transferred algorithm (values of λ are provided in Appendix E.1.3). The purpose of this baseline is to test whether the efficiency of our proposed transfer algorithms come from the transferred latent dynamics or from the auxiliary loss (or potentially both). • Fine-tune: To test whether our transfer algorithms benefit from loading the learned policy
head π(S), on the target domain, we load the weights of π(T ) from the trained source policy head π(S) and train the DQN or SAC agent without any auxiliary loss. • Time-aligned: Gupta et al. (2017) propose to learn aligned representations for two tasks,
under the assumption that the source-task agent and the target-task agent reach similar latent sates at the same time step, i.e. φ(T )(s(T )t ) = φ
(S)(s(S)t ). Note that this assumption is valid when the initial state is fixed and the transitions are all deterministic. Although in our setting, the agent can not learn both tasks simultaneously, we can adapt the idea of time-based alignment and encourage the target encoder to map target observations to the source representations happening at the same time step.
In our experiments, we storeN source trajectories { si0, a i 0, s i 1, a i 1, ..., }N i=1
collected during source task training. Then on the target domain, we first collectN trajectories following the same action as the one collected from the source domain. In other words, at time step t of the i-th trajectory, we take action ait. After the target trajectories are collected, we minimize the alignment loss Lalign(φ(T )) = E [( φ(T )(s(T )t )− φ(S)(s (S) t ) )2]
to enforce that observations from source and target domain at the same time-step have the same representations. In our experiments, we set N to be 10% of the training trajectories. (We also experimented with largerN ’, for example using all the training trajectories, but the differences are minor.) In terms the alignment loss, we optimize the loss for 1000 epochs with batch size equal to 256, where at each epoch we sample a batch of paired source and target observations and compute the alignment loss. After pre-training the target encoder, we load the weight into φ(T ) and resumes the normal DQN or SAC training. Our experimental results show that, although more training steps are given to the timealigned learner, it does not outperform the single-task learner, and sometimes fails to learn (e.g. in 3DBall). The main reason is that the time-based assumption does not hold in practice as initial states are usually randomly generated. Therefore, even though the agent exactly imitates the source-task policy at every step, the observations from source and target task do not necessarily match at every time-step. In environments with non-deterministic transitions, the state mismatch will be a more severe issue and may lead to an unreasonable encoder.

heading: E.1.2 ENVIRONMENTS

heading: Environment Settings in Vec-to-pixel Tasks
text: • CartPole: The source task is the same as the ordinary CartPole environment on Gym. For the pixel-input target task, we extract the screen of the environment which is of size
(400,600), and crop the pixel input to let the image be centered at the cart. The resulting observation has size (40,90) after cropping. We take the difference between two consecutive frames as the agent’s observation.
• Acrobot: The source task is the same as the ordinary Acrobot environment on Gym. For the pixel-input target task, we first extract the screen of the environment which is of size (150,150), and then down-sample the image to (40,40). We also take the difference between two consecutive frames as the agent’s observation.
• Cheetah-Run: The source task is the Cheetah Run Task provided by DeepMind Control Suite (DMC) (Tassa et al., 2018). For the target task, we use the image of size (84,84) rendered from the environment as the agent’s observation.
Environment Settings in More-sensor Tasks For the target task of MuJoCo environments, we add the center of the mass based inertia and velocity into the observations of the agent, concatenating them with the original observation on the source task. Consequently, in the target environments, the dimensionality of the observation space on target task become much larger than that of the source task. On Hopper, the dimensionality of the target observation is 91, whereas the the source observation space only has 11 dimensions. The dimensionalities of target tasks on HalfCheetah, Hopper and Walker are 145, 91, 145 respectively.
Environment Settings in Broken-sensor Tasks 3DBall is an example environment provided by the ML-Agents Toolkit (Juliani et al., 2018). In this task, the agent (a cube) is supposed to balance a ball on its top. At every step, the agent will be rewarded if the ball is still on its top. If the ball falls off, the episode will immediately end. The highest episodic return in this task is 100. There are two versions of this game, which only differ by their observation spaces. The simpler version (named 3DBall in the toolkit) has 8 observation features corresponding to the rotation of the agent cube, and the position and velocity of the ball. The harder version (named 3DBallHard in the toolkit) does not have access to the ball velocity, but observes a stack of 9 past frames, each of which corresponds to the rotation of the agent cube, and the position of the ball, resulting in 45 observation dimensions at every step. We regard 3DBall as the source task and 3DBallHard as the target task in our experiments.
E.1.3 IMPLEMENTATION OF BASE DRL ALGORITHMS AND HYPER-PARAMETER SETTINGS
Implementation of DQN To ensure that the base learning algorithm learns the pixel-input target tasks well, we follow the existing online codebases for pre-processing, architectures and hyperparameter settings in pixel CartPole1 and pixel Acrobot 2. On source domain, the DQN network has a 2-layer encoder and a 2-layer Q head of hidden size 64, and the representation dimension is set as 16. For pixel-input, the encoder has three convolution layers followed by a linear layer. The number of channels of the convolutional layers are equal to 16, 32, 32, respectively (kernel size=5 for all three layers). We use the Adam optimizer with learning rate 0.001 and β1, β2 = 0.9, 0.999. The target Q network is updated every 10 iterations. In CartPole, we use a replay buffer with size 10000. In the more challenging Acrobot, we use a prioritized replay buffer with size 100000.
Implementation of SAC For MuJoCo environments, we follow an elegant open-sourced SAC implementation3. The number of hidden units for all neural networks is 256. The actor has a twolayer encoder and a two-layer policy head. The two Q networks both have three linear layers. The activation function is ReLU and the learning rate is 3 · 10−4. We train the dynamics model and the reward model every 50k interactive steps in the source task. For the DMC environment CheetahRun, we follow the open-sourced SAC implementation with an autoencoder 4. The pixel encoder has three convolution layers and one linear layer. The number of channels for all convolutional layers is 32 and the kernel size is 3. For the 3DBall environment, as it can only be learned within the ML-Agents toolkit, we directly use the SAC implementation provided by the toolkit with the default hyperparameter settings.
1https://pytorch.org/tutorials/intermediate/reinforcement q learning.html 2https://github.comeyalbd2Deep RL Course 3https://github.com/pranz24/pytorch-soft-actor-critic 4https://github.com/denisyarats/pytorch sac ae
Implementation of Latent Dynamics Model Note that our goal is to learn a good representation by enforcing it predicting the latent dynamics, different from model-based RL (Hafner et al., 2019) that aims to learn accurate models for planning. Therefore, we let the dynamics models P̂ and R̂ be simple linear networks, so that the representation can be more informative in terms of representing dynamics and learning values/policies. For environments with discrete action spaces, we learn |A| linear transition networks and |A| linear reward models. For environments with continuous action spaces, we first learn an action encoder ψ : A → Rd with the same encoding size d as the state representation. Then, we learn a linear transition network and a linear reward network with P̂ (φ(o) ◦ ψ(a)) being the predicted next representation, and R̂(φ(o) ◦ ψ(a)) being the predicted reward, where ◦ denotes element-wise product. In practice, we find this implementation achieves good performance across many environments. In addition, note that due to the significant difference between source observation and target observation, the initial encoding scale could be very different in source and target tasks, making it hard for them to be regularized by the same dynamics model. Therefore, we normalize the output of both encoders to be a unit vector (l2 norm is 1), which remedies the potential mismatch in their scales.
Hyperparameter Settings for Transfer Learning In experiments, we find that it is better to set λ relatively large when the environment dynamics are simple and the dynamics model is of high quality. When the environment dynamics is complex, we choose to be more conservative and set λ to be smaller. Concretely, in CartPole, λ is set as 18; in 3DBall, λ is set as 10; in Acrobot, λ is set as 5; in the remaining MuJoCo environments where dynamics are more complicated, λ is set as 1. Although we use different λ’s in different environments based on domain knowledge, we find that different values of λ’s do not have much influence on the learning performance. Figure 7 provided in Appendix E.2 shows a test on the hyper-parameter λ, where we can see that our algorithm effectively transfers knowledge under various values of λ. Regarding the representation dimension, we set it to be smaller for simpler tasks, and larger for more complex tasks. In 3DBall, we set the encoding size to be 8; in CartPole, we set the encoding size as 16; in Acrobot, we set the encoding size as 32; in Cheetah-Run, we set the encoding size as 50; in MuJoCo tasks, we set the encoding size as 256. Again, we find that the feature size does not influence the performance too much. But based on the theoretical insights of learning minimal sufficient representation Achille & Soatto (2018), we believe that it is generally better to have a lower-dimensional representation while making sure it is sufficient for learning.

heading: E.2 ADDITIONAL EXPERIMENTAL RESULTS

heading: Ablation Study: Transferring Different Components
text: Figure 5 shows the ablation study of our method in continuous control tasks. We compare our method with the following variants: (1) learning auxiliary tasks without transfer, (2) only transferring transition models P̂ and (3) only transferring reward models R̂.
Compared with the single-task learning baseline (the blue curves), we find that all the variants of our method can make some improvements, which suggests that learning dynamics models as auxiliary tasks, transferring P̂ and R̂ are all effective designs for accelerating the target task learning. Finally, our method (the red curves) that combines the above components achieves the best performance, justifying the effectiveness of our transfer algorithm.
Sanity Check: Effectiveness of the Proposed Transfer Method We conduct another ablation study to evaluate each component of our algorithm in the CartPole environment as shown in Figure 6. We find that when transferring the dynamics models with only a linear value head (the green curve), the agent fails to learn a good policy as we analyzed in Section 4.3. If the dynamics models (P̂ , R̂) are randomly generated instead of being transferred from the source task (the orange curve), the agent does not learn, either. More importantly, if we learn dynamics models as auxiliary tasks in the target task without transferring them from the source (the purple curve), the agent learns a little better than a vanilla agent, but is worse than our proposed transfer algorithm. These empirical results have
verified our theoretical insights and shown the effectiveness of our algorithm design.

heading: Hyper-parameter Test
text: Figure 7 further visualizes how the hyperparameter λ (regularization weight) influences the transfer performance in the Vec-to-pixel CartPole environment. It can be found that the agent generally benefits from a larger λ, which suggests that the model-based regularization has a positive impact on the learning performance. For a wide range of λ’s, the agent always outperforms the learner without transfer (the learner with λ = 0). Therefore, our algorithm is not sensitive to the hyperparameter λ, and a larger λ is preferred to get better performance. In Appendix E.1.3, we have provided the λ selections for all experiments.

Review summaries:
The paper addresses the problem of adapting a policy to a novel representation of the environment. Different from most previous work the paper focuses on the problem of a change in the representation of the observation provided by the environment, i.e., assuming that the underlying dynamics P and R remains mostly similar. The idea is to learn source and target models of the environment jointly with the policy. This serves as a model-based regularization that guides policy and representation towards the learned models. This effectively leads to a learned representation that is shared for the policy for the source and the target environment representation.

This paper proposes to leverage models of the environment and rewards as regularizers to perform explicit transfer between different observation spaces in RL.  Being able to quickly adapt to different observations is quite a different problem setting than what the rest of the literature is tackling (and we could argue on how relevant it is, but let’s leave that aside). They provide some theoretical analysis of this setting, especially one interesting result on when their method can converge to a near-optimal solution through approximate policy iteration.  They tackle it using ideas very similar to the recent wave of work leveraging bisimulation metrics, hence novelty / comparison on this aspect will be important. They assess it on simple Mujoco control tasks, where their results are promising but perhaps still a bit early.

The paper considers the problem of RL environments where the observations space can change dramatically, e.g. from proprioceptive to pixel observations. The authors introduce various definitions for this setting and provide proofs regarding convergence (in the sense of Q-values converging to true values). Over 5 environments in continuous control, the authors demonstrate that their method enables transfer learning from proprioceptive to pixel observations and that such transfer learning can be beneficial.

This paper tackles the problem of observation space change for RL problems, i.e., the observation changes while the dynamics remain similar. To deal with this problem, the authors propose to learn a latent dynamics models based on encoded observations. When the observation space is changed, the learnt latent dynamics model can serve as a knowledge prior to transfer known dynamics information when learning with the new observation space.