Transfer RL across Observation Feature Spaces via Model-Based Regularization
Authors: Yanchao Sun
Ruijie Zheng
Xiyao Wang
Andrew E Cohen
Furong Huang
Venue: ICLR 2022 Poster (2022)
Keywords: transfer reinforcement learning
representation learning
observation space change
latent dynamics model

Abstract: In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task.

Published as a conference paper at ICLR 2022
TRANSFER RL ACROSS OBSERVATION FEATURE SPACES VIA MODEL-BASED REGULARIZATION
Yanchao Sun' Ruijie Zheng'
Xiyao Wang' Andrew Cohen
Furong Huang'
University of Maryland, College Park Unity Technologies
{ycs, zzheng12, xywang, furongh} umd.edu aandrew. aandrew.cohennemitty...
ABSTRACT
In many reinforcement learning (RL) applications, the observation space is spee- ified by human developers and restricted by physical realizations, and may thus subject to dramatic changes over time (c.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which inefficient in terms of computation and sample complexity. Following theoretical insights, we propose novel algorithm which extracts the latent-space dynamics the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation image- based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the effi- ciency and stability of learning in the target task.
INTRODUCTION
Deep Reinforcement Learning (DRL) has the potential to be used in many large-scale applications such as robotics, gaming and automotive. In these real-life scenarios, it is an essential ability for agents to utilize the knowledge learned in past tasks facilitate learning in unseen tasks, which is known as Transfer RL (TRL). Most existing TRL works (Taylor & Stone, 2009; Zhu et al., 2020) focus on tasks with the same state-action space but different dynamics/reward. However, these approaches do not apply to the case where the observation space changes significantly.
Observation change is common in practice as in the following scenarios. (1) Incremental environ- ment development. RL is used to train non-player characters (NPC) in games (Juliani et al., 2018), which may be frequently updated. When there are new scenes, characters, or obstacles added to the game, the agent's observation space will change accordingly. (2) Hardware upgrade/replaccmment For robots with sensory observations (Bohez et al., 2017), the observation space could change (e.g. from text to audio, from lidar to camera) as the sensor changes. (3) Restricted data access. In some RL applications (Ganesh et al., 2019), agent observation contains sensitive data (e.g. inventory) which may become unavailable in the future due to data restrictions. In these cases, the learner may have to discard the old policy and train a new policy from scratch, as the policy has a significantly different input space, even though the underlying dynamics are similar. But training an RL policy from scratch can be expensive and unstable. Therefore, there is a crucial need for technique that transfers knowledge across tasks with similar dynamics but different observation spaces.
Besides these existing common applications. there are more benefits of across- oorerv-observation transfer. For example, observations in real-world environments are usually rich and redundant, so that directly learning a policy is hard and expensive. If we can transfer knowledge from low-dimensional and informative vector observations (usually available in a simulator) to richer observations, the learning efficiency can be significantly improved. Therefore, an effective transfer learning method enables many novel and interesting applications, such as curriculum learning via observation design.
In this paper, we aim to fill the gap and propose new algorithm that can automatically transfer knowledge from the old environment to facilitate learning in a new environment with a (drastically)
"The work was done while the author was an intern at Unity Technologies.

---

Published as a conference paper at ICLR 2022
Observed
3
O
Goal: (4,3)
Me: (1,1)
(3(5)
same latent state
transfer learning
2
Source
O
(T)
1
2
3
4
Target
Representation Learning
Policy Learinng
Figure An example of the transfer problem with changed observation space. The source-task agent observes the x-y coordinates of itself and the goal, while the target-task agent observes top-down view/image of the whole maze. The two observation spaces are drastically different, hut the two tasks are structurally similar. Our goal is to transfer knowledge from the source task to accelerate learning in the target task, without knowing or learning any inter-task mapping.
different observation space. In order meet more practical needs, focus the challenging setting where the observation change is: (1) unpredictable (there is no prior knowledge about how the observations change), (2) drastic (the source and target tasks have significantly different observation feature spaces, e.g., vector to image), and (3) irretrievable (once the change happens, is impossible query the source task, that the agent can not interact with both environments simultaneously). Note that different from many prior works (Taylor et al., 2007; Mann & Choe, 2013), we not assume the knowledge of any inter-task mapping. That is, the agent does not know which new observation feature is corresponding the which old observation feature.
To remedy the above challenges and achieve knowledge transfer, we make key observation that, if only the observation features change, the source and target tasks share the same latent space and dynamics (e.g. Figure 1, Oi and O can be associated to the same latent state). Therefore, first disentangle representation learning from policy learning, and then accelerate the target-tast agent by regularizing the representation learning process with the latent dynamics model learned in the source task. We show by theoretical analysis and empirical evaluation that the target task can be learned more efficiently with our proposed transfer learning method than from scratch.
Summary of Contributions. (1) To the best of our knowledge, are the first diseuss the transfer problem where the source and target tasks have drastically different observation feature spaces, and there is no prior knowledge of an inter-task mapping. (2) We theoretically characterize what constitutes a "good representation" and analyze the sufficient conditions the representation should satisfy. (3) Theoretical analysis shows that a moodel-based regularizer enables efficient representation learning in the target task. Based on this, we propose novel algorithm that automatically transfers knowledge across observation representations. (4) Experiments in 7 environments show that our proposed algorithm significantly improves the learning performance of RL agents in the target task.
2 PRELIMINARIES AND BACKGROUND
Basic RL Notations. An RI. task can be modeled by Markov Decision Process (MDP) (Puterman, 2014), defined as a tuple M (O, A, P, R, 7), where O the state/observation space, A is the action space, P is the transition kernel, R is the reward function and 7 is the discount factor. At timestep i, the agent observes state or, takes action ag based on its policy π : O A(A)(where A(-) denotes the space of probability distributions), and receives reward r, = R(o,, a.). The environment then proceeds to the next state ot+1 P(-jot. az). The goal of an RL agent is to find policy in the policy space II with the highest cumulative reward, which is characterized by the value functions. The value of a policy æ for state o E O defined as V*((o) = E,,, E>= 2 2'relon = o). The Q value of a policy æ for state-action pair (o, a) O x A is defined Q" Ex
= o, ag = a). Appendix provides more background of RL.
Representation Learning in RL. Real-world applications usually have large observation spaces for which function approximation needed to learn the value or the policy. However, directly learning policy over the entire observation space could be difficult, information in the observation inputs. common solution is to map the large-scale observation into a smaller representation space via non-linear encoder (also called representation mapping)
there is usually redundant
2

---

Published as a conference paper at ICLR 2022
6 O Rd, where d is the representation dimension, and then learn the policy/value function over the representation space ((O). In DRL, the encoder and the policy/value are usually jointly learned.
PROBLEM SETUP: TRANSFER ACROSS DIFFERENT OBSERVATION SPACES
We aim to transfer knowledge learned from a souree MDP to a target MDP. whose observation spaces are different while dynamics are structurally similar. Denote the source MDP as M = (O A. pis: R 7). and the target MDP as A = (0 A. R R'm,,)) O can be significantly different, such as Otd) being low-dimensional vector space and O being a high-dimensional pixel space, which is challenging for policy transfer since the source target policy have different input shapes and would typically be very different architecturally.
Note that (18) and
In this work, as motivated in the Introduction, we focus on the setting wherein the dynamics ((P(; R and (Pm R) of the two MDPs between which transfer knowledge are defined on different observation spaces but share structural similarities. Specifically, we make the assumption that there exists a mapping between the source and target observation spaces such that the transition dynamics under the mapping in the target task share the same transition dynamics as in the source task. We formalize this in Assumption 1:
Assumption 1. There exists a function f O.0) + O15) such that Vo,
,0 o. E EMM, 11,,Y E A.
(o a) i(d (F(o))))(() a), R R
Remarks. (1) Assumption is mild as many real-world scenarios fall under this assumption. For instance, when upgrading the cameras of a patrol robot to have higher resolutions, such a mapping I can be a down-sampling function. (2) f is a general function without extra restrictions. f can be many-to-one mapping, i.c., more than one target observations can be related to the same observation in the source task. f can be non-surjective, i.c., there could exist source observations that do not correspond to any target observation.
Many prior works (Mann & Choe, 2013; Brys al., 2015) have similar assumptions, but require prior knowledge of such an inter-task mapping to achieve knowledge transfer. However, such a mapping might not be available in practice. As alternative, we propose a novel transfer algorithm in the next section that does not assume any prior knowledge of the mapping f. The proposed algorithm learns a latent representation of the observations and a dynamics model in this latent space, and then the dynamics model is transferred to speed up learning in the target task.
4 METHODOLOGY: TRANSFER WITH REGULARIZED REPRESENTATION
In this section, we first formally characterize "what a good representation is for RL" in Section 4.1, then introduce our proposed transfer algorithm based on representation regularization in Section 4.2, and next provide theoretical analysis of the algorithm in Section 4.3.
4.1 CHARACTERIZING CONDITIONS FOR GOOD REPRESENTATIONS
As discussed in Section 2, real-world applications usually have rich and redundant observations, where learning a good representation (Jaderberg et al., 2016; Dabney et al., 2020) is essential for efficiently finding an optimal policy. However, the properties that constitute good representation for an RL task are still an open question. Some prior works (Bellemare et al., 2019; Dabney et al., 2020; Gelada et al., 2019) have discussed the representation quality in DRL, but take different perspective and focus on characterizing the sufficient properties of representation for learning a task.
Given a representation mapping é. the Q value of any (o, a) O A can be approximately rep- resented by a function @(o), i.e., ((), = h()(o): 0.), where is a function parameterized by 0,. To study the relation between representation quality and approximation quality, we define an approximation operator H.g which finds the best Q-value approximation based on Φ. Formally, let θ denote the parameter space of function h E H. then Va E A. Ho,(oo,)) a) == h(o(o); 0z), where -- = E. ()(o( 8) - (()(o),or a)ll]. Such a function à can be realized by neural net- works universal function approximators (Hornik et al., 1989). Therefore, the value approximation error I|Q - 2I only depends on the representation quality, i.c., whether we can represent the Q value of any state function of the encoded state (b0).
3

---

Published as a conference paper at ICLR 2022
The quality of the encoder & is crucial for learning an accurate value function learning good policy. The ideal encoder o should discard irrelevant information in the raw observation but keep essential information. In supervised or self-supervised representation learning (Chen et al., 2020; Achille & Soatto, 2018), it is believed that a good representation ((X) of input X should con- tain minimal information of X which maintaining sufficient information for predicting the label Y. However, in RL, it is difficult to identify whether a representation is sufficient, since there is no label corresponding to each input. The focus of an agent is estimate the value of each input which is associated with some policy. Therefore, we point out that the representation quality in RL policy-dependens Below, we formally characterize the sufficiency of representation mapping in terms of fixed policy and learning task.
O,
Sufficiency for A Fixed Policy. If the agent is executing a fixed policy, and its goal is to estimate the expected future return from the environment, then representation is sufficient for the policy as long as it can encode the policy value V A formal definition provided by Definition 9 in Appendix B.
Sufficiency for Learning A Task. The goal of RL is to find optimal policy. Therefore, it is not adequate for the representation to only fit one policy. Intuitively, a representation mapping is sufficient for learning if we are able to find optimal policy over the representation space d(O), which requires multiple iterations of policy evaluation and policy improvement. Definition 2 below defines a set of "important" policies for learning with ((O).
Definition 2 (Encoded Deterministic Policies). For a given representation mapping Φ--), define an encoded deterministic policy set ITD as the set of policies that are deterministic and take the same actions for observations with the same representations. Formally,
no ( E
3# :
S.I. Vo O, <<)) #((()))),
(1)
where it is mapping from the representation space the action space.
A policy ı is in IT if it does not distinguish 0 and 02 when ((01) = Mo2). Therefore, TT can be regarded as deterministic policies that make decisions for encoded observations. Now, we define the concept of sufficient representation for learning in an MDP.
Definition 3 (Sufficient Representation for Learning). A representation mapping o is sufficient for task M w.r.t approximation operator Hyg if Q, Qx for all ng- Furthermore,
é is finearly-sufficient for learning M 30 s.t. Qx a) = ó(o) 0a Va A. d is e-sufficient for learning M if Hx Qx1 S. VT E 11
Definition 3 suggests that the representation is sufficient for learning a task as long as it is sufficient for policies in 1I-- Then, the lemma below justifies that a nearly sufficient representation can ensure that approximate policy iteration converges near-optimal solution. (See Appendix D) for analysis on approximate value iteration.)
Lemma 4 (Error Bound for Approximate Policy Iteration). If é is e-sufficient for task M (with los norm), then the approximated policy iteration with approximation operator H.s starting from any initial policy that is encoded by & (To E 112)) satisfies
2.²6
lim sup ||Q² - Q
(1 71
k
where Tek is the policy in the k-th iteration.
Lemma 4, proved in Appendix C, is extended from the error bound provided by Bertsekas & Tsit- siklis (1996), For simplicity. we consider the bound in exo, but tighter bounds can be derived with other norms (Munos, 2005), although a tighter bound is not the focus of this paper.
How Can We Learn A Sufficient Representation? So far we have provided a principle to define whether given representation sufficient for learning. DRI,, the representation is learned to- gether with the policy or value function using neural networks, but the quality of the representation may be poor (Dabney et al., 2020), which makes it hard for the agent to find an optimal policy. Based on Definition 3, a natural method to learn good representation is to let the representation fit as many policy values as possible as auxiliary tasks, which matches the ideas in other works. For ex- ample, Bellemare et al. (2019) propose to fit a set of representative policies (called adversarial value functions). Dabney et al. (2020) choose to fit the values of all past policies (along the value improve- ment path), which requires less computational resource. Different from these works that directly fit.
4

---

Published as a conference paper at ICLR 2022
Algorithm Source Task Learning
Require:
Regularization weight A; update frequency m for stable encoder. Initialize encoder e stable encoder Ris), policy x(,, transition prediction network P and
1:
reward prediction network R.
2: for 0, do
3: Take action O4 ~ (1''"('") (o)")) get next observation 0 and reward rt, store to buffer.
4: Sample a mini-batch (o,, Os To of from the buffer.
5: Update P and Rt using one-step gradient descent with V VpLp(0 P) and V A R), where Lp and LR are defined Equation (3).
6:
Update encoder and policy by min,(s) .810) Loas T (
7: if t m then Update the stable encoder (i:)) + dis).
Algorithm 2 Target Task Learning with Transferred Dynamics Models
Require: Regularization weight A; dynamics models P and it learned in the source task. 1: Initialize encoder D policy m!
2: for 0, 1.... do
3: Take action o, n. (6s"""" (oi")), get next observation oj41 and reward rt, store buffer.
4: Sample a mini-batch (os, ag Ti, / from the buffer.
5: Update encoder and policy by mingor (r() Lease ( ) L) R)),
where Lp and Lg are defined in Equation (3).
the value functions of multiple policies, in Section 4.2, propose to fit and transfer auxiliary policy-indepenulet dynamics model, which is an efficient way to achieve sufficient representation for learning and knowledge transfer, theoretically justified in Section 4.3.
4.2 ALGORITHM: LEARNING AND TRANSFERRING MODEL-BASER REGULARIZER
Our goal is use the knowledge learned in the source task learn a good representation in the target task, such that the agent learns the target task more easily than learning from scratch. Since we focus on developing generic transfer mechanism, the base learner can be any DRL algorithms. We use Lime to denote the loss function of the base learner.
As motivated Section 4.1, we propose learn policy-independent dynamics models for producing high-quality representations: (1) P which predicts the representation of the next state based on current state representation and action, and (2) Å which predicts the immediate reward based on current state representation and action. For batch of N transition samples {or, a, of, T4 J.N. define the transition loss and the reward loss as:
N
Lp P) P)=
Σ (P(o(o.). a.) o(0))) L)( N 1=1
1
Σ ()((la) - r;)² -r)
(3)
N
where o(o{)) denotes the representation the next state of with stop gradients. In order to a more diverse state distribution, transition samples are drawn from an off-policy buffer, which stores shuffled past trajectories.
The learning procedures for the source task and the target task are illustrated in Algorithm and Algo- rithm respectively. Figure depicts the architec- ture of the learning model for both source and tar- get tasks. coded observation and next observation. Given the current encoding and the action a, the dynamics models P and R return the predicted next encoding = P(2, a) and predicted reward A = R(z, a). Then the transition loss is the mean squared error (MSE) between z' and & in a batch; the reward loss the MSE between and in batch.
Encoder
Policy
x/ V
grideeee
Dynamics
Shared
Φ(0) and z' @(o') are the en-
R
Acroes
Tasks
Figure 2: The architecture of proposed method. P and R are learned in the source task, then trans- ferred the target task and fixed during training.
5

---

Published as a conference paper at ICLR 2022
In the source task (Algorithm 1): dynamics models P and R are learned by minimizing Lp and I.n which are computed based on recent copy of encoder called stable encoder (:s) (Line 5). The computation of the stable encoder is to help the dynamics models converge, as the actual encoder (69) changes at every step. Note that stable copy of the network is widely used in many DRL algorithms (e.g. the target network in DQN). which can be directly regarded as ((9) without maintaining an extra network. The actual encoder (59)) regularized by the auxiliary dynamics models P and R (Line 6).
dynamics model P and î are transferred from the source task and fixed during learning. Therefore, the learning of कt is regularized by static dynamics models, which leads to faster and more stable convergence than naively learning an auxiliary task.
In the target task (Algorithm 2):
Relation and Difference with Model-based RL and Bisimulation Metries. Learning dynamics model is common technique in model-based RL (Kipf et al., 2019; Grimm et al., 2020), whose goal is to learn an accurate world model and use the model for planning. The dynamics model could be learned on either raw observations or representations. In our framework, we also learn a dynamics model, but the model serves model-free base learner with Thsse Bisimulation methods (Castro, 2020; Zhang et al., 2020b) aim to approximate the bisimulation distances among states by learning dynamics models, whereas do not explicitly measure the distance among states. Note that we also do not require reconstruction loss that is common in literature (Lee al., 2019).
auxiliary task, and learning is still performed by the
4.3 THEORETICAL ANALYSIS: BENEFITS OF TRANSFERABLE DYNAMICS MODEL
The algorithms introduced in Section 4.2 consist of two designs: learning latent dynamics model as an auxiliary task, and transferring the dynamies model to the target task. In this section, we show theoretical justifications and practical advantages of our proposed method. We aim to answer the fol- lowing two questions: (1) How does learning an auxiliary dynamics model help with representation learning? ? (2) Is the auxiliary dynamics model transferable?
For notational simplicity, let Pa and Rg denote the transition and reward functions associated with action a E A. Note that P. and R. are independent of any policy. We then define the sufficiency of a representation mapping w.r.t. dynamics models as below.
Definition 5 (Policy-independen, Model Sufficiency). For an MDP M, a representation mapping sufficient for its dynamics (Pa, Ra)aEA Va A. there exists functions P. IRd + I2d and R. Rd + R such that Vo E O, Pa(e(o)) = Po)) (oo'), Rr (a(()) = R. (o).
Remarks. (1) ó is exactly sufficient for dynamics (Pa, RalaeA when the transition function P is deterministic. (2) If P is stochastic, but we have max.,a || Pa (o) Pa (x())) || S cp and max.,a (o) (*)))))) S <n, then is (ep; -fufsuffii for the dynamics of M.
Next we show by Proposition 6 and Theorem 7 that learning sufficiency can be achieved via ensuring model sufficiency.
Proposition 6 (Learning Sufficiency Induced by Policy-indepence Model Sufficiency). Consider an MDP M with deterministic transition function P and reward function R. If & is sufficient for (Pa, RA)AEA, then it is sufficient (but not necessarily linearly sufficient) for learning in M.
Proposition 6 shows that, if the transition is deterministic and the model errors Lp, LR are zero, then is exactly sufficient for learning. More generally, if the transition function P not deterministic, and model fitting is not perfect. the learned representation can still be nearly sufficient for learning as characterized by Theorem 7 below, which is extended from a variant of the value difference bound derived by Gelada et al. (2019). Proposition 6 and Theorem 7 justify that learning the latent dynamics model as an auxiliary task encourages the representation to be sufficient for learning. The model error I.p and L defined in Section 4.2 can indicate how good the representation is.
Theorem 7. For an MDP M, if representation mapping & is (Ep, (EP,ER)-suffiee sufficient for the dynamics of M, then approximate policy iteration with approximation operator Hi starting from any initial policy To TD satisfies
2n²
lim sup IQ - TTh S $11-2) 7)3 (cR Ko..).)
(4)
1-000
where K is an upper bound of the value Lipschitz constant defined in Appendix B.

---

Published as a conference paper at ICLR 2022
Transferring Model to Get Better Representation in Target. Although Proposition 6 shows that learning auxiliary dynamics models benefits representation learning, finding the optimal solution is non-trivial since one still has to learn P and R. Therefore, the main idea of our algorithm is to transfer the dynamics models P, ft from the source task to the target task, to ease the learning in the target task. Theorem 8 below guarantees that transferring the dynamics models is feasible. Our experimental result in Section 6 verifies that learning with transferred and fixed dynamics models outperforms learning with randomly initialized dynamics models.
Theorem 8 (Transferable Dynamics Models). Consider a source task Mis and a target task M with deterministic transition functions. Suppose (is) is sufficient for (Pas) R(a)) JaE' with func- tions Pa. Rig. then there exists a representation 6Thh satisfying Pa(o(o) = E B (o (o) Ra (á(0)) = -R (o), for all o E O and nThh is sufficient for learning in M
Theorem 8 shows that the learned latent dynamics models P, R are transferable from the source task to the target task. For simplicity, Theorem 8 focuses on exact sufficiency as in Proposition 6, but it can be easily extended to e-sufficiency if combined with Theorem 7. Proofs for Proposition 6, Theorem 7 and Theorem 8 are all provided in Appendix C.
Trade-off' between Approximation Complexity and Representation Complexity. As suggested by Proposition 6, fitting volicy-independent dynamics encourages the representation to be sufficient for learning. but not necessarily linearly sufficient. Therefore, we suggest using a non-linear pol- icy/value head following the representation to reduce the approximation error. Linear sufficiency can be achieved if ø is made linearly sufficient for Pa and R for all H2, where Pa and R, are transition and reward functions induced by policy (Proposition 10, Appendix B). However, using this method for transfer learning is expensive in terms of both computation and memory, as it re- quires to learn P= and R, for many different T and store these models for transferring the target task. Therefore, there is a trade-off between approximation complexity and representation com- plexity. Learning a linearly sufficient representation reduces the complexity of the approximation operator. But it requires more complexity in the representation itself as it has to satisfy much more constraints. To develop a practical and efficient transfer method, use slightly more complex approximation operator -non-linear policy head) while keeping the auxiliary task simple and easy to transfer across tasks. Please see Appendix B for more detailed discussion about linear sufficiency.
5 RELATED WORK
Transfer RL across Observation Feature Spaces. Transferring knowledge between tasks with different observation spaces has been studied for years. Many existing approaches(Taylor et al., 2007; Mann & Choe, 2013; Brys et al., 2015) require an explicit mapping between the source and target observation spaces, which may be hard to obtain in practice. Raiman et al. (2019) introduce network surgery that deals with the change in the input features by determining which components of a neural network model should be transferred and which require retraining. However, it requires knowledge of the input feature maps, and is not designed for drastic changes, e.g. vector to pixel. transfer learning algorithm that works for
Sun et al. (2020) propose a provably sample-efficice different observation spaces without knowing any inter-task mapping, but the algorithm is mainly designed for tabular RI. and model-based RL which uses the model to plan for a policy, differ- ent from our setting. Gupta et al. (2017) achieve transfer learning between two different tasks by learning an invariant feature space, with a key mase- alignment assumption. We empirically compared this method with our proposed transfer algorithm in Section 6. Our work also related to state abstraction in block MDPs, as studied by Zhang et al. (2020a). But the problem studied in Zhang et al. (2020a) is a multi-task setting where the agent aims learn generalizable abstract states from a series of tasks. Another related topic is domain adaptation in RI. (Higgins et al., 2017; Eysenbach et al., 2020; Zhang et al., 2020b), where the target observation space (e.g. real world) is different from the source observation (e.g. simulator). However, domain adaptation does not assume drastic observation changes (c.g. changed dimension). Moreover, the aim of domain adaptation is usually zero-shot generalization to new observations, thus prior knowledge few samples of the target domain is often needed (Eysenbach et al., 2020).
Representation Learning in RL. In environments with rich observations, representation learning is crucial for the efficiency of RL methods. Learning unsupervised auxiliary tasks (Jaderberg et al., 2016) is shown to be effective for learning a good representation. The relationship between learning
7

---

Published as a conference paper at ICLR 2022
policy- dolicy-dependent auxiliary tasks and learning good representations has been studied in some prior works (Bellemare et al., 2019; Dabney et al., 2020; Lyle et al., 2021), while our focus is to learn policy-independent auxiliary tasks to facilitate transfer learning. Using latent prediction models to regularize representation has been shown to be effective for various types of rich observations (Guo et al., 2020; Lee et al., 2019). Gelada al. (2019) theoretically justify that learning latent dy- namics model guarantees the quality of the learned representation, while we further characterize the relationship between representation and learning performance, and we utilize dynamics models to improve transfer learning. Zhang et al. (2020b) use a bisimulation metric to learn latent rep- resentations that are invariant task-irrelevant details in observation. As pointed out by Achille & Soatto (2018), invariant and sufficient representation is indeed minimal sufficient, so it is an interesting future direction to combine our method with bisimulation metric to learn minimal suf- ficient representations. There is also a line of work using contrastive learning to train an encoder for pixel observations (Srinivas et al., 2020; Yarats et al., 2021; Stooke et al., 2021), which usually pre-train encoder based on image samples using self-supervised learning. However, environment dynamics are usually not considered during pre-training. Our algorithm can be combined with these contrastive learning approaches to further improve learning performance in the target task.
6 EXPERIMENTAL EVALUATION
We empirically evaluate our transfer learning algorithm in various environments and multiple observation-chang scenarios. Detailed experiment setup and hyperparameters are in Appendix E.
Baselines. To verify the effectiveness of our proposed transfer learning method, we compare our transfer learning algorithm with 4 baselines: (1) Single: a single-task base learner. (2) Auxiliary: learns auxiliary models from scratch to regularize representation. (3) Fine-tune: loads and freezes the source policy head, and retrains an encoder in the target task. (4) Time-aligned (Gupta et al., 2017): supposes the target task and the source task proceed to the same latent state given the same action sequence, and pre-trains a target-task encoder with saved source-task trajectories. More de- tails of bascline implementations are in Appendix E.I.I.
Scenarios. As motivated in Section 1, there are many scenarios where one can benefit from transfer learning across observation feature spaces. We evaluate our proposed transfer algorithm in 7 envi- ronments that fit various scenarios, to simulate real-worle applications:
(1) Vec-to-pixed -ixell: a novel and challenging scenario. where the source task has low-dimensional vector observations and the target task has pixel observations. We use 3 vector-input environments Cart- Pole, Acrobot and Cheetah- as source tasks, and use the rendered image in the target task. (2) More-sensor: another challenging scenario where the target task has lot more sensors than the source task. We use 3 MuJoCo environments: HalfCheetah, Hopper and Walker2d, whose origi- nal observation dimensions are 17, 11 and 17, respectively. We add mass-based inertia and velocity (provided by MuJoCo's s API), resulting in 145, 91, 145 dimensions in the corresponding target tasks. an existing game 3DBall contained in the Unity ML-Agents Toolkit (Ju- liani et al., 2018), which has two different observation specifications that naturally fit our transfer setting: the source observation has 8 features containing the velocity of the ball; the target obser- vation does not have the ball's 's velocity, thus the agent has to stack the past 9 frames to infer the velocity. Please see Appendix 1.2 for more detailed descriptions of all the 7 environments.
(3) Broken-senson sensor::
Base DRL Learners. What we propose is a transfer learning mechanism that can be combined with any existing DRL methods. For environments with discrete action spaces (CartPole, Acrobot), we use the DQN algorithm (Mnih et al., 2015), while for environments with continuous action spaces (Cheetah-Rei Run, HalfCheetah, Hopper, Walker2d, 3DBall), we use the SAC algorithm (Haarnoja et al., 2018). To ensure a fair comparison, we use the same base DRL learner with the same hyperparam- eter settings for all tested methods, as detailed in Appendix 1.3. As common in prior works, our implementation of the RL algorithms is mostly proof of concept, thus many advanced training techniques are not included (c.g. Rainbow DQN).
Results. Experimental results all tested environments are shown in Figure 3. We can see that our proposed transfer method learns significantly better than the single-task learner, and also outper- forms all baselines in the challenging target tasks. Our transfer method outperforms Auxiliary since it transfers dynamics model from the source task instead of learning it from scratch, and outper- forms Fine-tine since it regularizes the challenging encoder learning with model- regularizer.
8

---

Published as a conference paper at ICLR 2022
Source (oracle)
Transfer (ours)
Single
Auxiliary
Fine tune
Time aligned
Vec-to-pixels CartPole
Vec-to-pixel: Acrobot
Vec-to-pixel: Cheetah- Run
150
100
800
200
600
RAMES
2000
+00
50-
400
5000
900
50
100
150
2001
250
0K:
2506:
500K
750%
10000
1250
UE
2516
500E
Sieps
More-sensor: HalfCheetah
More-sensor: Hopper
More- Walker2d
Broken-sensor: 3DBall
1.00
1.00
3.00
2.000
2,30
1.000
1,500
1
01
23.8
40.0%
3.80
1006
10055
Figure 3: Our proposed transfer method outperforms all baselines in target tasks over all tested scenarios. (The dashed green lines are the learning curves in source tasks.) Results are averaged over random seeds.
The Time-aligned method, although requires additional pre-training that is not shown in the figures, does not work better than Single in most environments, because the time-based alignment assump- tion may not hold as discussed in Appendix E.I.I. In some environments (c.g. Hopper, Walker2d, 3DBall), our transfer algorithm even achieves better asymptotic performance than the source-tase policy, which suggests that our method can be used for improving the policy with incremental obser- vation design. To the best of our knowledge, we the first achieve effective knowledge transfer from a vector-inpul environment to a pixel-inpul environment without any pre-defined mappings.
Ablation Study and Hyper-pane saree--prramete Test. To verify the effectiveness of proposed transfer method, we conduct abla- tion study and compare our method with its two variants: only transferring the transition model P and only transferring the reward model R. Figure 4 shows the comparison in HalfChee- tah, and Appendix E.2 demonstrates more results. We find that all the variants of our method can make some improvements, which suggests that transferring P and R are both effective designs for accelerating the target task learning. Figure 6 in Appendix E.2 shows another ablation study where we in- vestigate different selections of model regularizers and policy heads. In Algorithm 2, a hyper-parameter is needed to con- trol the weight of the transferred model-based regularizer. Figure 7 in Appendix E.2 shows that, for a wide range of X's, the agent consistently outperforms the single-task learner.
HalfCheenah
5,000
4,000
3,000
2,000
1,000
Thander Pindly
Traisfer only
Single
300k
500k
Stags
Figure 4: Ablation Study
Potential Limitations and Solutions. As Figure 3 shows, in some environments such as HalfChee- tah, our transfer algorithm significantly outperforms baselines without transfer. But in Walker2d, the improvement less significant, although transferring is still better than not transferring. This phenomenon is common in model-based learning (Nagabandi et al., 2018), as state predicting in Walker2d is harder than that in HalfCheetah due to the complexity of the dynamics. Therefore, we suggest using our method to transfer when the learned models (P, ft) the source task are rela- tively good (error is low). More techniques of improving model-based learning, such as bisimula- tion (Zhang et al., 2020b; Castro, 2020), can be applied to further improve the transfer performance.
CONCLUSION
In this paper, we identify and propose solution to an important but rarely studied problem: trans- ferring knowledge between tasks with drastically different observation spaces where inter-task map- pings not available. We propose leam latent dynamics model in the source task and transfer the model to the target task to facilitate representation learning. Theoretical analysis and empirical study justify the effectiveness of the proposed algorithm.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work is supported by Unity Technologies, National Science Foundation IIS-1850220 CRI Award 030742-00001, DOD-DARPA- Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD), and Adobe, Capital One and JP Morgan faculty fellow- ships.
ETHICS STATEMENT
Transfer learning aims to apply previously learned experience to new tasks to improve learning efficiency, which is becoming more and more important nowadays for training intelligent agents in complex systems. This paper focuses on practical but rarely studied transfer learning scenario, where the observation feature space of an RL environment subject to drastic changes. Driven by theoretical analysis representation learning and its relation to latent dynamics learning, we propose a novel algorithm that transfers knowledge between tasks with totally different observation spaces, without any prior knowledge of an inter-task mapping.
This work can benefit many applications as suggested by the examples below.
(1) In many real-life environments where deep RL is used (e.g. navigating in building), the under- lying dynamics (c.g. the structure of the building) are usually fixed, but what features the observation space has is designed by human developers (e.g. what sensors are installed) and thus may change frequently during the development. When the agent gets equipped with better sensors, our algorithm makes possible reuse previously learned models when learning with the new sensors.
(2) An agent usually learns better with a compact observation space (e.g. low-dimensional vector space containing its location and the goal's location) than a rich/noisy observation space (e.g. image containing the goal). However, a compact observation is usually more difficult to construct in practice as it may require expert knowledge and human work. In this case, one can extract compact observations in few samples and pre-train policy with our Algorithm 1, then train the agent in the real environment with rich observations with our Algorithm 2 using the learned dynamics models. Our experiment in Figure shows that the learning efficiency in the rich-observation environment can be significantly improved with our proposed transfer method.
REPRODUCIBILITY STATEMENT
For theoretical results, we provide concrete proofs in Appendix C and Appendix D. For empirical re- sults, we illustrate implementation details in Appendix E. The source code and running instructions are provided in the supplementary materials.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research,
1980, 2018.
Mare Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. Advances in neural information processing systems, 32:4358-4366, 2019.
D. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, Belmont, MA, 1996.
Steven Bohez, Tim Verbelen, Elias De Coninck, Bert Vankeirsbilck, Pieter Simoens, and Bart Dhoedt. Sensor fusion for robot control through deep reinforcement learning. In 2017 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 10.1109/RROS2201..2200
2017. doi:
Tim Brys, Anna Harutyunyan, Matthew Taylor, and Ann Nowé. Policy transfer using reward shaping. In AAMAS, pp. 181-188, 2015.
10

---

Published as a conference paper at ICLR 2022
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMIR, 2020.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and David Silver. The value- improvemee path: Towards better representations for reinforeement learning. CoRR, abs/2006.02243, 2020. URL. ht.ps:: / /awxiviv.vv/// org/ abs 2000 02243.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdi- nov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers, arXiv preprint arXiv:2006. 139/6, 2020.
Sumitra Ganesh, Nelson Vadori, Mengda Xu, Hua Zheng, Prashant Reddy, and Manuela Veloso. Reinforcement learning for market making a multi-agent dealer market. arXiv preprint arXiv:1911.05892 2019.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Oii Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pp. 2170-2179. PMLR, 2019.
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence prin- ciple for model-based reinforcement learning. arXiv preprint arXiv:201 1.00500 2020.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Basties Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap hatent-predictive representations for multi- task reinforcement learning. In International Conference Machine Learning, pp. 3875-3886. PMLR, 2020.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint 17Xiv:1703.02949, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0160) 2019.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In International Conference Machine Learning, pp. 1480-1490. PMLR, 2017.
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks uni- versal approximators. Neural Networks, 2(5):359-366, 1989. ISSN 0893-6080. doi: https: htwp:////00.1106/009900088880899 sci http://ttrtccetpe.ee http:///er.areee///////////0/000
URL https: //ww:////wwww/// sci www.weeiiiieieeeeeeeeoo rect...rr...... com/
Ronald A Howard. Dynamic programming and markov processes. 1960.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:161 055397, 2016.
Arthur Juliani, Vincent- Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy. Yuan Gao, Hunter Hemry, Marwan Mattar, et al. Unity: A general platform for intelli- gent agents. arXiv preprint arXiv: 1809. 2018.

---

Published as a conference paper at ICLR 2022
Thomas Kipf, Elise van der Pol. and Max Welling. Contrastive learning of structured world models. arxiv preprint 19X1:2791..12247, 2019.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. CoRR, abs/1907.00953, 2019. URL
http:://axtii wwtpaaawwwv..iiiaoaaaaraaaaaa hatp//bss199 00953.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, pp- 1-9. PMLR, 2021.
Timothy A Mann and Yoonsuck Choe. Directed exploration in reinforcement learning with trans- ferred knowledge. European Workshop on Reinforcement Learning, pp. 59-76. PMLR, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Rémi Munos. Error bounds for approximate value iteration. In Proceedings the National Confer- ence on Artificial Intelligence, volume 20, pp. 1006. Menlo Park, CA; Cambridge, MA; London; AÀAI Press; MIT Press; 1999, 2005.
the
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 7559-7566, 2018. doi:
10.1109//CA 84631889.4111..
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Jonathan Raiman, Susan Zhang, and Christy Dennison. Neural network surgery with sets. arXiv
preprint arXiv: 19/2.06719, 2019.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint XXi::2004.441:: 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning
from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
Yanchao Sun, Xiangyu Yin, and Furong Huang. Temple: Learning template of transitions for sample efficient multi-tase task rl. arXiv preprint arXiv:2002.06653 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 180: 100.00000000000, 2018
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for tem- poral difference learning. Journal of Machine Learning Research, 8(9), 2007.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. arXiv preprint 2rXiv:2102.1 /271, 2021.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference on Machine Learning, pp. 11214-11224. PMLR, 2020a.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020b.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprini arXiv:2009.0 008888, 2020.
12

---

Published as a conference paper at ICLR 2022
Appendix: Transfer RL across Observation Feature Spaces via Model-Based Regularization
A ADDITIONAL PRELIMINARY KNOWLEDGE
For any policy n, its value Q" is the unique fixed point of the Bellman operator
(5) The optimal policy can be found by policy iteration (Howard, 1960), where one starts from an initial policy To and repeats policy evaluation and policy improvement. More specifically, at it- eration k. the algorithm evaluates QT via Equation (5), then improves the policy by (x) a (o, a), Yo E O. It is well-known that the policy iteration algorithm converges to the optimal policy under mild conditions (Puterman, 2014). When the dynamics P and R are unknown, reinforcement learning algorithms use interaction samples from the environment to approximately solve 2m. Prior works (Bertsekas & Tsitsiklis, 1996; Munos, 2005) have shown that if the approx- imation error is bounded by a small constant, the performance of πT as k + is guaranteed be close to the optimal policy value Qr., which we also denote as Q"
(T)))oo,a) = Po'r.((o.a).' ((a)))))))a))) [ + 7Q(d', .d))
B ADDITIONAL DISCUSSION FOR REPRESENTATION SUFFICIENCY
Good Representation for A Fixed Policy We slightly abuse notation and use Hs to denote the approximation operator for state value function V O + R. similar to the approximation of as introduced in Section 4.1. The following definition characterizes the sufficiency of representation mapping in terms of evaluating a fixed policy.
Definition 9 (Sufficient Representation for A Fixed Policy). A representation mapping & is suf- ficient for a policy π w.z.t approximation operator Ho iff Hy V More generally, for a constant 0, & is c-sufficient for iff [HgV" V <.
Remarks. (1) If E O have different values under π, a good representation should be able to distinguish them, i.e., 6" (o1) # o² (02).
(2) The approximation operator Ho is linear if H = PVoj+(V) where Proje(-) denotes the orthogonal projection to the subspace spanned by the basis functions of (Ø1, "2,,". (9a)
Model Sufficiency over Policies Induces Linear Learning Sufficiency It can be found from Definition that o is sufficient as long as it represents Q, for all E 112. Fitting various value functions to improve representation quality is proposed by some prior works Bellemare al. (2019); Dabney et al. (2020) and shown to be effective. However, learning and fitting many policy values could be computationally expensive, and not easy to be applicd to transfer learning between tasks with different observation spaces. Can regularize the representation with the latent dynamics instead of policy values? Proposition 10 below shows that if is linearly sufficient for all dynamics pairs (Px, R2) induced by policies in no and the dynamics pairs associated with all actions, then is linearly sufficient for learning.
For notation simplicity, assume the state space finite. Then, let (Pr, Rz) be the transition matrix and reward vector induced by policy π, i.., Pa [i Ea-n((,) (o, jo,, a)). [ Ea-a(o;) R[o,, a)]. Similarly, let Pa = P(o,le lo;, a). Rali] = R(o;, a). We let & denote the representation matrix, where the i-th row of Φ refers to the feature of the i-the observation.
Proposition 10 (Linear Sufficiency Induced by Policy-based Model Sufficiency). For representation +, for all E Th a E A, there exist Pz, Ro, P.,, R. such that ΦP;; P. + tπt R.. P.P DR = R,, i.e. Tb is linearly sufficient both policy-based dynamics and policy- independent dynamics models, iee, then t is linearly sufficient for a task M w.r.l. approximation operator Ho
Proposition 10 proven in Appendix C.3 suggests that we can let representation fit (Px, R2)) for many different r's. However, it could be computationally intractable since the policy space is large. More
13

---

Published as a conference paper at ICLR 2022
importantly, it is not memory- store and transfer large number of dynamics models for all (Px, Rz)). In our Proposition 6, show that learning sufficiency can be induced by policy- independent model sufficiency, which is much simpler there is no need learn and store (P*, Rx) for many policies. As a trade-off, the polley-independent model induces non-linear sufficiency instead of linear sufficiency, requiring a more expressive approximation operator.
Latent MDP induced by Representation We follow the analysis by Gelada et al. (2019) and define new MDP under the representation mapping Φ: M = (O 7), where for all o E O, ((0) O. P(ó(o), a) Pa)(o), R(o(o), a) Ra (o). Let V denote the value function in M, and let A denote policy in M. We make the following mild assumption
Assumption 11. There exists a constant Kø.v such that (v(()) Va (d(02)) S Ka.vllo(o)) (02))l. Vi Ô + A. 01 02 E O.
This assumption mild as any MDP with bounded reward has bounded value functions.
C TECHNICAL PROOFS
C.I PROOF OF LEMMA 4
Proof of Lemma 4. We first show that policy iteration with approximation operator Hy starting from a policy To E D generates a sequence of policies that are in I That is, for all wk, and any 01.02 E O. (01) = Tx (02) if d(01) = 6(02). We prove this claim by induction. Base case: when & O, no 1.
Inductive step: assume T E & for l 2 0, then for iteration l: + 1, we know that k (o) == a (o,
(6)
where Q& HLp
Based on the definition of Ho. Q= a) f 0a) for some fa. Hence, if 01 and 02 have the same representation, (01.)) and Qx(0),)) are equal. As a result, (++1(01) and (T+1( are equal, so T .I
Next prove the error bound in Lemma 4. We start by restating the enror bounds for approximate policy iteration by Bertsekas & Tsitsiklis (1996):
27 (1->)> sup V/ V* |loc
limsup I|V* V** 1lx0
where mi the policy in the k-th iteration. Then we extend the above result to the action value Q.
For any Ty during the policy iteration (as proven above, тh E 11.), if ø is e-sufficient for M as defined in Definition with foo norm, then we have ||Q, 2n I e. That is, Vo E O e A, (o, a) Quz (o, a)l S Therefore, Vo O.
[V) - Vx, (o)) = Σ (aaooa (o,a))
(8)
aEA
On the other hand, we can derive
||Q" - QT, Ilee = max (o, Qr. (o)
(9)
max RRo, a) + Σ P(o'oo,a)V'(d) - R(o, a) - - Σ 0,a
P(o (o')l
O'E0
o'E0
(10)
= => Vr, Ilo
(11)
Combining Equation (8) and (11), we obtain
2n
limsup 1Q" - - /llc -
(12)
(1 - 7)
n
14

---

Published as a conference paper at ICLR 2022
C.2 PROOF OF PROPOSITION 6
Proof of Proposition 6. Given that P is deterministic for O,a E A, slightly abuse notation and let o' P(o,a) = Po( if a)) = 1. If ø is sufficient for the dynamics models, Le. Vo E 0, A, Pao(o) = (P)) Ra = Ro) Then, we can define new MDP where for all O. o(o) Ó. and P(o(o), Pao(o) = ((P(o.))) R(o(o), Rab(o) R)),a).
M
Any policy ग E 11 based on the definition of 11 can be written as FoØ, where 7 is a deterministic policy in M. Next, we show that for o O, E A. Qx(o) = (x(())).
By definition of the Q value, we know
Qu(o,a) ET
Y R(o,, ae) = 00 a|
(13)
t=0
(=((())) = E
7¹ RR( āz) = 6(0), ão = a|
(14)
t=0
We claim that in the above equations, o ((tr), ãi ae, for all 2 0. We prove the claim by induction.
When the claim holds as óu <(0o) Φ(0), āo ag = a.
Then, with inductive hypothesis that oN = o(oi), åi = we show the claim holds for t + 1:
Action: 0c+1 1()+1) F(((0++1)) ăe+1.
State: 0z+1 = P(or, az) P(o(or), at) (( a)) p(0++1).
Hence, we have shown o; = o(or), ā. a. for all t 2 O, then the reward in the t-th step of Equation (13) and (14) are the same, as R(ō ā,) = R(o(o,), a.) R(o, a.). Therefore, Q~ Q=((())).
Therefore, for any I1 its action value can be represented by Q+ d. Therefore, & is sufficient for learning in M.
Next, we show that @ is not necessarily linearly sufficient for learning the task.
Consider an arbitrary policy т E & Without loss of generality, suppose R2 is linearly represented
by Φ, i.c. Ro(o, a) = Φ(o) R. then we have
R2 (0) = Σ (o..a)
aeA
Σ (alb((o))b(o) ft.
aEA
= (π(o(o)). R o(o))
(Rm(o(o)), o(0))
where Ri [Ragi :... Raja J. We find that unless always takes the same action for all input states, Φ is not guaranteed to linearly encode Rg.
Similarly, for Pr. suppose Pa(-lo,a) a) = <(0) T Pa we have
(o) = Σ ((al6()))Pa(10.a)
aEA
Σ {(cl((0)))))) P.
aEA
=(T(6(0)))6(0):1
15

---

Published as a conference paper at ICLR 2022
where P : [Par: Pay: Pa is an |A] x d x d tensor, and P(...-) denotes the multi-linear operation. Hence, if и takes different actions in different states, & may not linearly encode the transition, either.
Therefore, ø is not guaranteed to linearly encode Rg and Pa. and thus is not guaranteed to linearly encode V and Q..
C.3 PROOF PROPOSITION 10
Proof of Proposition 10. We first show that for any т E II, Φ is linearly sufficient for (Px: Rw). then there exists a vector w E R such that V = V = Aw.
Since T is linearly sufficient for Px and R,, have p P, tb and कR, R, for some P, and RG. Let (T (P>) Rr, then the Bellman error of V-= aw can be computed
R₂ + TPpVV V/ R2 + 7P, tu
= +RR + +DDPpw dw
= =D(R (I -Pp=)(( - -7Px)t
= R,T)
0
Therefore, Vn is fixed point of the Bellman operator 7xx and thus equal to Va
Next, we know that Q+(-,a) R. + (Pa. V*), and $ Φ = PaT &R. = Ra, we can obtain Qx(;a)) Ra + 7 (Pa V*)
R. + PPaw
4R + P
= (R, + PPww)
Therefore, for any æ 11g. Qn can be linearly represented by 4, and thus is linearly sufficient for learning by definition.
C.4 PROOF OF THEOREM 7
Proof of Theorem 7. Lemma in Gelada et al. (2019) is based one policy in the induced MDP and bounded model errors. We replace the Wasserstein distance W( a). P(-l@(o), a)) by the Euclidean distance [| a), P(o(o), a) we focus on deterministic transitions.
For any policy π E II that can be written as o d, we have
(Qx(o, (o(o)) en KKy.er
(15)
1-7
Therefore, o is (1 - -11 +
for learning M.
Combined with Lemma 4, we can obtain the bound Theorem 7.
C.5 PROOF OF THEOREM 8
Proof of Theorem 8. First of all, there exists p09) satisfying Pa(á(o:)) = pmT (o(o)))) ( =
R (i), Vo: Om, then it is sufficient for the dynamies of the target task, and thus sufficient for
16

---

Published as a conference paper at ICLR 2022
learning the target task stated in Proposition 6. Therefore, our focus is to show the existence of such a representation.
As de is sufficient for p and R for all A. we have Pa (o p( (o (ol3) (8) a) 16)) d .))
(16)
(a = R. (o')) a)
(17)
where we let (o a) denote the next state of (o .), given that P is deterministic.
Based on Assumption
know that there exists function f such that Vo O ( O68, and (o a)) = (f(o") ),(o)))))) a), (o") a) (f(o") a). Hence, we can obtain
Pa Pa(o"((("')) "0 (flo") a)) のt (f(P (f (o"),
(18)
(a R (f(o") ). RTT(
(19)
Let 6Th) = v0 o then we get
Pa() = (P a)) (o( R (o
(00)
(21)
Therefore, 8:T) is feasible solution satisfying model sufficiency in the target task, and thus is sufficient for learning.
Theorem 8 holds since we have shown (1) all feasible solutions to क.)) Pa = Pa + and t Ra = R are sufficient for learning in M and (2) there exists at least one feasible solution to p P. P and कh R., = R
D REPRESENTATION LEARNING FOR APPROXIMATE VALUE ITERATION
Now we illustrate how our transfer algorithm and the proposed model-based regularization work for approximate value iteration. We focus on the case where the reward function R(o, a) 2 for all E O and E A.
Preliminaries function, we have
The bases of value iteration is the Bellman optimality operator T For the value
('V max[A(o,a)) a) + aEA
Σ
P(d'lo,a)V(o'))
(22)
oEO
For the Q function, we have
Q²)(o,)) R(o, a) + a(dlo,a) max
max Q(d
(23)
oEO
where we slightly abuse notation and use T for both the value function and the Q function when there is no ambiguity.
Starting from some initial Qo (or Vo) and iteratively applying T i.e. QK+1 = T*Qu. Qu (or Via) can finally converge to Q" or V when k - oo, which is known value iteration. When an approximation operator 7 is used, the process is called approximate value iteration (AVI): Q = H.T T*Qk. A prior work has shown the following asymptotic result:
Lemma 12 (Approximate Value Iteration for Q). For reinforcement learning algorithm based on value iteration with approximation operator H. if | - TxQll I c, for all Qk along the value iteration path, then have
2e
lim V VT (1-7)2 10000
(24)
Value Iteration Learning with Given Representation Given a representation mapping Φ, we aim learn an approximation function : ((O) - RIAI such that Q(o,) = h(((0)) as Q(o, -)- For notation simplicity, we further use h(((o), a) to denote the approximated Q value Q a) for A. We start from initial hu that has uniform value for all inputs, where > 0 can be randomly selected. The initial approximation for value is Qo ho Φ. Then, iteration
17

---

model sufficiency and Published as a conference paper at ICLR 2022
k > O, solve Q2 1 h2 o, where hyg := argmin, |h - - We use a neural network (universal function approximator) to parameterize h, so the approximation error ||/ a Φ - T @' 1100 depends on the representation quality of øb
Therefore, representation mapping Φ is c-sufficient for learning with value iteration if T Te I c. Next, we identify the relationship between the learning sufficiency with value iteration methods.
We first make the following as- Assumption 13 (Lipschitz Value Approximation). There exists constant Ka such that Vk 2 O, 01, 02 E O, a E A.
Guaranteed Learning with regulazegl Representation
sumption for the learned approximation function h.
( a) hu (o(2)))) & Ka 6(02)ll.
(25)
where by is the approximation function in the k-th iteration.
Then, the following Theorem holds, which justifies that learning with model-regularized represen- tation helps with value iteration learning.
Theorem 14. For an MDP M, if encoder do satisfies maxogO,aE R[o,, a) (b(())) S en and max.c0, [P(joa))) ((d') Pa(o()))l2 S ep for dynamics models (Pa, Ra)aEA then the approximated value iteration with approximation operator Ho under Assumption 13 satisfies
2
lim sup IV R-+00
V |lee
(1 (1- (ER + YepKo..).
(26)
Proof of Theorem 14. Let h2 be the approximation function in the k-th iteration. That is, the ap- proximated Q function in the k-th iteration is Q2 = hy Φb As the rewards of all state-action pairs are non-negative, we have h4 (o(o), a) 2 0.
Define a function hx+1 as
(6(0), = Ra(o(o)) + max hu (P
(27)
d'EA
Given that
T)) R(o,a) a) + BB. P()o,)) mmax Qu(o', a')]
(28)
d'EA
= = 7B. a'EA
h (o(')
(29)
have that for any E O,
(30)
Ra(o(o)) + m d'E
max h4 (o(o))) d') - (R(o, a) + +E P ((
[max (m(o'), a')l)
(31)
a'EA
$ Ra((o)) - R(o, a)
+
max hu (Pa((())) d') B (( a'EA
mmax (nax))((lo'),el
(32)
a'EAA
sen + T., P((oa))
max maxxuu (P(()((()))) - max ha (o(o'),
(33)
a'EA
a'EA
max Sen + T. P(loa))
h (Pa()(o))) hu(o(o'),aa)
(34)
d'EA
Sen + 7.Bo max Ko.h ( -bo0i
(35)
d'EA
SEA + KepKa.h
(36)
where (34) is due to the non-negativity of h4, and (35) is due to Assumption 13.
Now we have shown that the constructed h4+1 satisfies
| oØ- Qu +
(37)
According to the definition of Hg. we obtain
[ o Qxll llaets a Qxlllo S SEn + Ker
(38)
18

---

Published as a conference paper at ICLR 2022
since Hy finds h4+1 that minimizes the approximation enror.
Therefore, Theorem 14 follows by combining Incquality (38) and Lemma 12.
E EXPERIMENT DETAILS AND ADDITIONAL RESULTS
E. EXPERIMENT SETTING DETAILS
E.1.1 BASELINES
Single: DQN or SAC learner on the target domain without any auxiliary tasks. Auxiliary: On the target domain, the encoder di is optimized based the loss Lowe( ()) + > ( + L Rm))] Compared with our transfer algo- rithms which transfer the learned dynamics from source domain to the target domain, learns the dynamics model (Pm Rm)) on the target domain from seratch. Here we set À to be the same as our transferred algorithm (values of À are provided in Appendix E.1.3). The purpose of this baseline is to test whether the efficiency our proposed transfer algorithms come from the transferred latent dynamics or from the auxiliary loss (or potentially both).
Fine-tune: To test whether our transfer algorithms benefit from loading the learned policy head T19, on the target domain, we load the weights of x² from the trained source policy head π and train the DQN or SAC agent without any auxiliary loss.
Time-aligned:
Gupta al. (2017) propose learn aligned representations for two tasks, under the assumption that the source-task agent and the target-task agent reach similar latent sates at the same time step. i.e. (s" = (s Note that this assumption is valid when the initial state is fixed and the transitions are all deterministic. Although in our setting, the agent can not learn both tasks simultaneously, we can adapt the idea of time-based alignment and encourage the target encoder to map target observations to the source representations happening at the same time step.
In our experiments, we store N source trajectories { ab. $1 o). } collected during
N
source task training. Then on the target domain, we first collect N trajectories following the same action the one collected from the source domain. In other words, at time step of the i-th trajectory, we take action a/_ After the target trajectories are collected, we minimize
the alignment loss (
E (s("))
" )s)
to enforce that observations
from source and target domain the same time-step have the same representations.
In our experiments, we set N to be 10% of the training trajectories. (We also experimented with larger N', for example using all the training trajectories, but the differences are minor.) In terms the alignment loss, we optimize the loss for 1000 epochs with batch size equal to 256, where each epoch we sample batch of paired source and target observations and compute the alignment loss. After pre-training the target encoder, we load the weight into eTt and resumes the normal DQN or SAC training.
Our experimental results show that, although more training steps are given to the time- aligned learner, it does not outperform the single-task learner, and sometimes fails to learn (e.g. in 3DBall). The main reason is that the time-based assumption does not hold in practice as initial states are usually randomly generated. Therefore, even though the agent exactly imitates the source-task policy every step, the observations from source and target task do not necessarily match at every time-step. In environments with non-deterministic transitions, the state mismatch will be more severe issue and may lead an unreasonable encoder.
E.1.2 ENVIRONMENTS
Environment Settings in Vec-to-pixel Tasks
CartPole:
The source task is the same as the ordinary CartPole environment on Gym. For the pixel-input target task, we extract the screen of the environment which is of size
19

---

Published as a conference paper at ICLR 2022
(400,600), and crop the pixel input to let the image be centered at the cart. The resulting ob- servation has size (40,90) after cropping. We take the difference between two consecutive frames as the agent's 's observation.
Acrobot: The source task the same as the ordinary Acrobot environment on Gym. For the pixel-input target task, we first extract the screen of the environment which is of size (150, and then down-sample the image to (40,40). We also take the difference between two consecutive frames as the agent's ss observation.
The source task is the Cheetah Run Task provided by DeepMind Control Suite (DMC) (Tassa et al., 2018). For the target task, we use the image of size (84,84) rendered from the environment as the agent's 's observation.
Cheetah-Run:
Environment Settings in More-sensor Tasks For the target task of MuJoCo environments, add the center of the mass based inertia and velocity into the observations of the agent, concatenating them with the original observation the dimensionality of the observation space on target task become much larger than that of the source task. On Hopper, the dimensionality of the target observation is 91, whereas the the source observation space only has 11 dimensions. The dimensionalities of target tasks on HalfCheetah, Hopper and Walker are 145, 91, 145 respectively.
the source task. Consequently, in the target environments,
Environment Settings in Broken-sensor Tasks 3DBall is an example environment provided by the ML-Agents Toolkit (Juliani et al., 2018). In this task, the agent (a cube) is supposed balance a ball on its top. At every step, the agent will be rewarded if the ball is still on its top. If the ball falls off, the episode will immediately end. The highest episodic return in this task 100. There are two versions of this game, which only differ by their observation spaces. The simpler version (named 3DBall in the toolkit) has 8 observation features corresponding to the rotation of the agent cube, and the position and velocity of the ball. The harder version (named 3DBallHard in the toolkit) does not have access to the ball velocity, but observes a stack of 9 past frames, each of which corresponds to the rotation of the agent cube, and the position of the ball, resulting in 45 observation dimensions at every step. We regard 3DBall as the source task and 3DBallHard as the target task in our experiments.
E.1. IMPLEMENTATION OF BASE DRL ALGORITHMS AND
SETTINGS
Implementation of DQN To ensure that the base learning algorithm learns the pixel-inprn target tasks well, we follow the existing online codebases for pre-processing, architectures and hyperpa- rameter settings in pixel CartPole' and pixel Acrobot 2. On source domain, the DQN network has a 2-layer encoder and a 2-layer Q head of hidden size 64, and the representation dimension is set as 16. For pixel-input, the encoder has three convolution layers followed by linear layer. The number of channels of the convolutional layers are equal to 16, 32, 32, respectively (kernel size=5 for all three layers). We the Adam optimizer with learning rate 0.001 and B B2 0.9, 0.999. The target Q network is updated every 10 iterations. In CartPole, we use a replay buffer with size 10000. In the more challenging Acrobot, we use prioritized replay buffer with size 100000.
Implementation of SAC For MuloCo environments, we follow an elegant open-sourced SAC implementation', The number of hidden units for all neural networks is 256. The actor has two- layer encoder and a two-layer policy head. The two Q networks both have three linear layers. The activation function is ReLU and the learning rate is 3 10-4, We train the dynamics model and the reward model every 50k interactive steps in the source task. For the DMC environment Cheetah- Run, we follow the open-sourced SAC implementation with an autoencoder 4. The pixel encoder has three convolution layers and one linear layer. The number of channels for all convolutional layers is 32 and the kernel size is 3. For the 3DBall environment, as can only be learned within the ML-Agents toolkit, we directly use the SAC implementation provided by the toolkit with the default hyperparameter settings.
q learningg
http:s//gihuub.comeyaall222 Course
thtps://github.coo http:://ggtth..cmmm/dmeiaannnnno sac ae
20

---

Published as a conference paper at ICLR 2022
Implementation of Latent Dynamics Model Note that our goal is to learn good representation by enforeing it predicting the latent dynamics, different from model-based RL (Hafner et al., 2019) that aims to learn accurate models for planning. Therefore, we let the dynamics models P and R be simple linear networks, that the representation ing dynamics and learning values/policies. For environments with discrete action spaces, we learn [A] linear transition networks and |A] linear reward models. For environments with continuous ac- tion spaces, we first learn an action encoder v AA- + F with the same encoding size d as the state representation. Then, learn linear transition network and a linear reward network with P()(o) o (aa)) being the predicted next representation, and R(p(() (a)) being the predicted reward, where denotes element-wise product. In practice, we find this implementation achieves good performance across many environments.
be more informative in terms of represent-
In addition, note that due to the significant difference between source observation and target obser- vation, the initial encoding scale could be very different in source and target tasks, making it hard for them to be regularized by the same dynamics model. Therefore, normalize the output of both encoders to be a unit vector (12 norm is 1), which remedies the potential mismatch in their scales.
Hyperparameter Settings for Transfer Learning In experiments, we find that it is better set relatively large when the environment dynamics are simple and the dynamics model is of high quality. When the environment dynamics is complex, we choose to be more conservative and set to be smaller. Concretely, in CartPole, À is set as 18; in 3DBall, is set as 10; in Acrobot, À is set as 5; in the remaining MuJoCo environments where dynamics are more complicated, À is set as Although we use different X's in different environments based on domain knowledge, we find that different values of \'s not have much influence on the learning performance. Figure provided in Appendix E.2 shows test on the hyper-pat paper-parameter X, where we can see that our algorithm effectively transfers knowledge under various values of X.
Regarding the representation dimension, we set to be smaller for simpler tasks, and larger for more complex tasks. In 3DBall, we set the encoding size to be 8; in CartPole, we set the encoding size 16; in Acrobot, set the encoding size as 32: in Cheetah-Run, we set the encoding size as 50; in MuJoCo tasks, we set the encoding size as 256. Again, we find that the feature size does not influence the performance too much. But based sufficient representation Achille & Soatto (2018), we believe that it is generally better to have a lower-dimensional representation while making it is sufficient for learning.
the theoretical insights of learning minimal
21

---

Published as a conference paper at ICLR 2022
E.2 ADDITIONAL EXPERIMENTAL RESULTS
Ablation Study:
Transferring Different Components
Figure 5 shows the ablation study of our method in continuous control tasks. We compare our method with the following variants:
(1) learning auxiliary tasks without transfer,
(2) only transferring transition models P and
(3) only transferring reward models ft.
Compared with the single-task learning baseline (the blue curves), we find that all the variants of our method can make some improvements, which suggests that learning dynamics models as auxiliary tasks, transferring P and it are all effective designs for accelerating the target task learning. Finally, method (the red curves) that combines the above components achieves the best performance, justifying the effectiveness of our transfer algorithm.
HallCheetah
Hopper
5,000
3,000
4,000
Return
3,000
Return
2,000
2,000
Without Transfer Auxiliary Tasks
Without Transfer
Auxiliary Tasks
1,000
Transfer P
1,000
Transfer
Transfer At Our Method
Transfer
Our Method
Ok
200k
400k
600k
800k
1000k
200k
400k
600k
800k
1000k
Steps
steps
Walker2d
3DBall
4,000
80
3,000
60
Return
Return
2,000
Without Transfer Auxiliary Tasks
40
Without Transfer
Auxiliary Tasks
Transfer P
Transfer
1,000
20
Transfer ft Our Method
Transfer
Our Method
0k
200k
400k
600k
800k
1000k
50k
100k
150k
200k
250k
300k
350k
Steps
Steps
Figure 5: Ablation study of our method on different transferred components.
22

---

Published as a conference paper at ICLR 2022
Sanity Check: Effectiveness of the Proposed Transfer Method
150
Method
Auxiliary Tasks
We conduct another ablation study evaluate each component of our algorithm in the Cart- Pole environment shown in Figure 6. We find that when transferring the dynamics mod- els with only linear value head (the green curve), the agent fails to learn good policy as we analyzed in Section 4.3. If the dynam- ies models (P, R) are randomly generated in- stead of being transferred from the source task (the orange curve), the agent does not learn, ei- ther. More importantly, if we learn dynamics models as auxiliary tasks in the target task with- out transferring them from the source (the pur- ple curve), the agent learns little better than vanilla agent, but is worse than proposed transfer algorithm. These empirical results have verified our theoretical insights and shown the effectiveness our algorithm design.
Without Transfer
Episndic Retur
Random Models
100
Lincar Value Head
50
50
100
150
200
Episodes
Figure 6: In the Vec-to-pixel CartPolc environment, sanity check verifies the effectiveness of our algorithm design. Results averaged over 20 random seeds.
Test
Figure/ 7 further visualizes how the hyperpa- rameter X (regularization weight) influences the transfer performance in the Vec-to-pixel Cart- Pole environment. It can be found that the agent generally benefits from a larger X, which suggests that the model-based regularization has a positive impact on the learning perfor- mance. For wide range of X's, the agent al- ways outperforms the learner without transfer
80
Transfer with Various
Without Transfer
Average Return
70
60
(the learner with À 0). Therefore, rithm not sensitive to the hyperparameter X, and larger À is preferred to get better perfor- mance. In Appendix 1.3, we have provided the À selections for all experiments.
algo-
50
25
Regularization Weight
Figure 7: Vec-to-pixel CartPole environment, under different selections of hyperparameter A, al- gorithm works better than learning from scratch (when 0). Results are averaged over 20 random seeds.
23


Reviews:
[{"review_id": "cwT133NiTuW", "reviewer": "Reviewer_L4Qx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n* The novel problem formation is well motivated from real-life applications.\n* The structural similarity assumption 1 made for the problem setting is reasonable and the proposed solution elegantly solves the problem by leveraging the structural similarity. \n\n## Cons/questions/suggestions\n* __Comparisons that demonstrate the practical benefits when observation quality is improved are missing__ The example of upgrading sensors to improve observation quality is a great motivating example for the problem of transfer RL between different observation spaces. For some RL applications, by reducing the noise or augmenting the observations, one might be able to learn a better policy than the policy trained on the original representation. Based on this understanding, I think an experiment that demonstrates the usefulness of the proposed algorithm in this setting would greatly strengthen the current work. Specifically, one could study the transfer between pixel inputs or vector inputs with different levels of background noise or image resolution. The knowledge transferred from poorer observations should enable an efficient learning when the observation quality is improved (_which is shown in Fig. 4_), and the asymptotic performance of the policy under better representations should improve upon the poorer representation's corresponding policy (_this is missing_). If such results can be included, I believe readers can better appreciate the importance of this novel problem, because it might allow one to improve observation qualities to get better policies, without having to train from scratch on the new observations. \n* __Details on observation encoder initialization__ For vector-image transfer, the encoder architectures would be different and the latent output $z$ could shift a lot for the same underlying state under the vector observation and the image observation. Would this large latent vector shift lead to unstable training? Does one need to control how these encoders are initialized? \n\n## Minor comments\n* The authors claim that the proposed method can make it easier to learn a good representation from the new observation space. The results in figure 3 and 4 indirectly support the authors' claim. I wonder whether it would be possible to directly compare the representation learning progress with/without the latent dynamics transfer. For example, use the learned representations for some auxiliary tasks such as clarification/prediction/clustering and compare the performance.\n", "summary_of_the_paper": "This paper tackles the problem of observation space change for RL problems, i.e., the observation changes while the dynamics remain similar. To deal with this problem, the authors propose to learn a latent dynamics models based on encoded observations. When the observation space is changed, the learnt latent dynamics model can serve as a knowledge prior to transfer known dynamics information when learning with the new observation space. \n\n", "summary_of_the_review": "The novel problem setting proposed by the authors is of practical interests and the authors's proposed solution is well justified. Though the proposed method does not provide significant benefit when it's hard to learn the transition dynamics, the novel problem formulation could inspire future work along this line to develop new methods that can achieve better transfer for more challenging environments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Md2FVDVUZw0", "reviewer": "Reviewer_igV9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written, although some technical points are unclear. The problem that the authors consider is interesting and relatively novel. Specifically, the setting is close to many previously studied settings, but not identical to the best of my knowledge. The paper combines several theoretical results with empirical evaluation. There are three main issues with the paper: motivation, technical clarity, and experimental evaluation. \n\n## Motivation\n\nThe setting the authors consider, transferring between observation spaces, has not been tried before to the best of my knowledge. The authors claim that it is an important problem, but I do not find the motivation for this very persuasive. The authors state that “Observation change is common in practice due to hardware upgrading, data restriction or curriculum design.”. Could specific citations be provided here? Figure 1 shows an example where observation spaces could change, but it seems to be an abstract toy example. It would be better to use a real-world example here. Maybe sim-to-real is a better example since simulated environments can expose data structures for the simulator, which real environments cannot. \n\nOverall, I think the sim-to-real problem is the setup closest to what the authors are trying to achieve. I would encourage the authors to discuss sim-to-real more, and maybe use some baselines from there. See e.g. ​​Sim-to-Real: Learning Agile Locomotion For Quadruped Robots.\n\n\n## Technical Clarity\n\nThe role of the approximation operator is a little unclear. What does this operator model? Does it just mean the best Q-value estimates available under given features? The paper would be easier to read if a motivating example was given when it was introduced. \n\nOverall, the theoretical arguments appear to be somewhat circular. The authors define a representation mapping to be sufficient if it enables accurate Q-values. It is then shown that if a representation mapping is sufficient, one can obtain accurate q-values. I have a hard time assessing what the theoretical results really prove. \n\nIs it possible to measure if some representation mapping is sufficient? If not, the definition might not be useful in practice.\n\nAlso, do the theoretical results apply to the empirical setting considered later? \n\nFor the experiment section, the setup details are relegated to Appendix D. There, however, there are not enough details to really understand what is going on. E.g., what is the size of the screen, and what is the size of the crop? What parameters, beyond learning rate, are used for adam? What settings are used for the replay buffer? For continuous control, are the actions discretized in any way to enable an argmax policy?\n\n## Experimental evaluation\n\t\t\nThe authors state “We implement a DQN learner”. There are many high-quality DQN implementations available online, and doing your own implementation can introduce bugs. It is not clear that the DQN implementation works well, and if it does not, improving upon it is not very meaningful.\n\nIn Figure 4, results on tasks hopper and walker do not appear to be statistically significant. \n\nHow are the tasks selected? How are the architectures selected? How are the hyperparameters selected? \n\nFor latent space models, there are many excellent implementations available online e.g. dreamerV2. Why not use one of these?\n\nWhy are the proprioceptive states of the environments modified? E.g. in Appendix D, the authors write: “For the target task of MuJoCo environments, we add the center of the mass based inertia and velocity into the observations of the agent”.\n\n\n\n## Post rebuttal update\n\nI thank the authors for the many clarifications and improvements to the paper! I want to retain my score of 5 for two reasons. Firstly, I still think that the empirical results are not strong enough. In Figure 3, it seems like the improvements are only significantly outside the error bars for two tasks (Vec-to-pixel: Cheetah-Run and More-sensor: HalfCheetah). Secondly, while the authors have improved the motivation by giving relevant real-world scenarios where their methods apply, the experimental evaluation is pretty far away from these examples. Thus, while their methods improve performance for these toy tasks, I am not convinced their methods will improve a real-world problem. One specific direction to pursue here might be that of this paper (https://arxiv.org/abs/1912.12294). \n\n\n\n\n", "summary_of_the_paper": "The paper considers the problem of RL environments where the observations space can change dramatically, e.g. from proprioceptive to pixel observations. The authors introduce various definitions for this setting and provide proofs regarding convergence (in the sense of Q-values converging to true values). Over 5 environments in continuous control, the authors demonstrate that their method enables transfer learning from proprioceptive to pixel observations and that such transfer learning can be beneficial.", "summary_of_the_review": "While the paper often is well-written and considers an interesting problem, the motivation for the problem is lacking. Additionally, lack of technical clarity on both the empirical and theoretical sides makes it hard to assess what the paper really proves. At last, the empirical evaluation is problematic -- some results are not statistically meaningful, many parameter choices are not motivated and it is not clear that the baseline DQN implementation works well in the first place.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LgHzvZ3gTol", "reviewer": "Reviewer_gWD8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I generally like the idea that the paper follows and its theoretical rigorousness. However, I have a few concerns about the exposition of the paper as well as its empirical evidence. Also, I think the number of real-world applications this solution addresses is very limited, making it a very narrow field of application.\n\nWhile I especially like the insights to non-linear policy heads at the end of Sec. 4, I suggest shortening Secs. 4.1 and 4.3 and reduce them to their general insights (putting the rest into the appendix). This makes more space available for the presentation and discussion of additional experimental results.\n\n\n*** Motivation of the paper and real-world application. ***\nAt points I was confused by the motivation of the paper. This started in the very beginning with “the observation space is specified by human developers and restricted by physical realizations and may thus be subject to dramatic changes” – I wonder in which real-world use-cases this really is the case (I mean – without considering simulation environments etc.). Even more: switching from vector-based observation to image-based observation sounds interesting but might finally be a fictional problem setup. \n\nA much more intuitive scenario would be a sensor fusion setup where you have different sensors that measure different physical properties but which, in their sum, are redundant. If you remove a sensor (breakage?) the policy becomes useless as it expects a different input encoding while still (as the authors propose) the dynamics remains the same.\n\n\n*** Related work ***\n-\tThe papers argues that it is not easy to scale up the work of Sun et al (2020) for continuous domains. However, while not originally being proposed it seems more or less straight-forward? This is also lined out in the original paper. Hence, this work might also serve as a baseline\n-\tThe paper argues that the work proposed by Raiman et al (2019) does not work for drastic changes. I cannot completely agree. The originally proposed gradient mapping could also be applied to the embedding where there is a sufficient representation. This work could hence also serve as a baseline. Even if it fails for the ‘drastic change’ it might work on the proposed Mujoco experiments.\n\n\n*** Experimental Results ***\n\nAlthough the authors claim that there is no previous work to which they can compare their algorithm to, I have a few suggestions/remarks:\n-\t(see lsao before) In the introduction you mention previous work of Raiman et al. (2019) who add or remove individual observation features. This is exactly what the authors do in their Mujoco experiments. Hence, this work serves as a baseline to which the authors should compare their approach to\n-\tAlso an option would be to not starting to train the policy from scratch but to fine-tune the policy on the new observation representation. At least for the Mujoco setup this should be a viable starting point.\n-\tAs the authors effectively propose a model-based RL algorithm/extension they should also run their algos against MBRL baselines.\n\nQuestions on DQN & Cartpole:\n-\tWhat exactly has been used in DQN? Dueling Double DQN with prioritized experience replay and noise? What about exploration parameters? As the results are not clearly showing the approach to outperform I wonder how well it would perform under perfect DRL algorithm configuration….\n-\tPresenting a screenshot of the cartpole to the agent forms a POMDP as we do not get information on the angular velocity and the velocity of the pole’s tip – which are both the most important components. I wonder why the authors formulate the experiment like this. Or did you stack frames to ensure Markovian state representation?\n\nQuestions on SAC & Mujoco/3DBall:\n-\tIn my opinion, the observation representation change in the Mujoco environments is too simplistic. There are only features stacked on top of the existing features. Here, I expect a fine tuning of the policy whily freezing anything but the early layers should outperform both approaches, right?\n-\tThe 3DBall modification is much more challenging and effectively also resembles the motivation of the paper. However, the results are not interpreted at all? Moreover, the results seem not convincing. It looks like both perform en par while the “without-transfer” approach does not reach the final performance. (Or does it towards the end???) There is too few information on the task and environment to interpret the results….\n\nIn general, there is a lack of discussion of the results of the approach and its limitations. It would also be interesting to see an ablation study that shows how much the representation effectively needs to change so that the proposed approach beats a fine-tuning of the policy. Also: what is the motivation of using DQNs in the first experiment and SAC in the last? I have the feeling that in some/many combinations of DRL agents and environments the proposed method also might lack behind and retraining of the policy…\n\nMinor remarks:\n-\tTypo in the introduction: automotive[-s-]\n-\tThe concept of the target networks (p5 bottom) is confusing at first as it refers to target/evaluation nets known from DQN. At this point the readers first thinks that the paper talks about the network that encodes target environment dynamics.\n", "summary_of_the_paper": "The paper addresses the problem of adapting a policy to a novel representation of the environment. Different from most previous work the paper focuses on the problem of a change in the representation of the observation provided by the environment, i.e., assuming that the underlying dynamics P and R remains mostly similar. The idea is to learn source and target models of the environment jointly with the policy. This serves as a model-based regularization that guides policy and representation towards the learned models. This effectively leads to a learned representation that is shared for the policy for the source and the target environment representation.", "summary_of_the_review": "I generally like the idea of the paper but in my opinion the paper is not ready yet for publication in ICLR. The main reason for that is the lack of experimental evidence and lack of comparison to previous work exposed to ablated variants of the proposed problem setting. Moreover, the application of the methods seems quite narrow given the assumptions of drastic representation changes during evolvement of environments.\n\n*** update of the review ***\n\nI would like to thank the authors for their thorough submission update. The other reviews also share some of my criticism and the authors imho did a very good job updating their work. Exposition and motivation are much clearer know the the experiments have improved considerably.\n\nBottom line: an increase of my score is definitely justified and I will increase my score to 5. The reason while I am still not leaning towards acceptance is because I still have concerns:\nThe fine-tuning experiment is not what I actually meant. In the experiments only the policy head is kept but I suggested to keep on training with the new representation (on the cases where the observation just gets bigger; just add the missing neurons at the beginning and initialise them). This should be faster/better. \nAnd against such a baseline, I am still missing an ablation study that answers the following question: How much different must a representation be such that the transfer approach is better?\n\nP.S. there are two typos:\n- feature is corresponding [to] which old observation feature.\n- Page 8, last line: 'Fine-tine' instead of 'fine-tune'", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4gr6pk6ggBA", "reviewer": "Reviewer_RpCP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found this work interesting but slightly lacking in focus, and especially the theoretical parts are slightly divorced from the effective implementation. The empirical work is also quite similar to previous work, despite the rather different problem setting considered.\n\nHere are additional comments and questions which would be worth clarifying:\n\n1. The problem considered is quite novel and I feel could lead to novel applications and approaches. However, in my opinion this would be more interesting if only the environment dynamics were considered to be shared between the source and target task, and when action spaces would differ. In its current instantiation, this restricts this to very particular situations.\n   1. A more generic solution would be to consider MDP homomorphism, something that was tackled by van der Pol et al, 2020.\n   2. This paper isn’t cited and discussed and is quite clearly relevant to this current work, alongside more recent works from Amy Zhang.\n   3. Some other works in bisimulation metrics and behavioral similarity might also deserve to be contrasted and discussed.\n2. I liked how Section 4.1 explicitly spells out that a representation needs to support the policy improvement path and not just the fixed point. However, I am not yet fully convinced of how novel Definition 3 and Lemma 4 really are. The proofs appear quite similar in construction to many existing results in the (recent) bisimulation literature, and the end of this section makes statements which can be related to the Value Equivalence Principle from Grimm et al 2020. Similarly, the assumption of linearity brings the setting closer to successor features and their more novel versions, something that has been thoroughly explored over the years.\n   1. Again, I’m looking to understand how novel and different the current work is compared to previous work which aren’t explicitly discussed in this draft.\n3. The loss used in practice is not that related to the theoretical derivations, and hence the paper isn’t as focused as it could be. \n   1. In effect, L_P and L_R are rather standard in the model-based literature (e.g. they are used exactly as is in Van der Pol 2020), so the only novelty is the constraint to still use the same frozen Transition and Reward models in the target task (which is interesting).\n4. Figure 3 presents an interesting setting (transfering from joints to pixel observations), and the method does help.\n   1. However, Figure a and b do not appear to have converged for the “Pixel Obs, Without Transfer” condition. I would not be comfortable drawing conclusions given the current curves.\n   2. This is also performed on Cartpole, which is rather simplistic.\n5. I wished Figure 4 was also covering a similar setting (joints -> pixels), instead of the rather arbitrary addition of extra joints. \n   1. Results are also not as promising, no setting apart from HalfCheetah can convincingly demonstrate the benefit of the method.\n6. I found Section D.2 in the Appendix quite interesting, especially the ablation about transferring P or R was something that I was about to ask about, so I wished it was more discussed. Recent work indicates that matching R in bisimulation metrics is prone to overfitting, so it’s interesting that these results do seem to indicate that it is still helpful to capture it.\n\n\nReferences:\n   * Van der Pol 2020: https://arxiv.org/abs/2002.11963 \n   * Zhang 2020, https://arxiv.org/abs/2003.06016 \n   * Castro 2020, https://arxiv.org/abs/1911.09291 \n   * Grimm 2020, https://arxiv.org/abs/2011.03506 \n   * Also slightly related: https://arxiv.org/abs/1911.12247 , https://arxiv.org/abs/2107.11676 \n\n", "summary_of_the_paper": "This paper proposes to leverage models of the environment and rewards as regularizers to perform explicit transfer between different observation spaces in RL.\n\nBeing able to quickly adapt to different observations is quite a different problem setting than what the rest of the literature is tackling (and we could argue on how relevant it is, but let’s leave that aside). They provide some theoretical analysis of this setting, especially one interesting result on when their method can converge to a near-optimal solution through approximate policy iteration.\n\nThey tackle it using ideas very similar to the recent wave of work leveraging bisimulation metrics, hence novelty / comparison on this aspect will be important.\nThey assess it on simple Mujoco control tasks, where their results are promising but perhaps still a bit early.", "summary_of_the_review": "In conclusion, I found this paper interesting, and tackling a rather novel problem setting. However, it currently is quite theory centric albeit without contrasting the novelty of their analysis compared to previous works as much as I’d like it to.\nIn addition, the empirical instantiation is rather divorced from the analysis, and is not novel compared to the large body of literature assessing model-based regularization and representation learning.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work suggests using models of the environment as regularizers for performing explicit transfer in RL. Here are some of the highlights from the reviews and subsequent discussions:
  * Novel problem 
  * Unclear to some of the reviewers why the problem setting is in fact important.
  * Well-written
  * Interesting theoretical results
  * Somewhat limited experimental results
Post-rebuttal, while there is not necessarily a great consensus, the reviewers all feel that it's an improved piece of work. While I am myself not fully convinced that the problem setting motivation truly aligns with the kind of empirical results that the work provides, on the balance I think this work is interesting and has sufficient novel contributions to be accepted at ICLR.
Decision: Accept (Poster)