AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning
Authors: Biwei Huang
Fan Feng
Chaochao Lu
Sara Magliacane
Kun Zhang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Transfer RL
Graphical models
Efficient adaptation

Abstract: One practical challenge in reinforcement learning (RL) is how to make quick adaptations when faced with new environments. In this paper, we propose a principled framework for adaptive RL, called AdaRL, that adapts reliably and efficiently to changes across domains with a few samples from the target domain, even in partially observable environments. Specifically, we leverage a parsimonious graphical representation that characterizes structural relationships over variables in the RL system. Such graphical representations provide a compact way to encode what and where the changes across domains are, and furthermore inform us with a minimal set of changes that one has to consider for the purpose of policy adaptation. We show that by explicitly leveraging this compact representation to encode changes, we can efficiently adapt the policy to the target domain, in which only a few samples are needed and further policy optimization is avoided. We illustrate the efficacy of AdaRL through a series of experiments that vary factors in the observation, transition and reward functions for Cartpole and Atari games.

Published as a conference paper at ICLR 2022
ADARL: WHAT, WHERE, AND How TO ADAPT IN TRANSFER REINFORCEMENT LEARNING
Biwei Huang
Fan Feng
Careegie Mellon University bi biweiht andrew.w............. cmw.edu edd
City University of Hong Kong
f fengl 01 7 70 1, com
Chaochao Lu
University of Cambridge & Max Planck Institute for Intelligent Systems 216410cam, ac ac
Sara Magliacane University of Amsterdam & MIT-IBM Watson Al Lab sara. sara.magl iaraagiacannegpge com
Kun Zhang
Carnegie Mellon University & Mohamed bin Zayed University Artificial Intelligence kunz @ cemu.edu
ABSTRACT
One practical challenge in reinforcement learning (RL) is how to make quick adaptations when faced with new environments. In this paper, we propose a principled framework for adaptive RL, called AdaRL, that adapts reliably and efficiently to changes across domains with few samples from the target domain, even in partially observable environments. Specifically, we leverage parsimonious graphical representation that characterizes structural relationships over variables in the RL system. Such graphical representations provide a compact way to encode what and where the changes across domains are, and furthermore inform us with a minimal set changes that one has to consider for the purpose of policy adaptation. We show that by explicitly leveraging this compact representation encode changes, we can efficiently adapt the policy to the target domain, in which only few samples are needed and further policy optimization is avoided. We illustrate the efficacy of AdaRL. through a series of experiments that vary factors in the observation, transition and reward functions for Cartpole and Atari games
INTRODUCTION AND RELATED WORK
Over the last decades, reinforcement learning (RL) (Sutton and Barto, 1998) has been successful in many tasks (Mnih et al., 2013; Silver et al., 2016). Most of these early successes focus on a fixed task in fixed environment. However, in real applications we often have changing environments, and has been demonstrated that the optimal policy learned in specific domain may not be generalized to other domains (Taylor and Stone, 2009). contrast, humans are usually good at transferring acquired knowledge to new environments and tasks both efficiently and effectively (Pearl and Mackenzie, 2018), thanks to the ability to understand the environments. Generally speaking, to achieve reliable, pocesss-wiihh
low-cost, and interpretable transfer, it is essential to understand the underlying decision-makin factors have changes, where the changes are, and how they change, instead of transferring blindly (e.g., transferring the distribution of high-dimensional images directly).
There are roughly two research lines in transfer RL (Taylor and Stone, 2009; Zhu et al., 2020): (1) finding policies that are robust to environment variations, and (2) adapting policies from the source domain to the target domain as efficiently as possible. For the first line, the focus is on learning policies that robust to environment variations, e.g., by maximizing risk-sensitive objective over a distribution of environments (Tamar et al., 2015) or by extracting a set of invariant states (Zhang et al., 2020a; 2021a; a;; Tomar al., 2021). A more recent method encodes task-relevant invariances by putting behaviorally equivalent states together, which helps better generalization (Agarwal et al., 2021). On the other hand, with the increase of the number of domains, the common part may get
"Code link: https//githuh..oo///apppii httpa///Addaaaiieee/Alaaaaeeeo.

---

Published as a conference paper at ICLR 2022
Source domains
Target domain
Domain 1
Domsin-speciic parsmnatrss
Identily compect domain- generafisable repreeentations (s,th, ((*))
Domain 2
Domain-shared represantation
Estimate domsin-specife parameters Gomp
Policy learning source domeins
(At Optimal perametr eed policy
(0
timesiloe
Optimal target polcy
Model estimation
Figure 1: The overall AdaRL framework. We learn Dynamic Bayesian Network (DBN) over the observations, latent states, reward, actions and domain-specific change factors that is shared across the domains. We then characterize a minimal set of representations that suffice for policy transfer, so that we can quickly adapt the optimal source policy with only a few samples from the target domain.
even smaller, running counter to the intention of collecting more information with more domains. Moreover, focusing only on the invariant part and disregarding domain-speciffe information may not be optimal; for instance, in the context of domain adaptation, it has been demonstrated that the variable part also contains information helpful to improve prediction accuracy (Zhang et al., 2020b).
In this paper, we propose method along the second line, adapting source policies to the target. Approaches along this line adapt knowledge from source domains and reuse it in the target domain to improve data efficiency, i.e., in order for the agent to require fewer explorations to Iearn the target policy. For example, an agent could use importance reweighting on samples (s, a, 8') from sources (Tirinzoni et al., 2018; 2019) start from the optimal source policy initialize learner in the target domain, as a near-optimal initializer (Taylor et al., 2007; Fernández et al., 2010). Another widely-used technique is finetuning: model is pretrained on source domain and the output layers are finetuned via backpropagation in the target domain (Hinton and Salakhutdinov, 2006; Mesnil al., 2012). PNNs (Rusu et al., 2016), instead, retain a pool of pretrained models and learn lateral connections from them to extract useful features for a new task. Moreover, a set of approaches focus on sim2real transfer by adapting the parameters (Yu et al., 2017: Peng et al., 2020). However, many of these approaches still require a large amount of explorations and optimization in the target domain.
Recently, meta-RL approches such as MAML (Finn et al., 2017), PEARL (Rakelly et al., 2019), CAVIA (Zintgraf et al., 2019), Meta-Q learning (Fakoor et al., 2020), and others (Mendonca et al., 2019; Clavera et al., 2019; Duan et al., 2017) have been successfully applied to learn an inductive bias that accelerates the learning of new task by training methods (c.g.. CAVIA and PEARL), as well as some prior work (c.g., HiMDPs (Doshi- Velez and Konidaris, 2016)) and recent follow-ups (Zhang et al., 2021b), have similar motivation in new environment not all parameters need to be updated, so we can force the model to only adapt of context parameters. However, these methods mostly focus on MDPs (except the Block MDP assumption in Zhang et al. (2021b)) and model all changes as a black-box, which may be less efficient for adaptation, as opposed a factorized representation of change factors.
large number of tasks. Some of these
work:
Considering these limitations, we propose AdaRL, a transfer RL approach that achieves low-cost, reliable, and interpretable transfer for partially observable Markov decision processes (POMDPs), with MDPs as a special case. In contrast to ttate-of-the-a approaches, we learn a parsimonious graphical representation that able characterize structural relationships among different dimensions of states, change factors, the perception, the reward variable, and the action variable. It allows us to model changes transition, observation, and reward functions a component-wise way. This representation is related to Factored MDPs (Kearns and Koller, 1999; Boutilier et al., 2000; Strehl et al., 2007) and Factored POMDPs (Katt et al., 2019), but augmented with change factors that represent din-eimesiional embedding of the changes across domains. Our main motivation is that distribution shifts are usually localized they are often due to the changes of only a few variables in the generative processes, can just adapt the distribution of small portion of variables (Huang
2

---

Published as a conference paper at ICLR 2022
et al., 2020; Schölkopf et al., 2021) and, furthermore, factorized according to the graph structure, each distribution module can be adapted separately (Schölkopf, 2019; Zhang et al., 2020b).
In Fig. 1 we give motivating example and general description of AdaRL. In this example, consider learning policies for Pong (Bellemare et al., 2013) that can easily generalize to different rotations w and to images corrupted with white noise. Specifically, given data from n source domains with different rotations and noise variances, we learn a parsimonious latent state representation shared by all domains, denoted by St, and characterize the changes across domains by a two-dimensional factor Og. We identify set of minimal sufficient representations (sman, .x%i)) ) for policy transfer. For instance, here only the rotation factor w needs adapting (i.e., 0ginn wa), since the noise factor does not affect the optimal policy. Similarly, as we will show formally in the rest of the paper, not all components $t, of the state vector St are necessary for policy transfer. For example, $2,1 & s sqmin, since never affects the future reward. We learn an optimal policy x (- (-10 on source domains. In the target domain, we only need a few samples to quickly estimate the value of the monddimessional Omin and then we can apply x (-(Otarge) directly. Our main contributions are summarized below:
We assume generative environment model, which explicitly takes into account the structural relationships among variables in the RI system. Such graphical representations provide compact way to encode what and where the changes across domains are.
Based on this model, we characterize a minimal set of representations that suffice for policy learning across domains, including the domain-speciff change factors and domain-shared state representations. With this characterization, we adapt the policy with only few target samples and without policy optimization in the target domain, achieving low-cost and reliable policy transfer. By leveraging a compact way to encode the changes, we also benefit from multi-task learning in model estimation. In particular, we propose the Multi-modee Structured Sequential Variational Auto-Encoder (MiSS- for reliable model estimation in general cases.
2 A COMPACT REPRESENTATION OF ENVIRONMENTAL SHIFTS
Suppose there are n source domains and ní target domains. In each source domain, observe sequences {or, ae, re) IT where o, E O are the perceived signals at time (c.g., images), a E A is the executed action, and r, E R is the reward signal. We denote the underlying latent states by 8; = (81,1, s8,,) where d is the dimensionality of latent states. We assume that the generative process of the environment in the k-th domain (with k L.... n + n²' can be described in terms of the transition function for each dimension of s and the observation and reward functions as
{
$i, filctts @ St- og -0-1. cL. 0% C.,): for = -,...,,. .d.
Ot = g(o 3 8e, auto 0g)
Tt = h(c St-1, 1,--1.00 car. aa-1, eutr 0g, E.),
where @ denotes the element- product, the ei,,; Ei of terms are i.idd. random noises. As explained below, e are masks (binary vectors or scalars that represent structural relationships from one variable to the other), and O2 = (0%, 0g. 0%) are the change factors that have a constant value in each domain, but vary across domains in the transition, observation, and reward function, respectively. The latent states S141 form an MDP: given St and a,, S141 is independent of previous states and actions. The perceived signals oe generated from the underlying states sy. The actions a, directly influence the latent states 8,41, instead of the observed signals ot, and the reward is determined by the latent states and the action. Eq. can also represent MDPs special case if states St are directly observed, in which case the observation function of og is not needed.
Structural relationships and graphs. Often the action variable a;-1 does not influence every dimension of st, and similarly, the reward Tt may not be influenced by every dimension of s,-1- Furthermore, there are structural relationships between different dimensions of s and s,. To characterize these constraints, we explicitly take into account the graph structure G over the variables in the system characterized by a Dynamic Bayesian Network (Murphy, 2002) and encode the edges with masks c In the first equation in Fq. I the transition function for the state component 8;, where the jth entry of ce*s E (0, 1 is 1 if and only if 8J.t influences 81,411 (graphically represented by an edge). while of E {0, 1) is 1 if and only if the action a, has any effect $1,++1. Similarly, the binary vector c, 0.*5 E {0, 1)2 encodes which components of the change factor 0z (0f, ) affect 8L.+1 The masks in the observation function g and reward function h have similar functions. The
3

---

Published as a conference paper at ICLR 2022
masks and the parameters of the functions f. g. and h, are invariant; all changes are encoded in 0k- For simplicity of notation, we collect all the transition mask vectors in the matrices C** := [c 1=11
and CO.>* : |c |c Ji=1 and the scalars in the vector ca := 1 |c li=1.
Characterization of change factors in a compact way. In practical scenarios, the environment model may change across domains. Moreover, it is often the case that given a high-dimensional input, only a few factors may change, which known as minimal change principle (Ghassami al., 2018) or sparse mechanism shift assumption (Schölkopf et al., 2021). In such a case, instead of learning the distribution shift over the high- tigh-dimensional input, thanks to the parsimonious graphical representation, we introduce now-dimensional vector 0g to characterize the domain-specifer information in compact way (Zhang et al., 2020b). Specifically, 0g. 0%. and 0% capture the change factors in the observation function, reward function, and transition dynamics, respectively; each of them can be multi-dimensional and that they are constant within each domain. In general, 0g can capture both the changes in the influencing strength and those in the graph structure, e.g., some edges may appear only in some domains. Since we assume that the structural relationships in Eq. are invariant across domains, this means that the masks c have to encode an edge even if it presents only in one domain, and furthermore, since 0g encodes the changes, it can switch the edge off in other domains. Fig. 1 shows an example of the graphical representation of the (estimated) environment model. Specifically, in this example, 0% only influences s1,t, ar-1 does not have an edge to $1,c, and among the states, only s has an edge to rt. In this example, we consider the case when the control signals are random, so there is no edge between S and az.
WHAT, WHERE, AND HOW TO ADAPT IN RL
We first assume that the environment model in Eq. 1 is known (we will explain how to learn it in Sec. 3.1), and characterize which changes have an effect on the policy transfer the target domain. In Eq. 1, we allow the model to change across domains, including all involved functions, and we leverage 9₂ to capture the changes in a compact way. The varying model implies that the optimal policy function may also vary across domains. How can we then characterize the changes in the optimal policy function in a compact way, as we did in the model? Interestingly, we find that the change factor O2 and the latent state S, are sufficient for policy learning, but not every dimension of 9₂ or s is necessary, since they may not ever have an effect on the reward, even in future steps. We first give the definitions of compact domain-sharre representations and compact domain-spect representations, according to the graph structure, and we further show that they are the minimal set of dimensions that suffice for policy learning across domains (proof in Appendix).
Definition 1. Given the graphical representation of an environment model G that is encoded in the binary masky c²,, we define recursively the representations that affect the reward in the future as:
compact domain representations sqmin: the latent state components Si,t E St. that either have an edge to the reward in the next step Ts+1, i.e., ct"" =1, or have an edge to another state component in the next time-step sj,+1, i.e., 3 1, such that the same component at time is compact domain-sharet representation, i.e., $j E sqnin compact domain-specific representations Opin. the latent change factors 0z,k E On that either: have an edge to the reward in the next time-step T141, i.e., 0 0% and con >r = 1, or By =
such
have an edge to a state component SJ, E sqnnn i.e. G,
Proposition 1. Under the assumption that the graph G is Markov and faithful the measured data, the union of compact domain- seeiii---eificccee 0mann and compact shared representations sqiin are the minimal and sufficient dimensions for policy learning across domains.
For the example in Fig. smin = (81.2; Sa,t) and 0% = {0;, )g Note that 0g is never in 0 and thus if only the observation function changes, the optimal policy function _ remains the same across domains. For example in Cartpole a change of color does not affect the optimal policy. Moreover, if dutt 1, then 0f E Opein which is the case for multi-task learning.
3.1 SIMULTANEOUS ESTIMATION OF DOMAIN- VARYING MODELS
In this section, we give the estimation procedure of the environment model in Eq. from observed sequences {((y de.x) ) from each source domain k; where Yı,k (o,,..T[.A) are the observations and reward at time in domain k. Instead of estimating the model in each domain
4

---

Published as a conference paper at ICLR 2022
separately, we estimate models from different domains simultancously, by exploiting commonalities across domains while at the same time preserving specific information for each domain. In particular, we propose the Multi-model Structured Sequential Variational Auto-Encoder (MiSS-VAE) which contains the following three essential components. (1) "Sequential VAF" component handles the sequential data, with the underlying latent states satisfying an MDP. It is implemented by adding an LSTM (Hochreiter and Schmidhuber, 1997) to encode the sequential information in the encoder to learn the inference model qo (Sr, 1, & 11:--185 2x). (2) "Multi-model" component handles models from different domains at the same time, using the domain index k as an input and learning the domain- -specific factors 9z. (3) "Structured" component: exploits the structural information that explicitly encoded with the binary masks, i.e., c in Eq. 1. Here, the joint distribution of latent states are factorized according to structures, instead of being marginally independent as in traditional VAEs (Kingma and Welling, 2014). Fig. A3 in Appendix gives the diagram of neural network architecture in model training. Let 1: = ([yc.>Y-11. In By taking into account the above three components, we maximize the following objective function L:
Ci (31, 12,3,,,, Φ. ).) = _ cy1:7 (32, o, +))+ (y:: (92, 0))- (y1: (é. 7, c -
In particular, L is the reconstruction loss for both observed images and rewards, to learn the observation and the reward function, respectively. We also consider the one-step prediction loss Crred
L Σ
B unaal (0,) Plogps, 1s. 0g. 20to e log (1241, ore 0 oarr etrr c
Am1
gredd= >
L B Bes [logpa, (. 0g, 0t) + logpa, 2 or 1 0% 0g),
where p3, and P&g denote the generative models with parameters 31 and 82, respectively, that are shared across domains, and go the inference model with shared parameters 6. We also use the following KL-divergence loss to constrain the latent space:
CKL =o
Σ E
" (L St-1,ks 41:-114: (stllss: 10- 0g. C cats ClO+t)),
where we explicitly model the transition dynamics py with the parameters 7 shared across domains; this is essential for establishing a Markov chain in latent space and learning representation for long-term predictions. Moreover, the KL loss helps to constrain the latent space to (1) ensure that the disentanglement between the inferred latent factors ((8,,el-) and g(85,xl.) +) for i A j. since we do not consider the instantaneous connections among state dimensions, and (2) ensure that the latent representations s, are maximally compressive about the observed high-dimensional data. Furthermore, according to the edge-minimality property (Zhang and Spirtes, 201 I) and the minimal change principle (Ghassami et al., 2018), we add sparsity constraints on structural matrices and on the change of Nomain-specifte factors across domains, respectively, to achieve better identifiability: _ A >/|| >2 |= >a + Aal|C** "||/""**'+|>>>>>>>>>>>> + > AellC "0111 A7
Σ 1SS.AS
10, O.l.
Note that besides the shared parameters {B1, 32, o, 7). the structural relationships (encoded in binary masks c) also involved in the shared parameters. Each factor in Pạ. PA., and p-, is modeled with a mixture of Gaussians, because with a suitable number of Gaussians, it can approximate a wide class of continuous distributions. Moreover, in model estimation, the domain-specific factors O2 {0% 02, } are treated as parameters; they are constant within the same domain, but may differ in different domains. We explicitly consider O₂ not only in the generative models pa, and p.,, but also in the inference model 4a- In this way, except for θR, all other parameters in MiSS- VAE are shared across domains, so that all we need to update in the target domain the oin-dimesioonal 0s. which greatly improves the sample efficiency and the statistical efficiency in the target domain.
3.2 Low-COST AND INTERPRETABLE POLICY TRANSFER
After identifying what and where to transfer, we show how to adapt. Instead of learning the optimal policy in each domain separately, which is time and sample inefficient, we leverage multi-task learning strategy: policies in different domains are optimized at the same time exploiting both commonalities and differences across domains. Given the compact domain-shared s, min and domain- specific representations 0m we represent the optimal policies across domains in a unified way:
04 = n (s min mmin
(2)
where gmin explicitly and compactly encodes the changes in the policy function in each domain k, and all other parameters in the optimal policy function x* are shared across domains. In other
5

---

Published as a conference paper at ICLR 2022
words, by learning x² in the source domains, and estimating the value of the change factor 6 farge and inferring latent states Staget from the target domain, we can immediately derive the optimal policy in the target domain without further policy optimization by just applying Eq. 2.
The AdaRL framework answers what and where the change factors are and which change factors need to adapt across domains in an interpretable way. Moreover, AdaRL only requires a few samples to update the low-dimensional domain-specific parameters Ouagee to achieve the optimal policy in the target domain, without further policy optimization, achieving the low cost. We provide the pseudocode for the AdaRL. algorithm in Alg. 1. The algorithm has three parts: (1) data collection with random policy or any initial policy from n source domains (line 2), (2) model estimation from the n source domains with multi-task learning (lines 2-3, see Sec. 3.1 for details), and (3) learning the optimal policy x with deep Q-learning, by making use -pmaii- factors and the inferred domain- sharee state representations (lines 4-21). Specifically, because we do not directly observe the states 5;, infer q the kth domain (lines 7 and 13). Moreover, the action-valus function Q is learned by considering the averaged error over the n source domains (line 18). AdaRL can implemented with a wide class of policy-learnin algorithms, e.g., DDPG (Lillicrap et al., 2015), Q-learning (Mnih et al., 2015), and Actor-Critic methods (Schulman et al., 2016; Mnih et al., 2016). Then, in the target domain, we only need to collect few rollouts to estimate the low-dimensional domain-specific representations Parpett with all other parameters being fixed (lines 22-23).
Sttt,, agt,,, 0** and sample 1t+1,k from its posterior, for
Algorithm 1 (AdaRL. with Domains Shifts)
I: Initialize action-value function Q, target action-value function Q', and replay buffer B. Record multiple rollouts for each source domain (k = 1, n)) and estimate the model in Eq. Identify the dimension indices of sqnnn and the values of 0 according to the learned model. for episode- ..... M do for source domain k = I..... do
Receive initial observations 01 and rı, for the k-tt domain.
Infer the posterior q ( 10 11 0min) and sample initial inferred state smin end for
7:
for timestep = 1..,, T do
..... for source domain = I..... do
10:
Select a randomly with probability e; otherwise ar, arg max, (s a, O Execute action 0r,k, and receive reward 1,41, and observation 01+1,, in the kth domain. Infer the posterior (($/41,0004441 ISttIA, ast,A; 0 and sample smin Store transition (st.k" at,k, T++ 1,,, 1 0min) in reply buffer B. end for
12:
13:
14:
mmn
15:
Randomly sample minibatch of N transitions (smin Gi.j. Fi+1.j st41. @miii from B.
16.
Set æı.j = ri+i. + À max. Q*(*4ir a', 0 Update action-value function Q by minimizing the loss:
17:
18:
L
Σ 4
(yi. - Q(073
mii (iij ))
n N
19: end for
20: Update the target network Q': Q'= Q.
21: end for
22. Record few rollouts from the target domain.
23. Estimate the values of omin for the target domain, with all other parameters fixed.
3.3 THEORETICAL PROPERTIES
Below we show the conditions under which we can identify the true graph G from observational data, when the model in Eq. 1 is unknown. Furthermore, we derive generalization bound of the state-value function under the PAC-Bayes framework (McAllester, 1999).
Theorem (Structural Identifiability). Suppose the underlying states s; observed, i.e., Eq. (1) is an MDP. Then under the Markov condition and faithfulness assumption, the structural matrices a - str, otr, Chxts and Betr are identifiable.

---

Published as a conference paper at ICLR 2022
This theorem shows that in the MDP scenario, where the underlying states observed and ch and c are not considered by definition, we can uniquely determine the structural relationships over (s,-1, St; 14-1. r:: 0x i.e., the Dynamic Bayesian network G, from observed data under mild conditions, without knowing the generative environment model. Even if 0 is not directly observed, we can identify which state dimension changes and if there is change in the reward function.
Suppose there are n source domains, and for the kth domain, we have S₁ ((s),es, (S).))). (Smg,k, v" (Sm,,.))) where my is the number of samples from the kth do- main, S., is state sampled from the kth domain, and u (s...) ) is its corresponding optimal state-value. For any value function hag (-) parameterized by 0 we define the loss function ( (Sx, v² (St,)))) = Dusae (hage (8t,k), v² (8t,x)); where Daist is a distance function that measures the discrepancy between the learned value and the optimal value. The following theorem gives generalization bound of the state-value function under the PAC- framework.
Theorem 2 (Generalization Bound), Let Q be an arbitrary distribution over 0 and P the prior distribution over 0 Then for any 5 E (0, 1}, with probability at least 1 5, the following inequality holds uniformly for all Q
(r()) = = E (T((..s)) + 21 (DkI(e1)P) + log 2nmm
+ 2(n-1) (Dxi(Q) P) + log 2n
where er( Q) and ér( S&) are the generalization error and the training error between the estimated value and the optimal value, respectively.
Theorem 2 states that with high probability the generalization error er(Q) is upper bounded by the empirical error plus two complexity terms. Specifically, the first one is the average of the task- complexity terms from the observed domains, which converges to zero in the limit of samples in each domain, i.e., my - oc. The second is an environment-complexity term, which converges to zero if infinite domains are observed, i..., n + cc. Moreover, if assuming different dimensions of 0 are
independent, then DxL (Q)P)) = Σ omi51 DKL (Q,l(),) which indicates that a low-dimensional 0 usually has a smaller KI. divergence, so does the upper bound of the generalization error.
4 EVALUATION
We modify the Cartpole and Atari Pong environments in OpenAI Gym (Brockman et al., 2016). Here, we present subset of the results; see Appendix for the complete results and the detailed settings. We consider changes in the state dynamics (e.g., the change of gravity or cart mass Cartpole, change of orientation in Pong), changes in observations (e.g., different noise levels in images or different colors in Pong), and changes in reward functions (c.g., different reward functions in Pong based on the contact point of the ball), as shown in Fig. 2 for Pong. For each of these factors, we take into account both interpolation (where the factor value in the target domain is in the support of that in source domains), and extrapolation (where it is out of the support w.r.t. the source domains). We train on n source domains based on the trajectory data generated by random policy. In Cartpole, for each domain we collect 10000 trials with 40 steps. For Pong experiments, each domain contains 40 episodes data and each of them takes a maximum of 10000 steps. In the target domain we consider different sample sizes with Niarget = {20, 50, 10000} to estimate Omin For both games, we evaluate the POMDP case, where the inputs are high-dimensional images; note that we did not stack multiple frames, so some properties (e.g., velocity) are not observed, resulting in a POMDP. For Cartpole, we also consider the MDP case, where the true states (cart position and velocity, pole angle and angular velocity) are used as the input to the model. In Cartpole we also experiment with multiple factors changing at the same time (e.g., gravity and mass change concurrently in the target domain).
Modified Cartpole setting The Cartpole problem consists of cart and a vertical pendulum attached to the cart using a passive pivot joint. The task is to prevent the vertical pendulum from falling by putting a force on the cart to move it left or right. We introduce two change factors for the state dynamics f: varying gravity and varying mass of the cart. In terms of changes on the observation function 8g. we add Gaussian noise on the images. Since 0% does not influence the optimal policy (as shown in Prop 1), we need it only for the model estimation, but not for policy optimization. Moreover, if 0 { the optimal policy is shared across domains.
7

---

Published as a conference paper at ICLR 2022
Orignal
Reward
Krrwing diseclone
Agent
Hand-caded
Figure 2: Illustrations of the change factors modified Pong game.
Oracle Upper bound
Non-t lower bound
CAVIA
PEARL
AdaRL" Ours w/o masks
AdaRL
(Zintgral et al., 2019)
(Rakelly et al., 2019)
Ours
2186.1
1098.5
1603.0 (#877.4)
1647.4 (1:617.2)
1940.5 (=:8117)
2217.6
G_in
(4:369.7)
(#:472.1)
(#:911.5)
603.9
204.6
392.0
434.5
439.5
G_out
(+100.6)
(#39.8)
(+125.8)
(=102.4)
(=157.8)
(+138.2)
M_in
2678.2 2
748.5
2139.7
1784.0
1946.2
2260.2
(#:60.5)
(#312.8)
(#859.6)
(+:813.3)
(=196.5)
(#:682.8)
1405.6 6
371.0
972.6
793.9
874.5
1001.7
(#:30800)
(4:2.5)
(#:401.4)
(#:304.2)
(=290.8)
(#273.33
G_in
1984.2
365.0
1012.5
1260.8 e
1157.4
1428.4
& M_in
(#871.3)
(#144.))
(#:64.9)
(#792.0)
(#578.6)
(#-495.6)
G M_out
939.4
689.4
(#270.5)
(=139.6)
(#111..)
(+175.2)
(=181.3)
(+222.33
Table 1: Average final scores on modified Cartpole (MDP) with Ntarget = 50. The best non-oracle results w.r.t. the mean marked in red, while bold indicates statistically significant result w.r.t. all the baselines, and "e" indicates the baseline for which the improvements significant (via Wilcoxon signed-rank test at 5% significance level). G and M denote the gravity and mass respectively, and "*in" and "*out" denote the interpolation and extrapolation, respectively.
AdaRL are statistically
Modified Pong setting In Pong, one of the established Atari benchmarks (Bellemare et al., 2013), the agent controls paddle moving up and down vertically, aiming hitting the ball. We consider changes in observation function 02 state dynamics 0% and reward function Of, as shown in Fig. 2. We consider three change factors on perceived signals ff: different image sizes, different image colors, and different noise levels. For the setting with different image colors, we use RGB images as inputs and consider souree domains with varying RGB colors (original, green, red) and target domains with colors [yellow, )hite), but for other settings, we convert the images to grayscale as input. To change the state dynamics, we rotate the images w degrees clockwise. To test the changes in the reward function, we modify the reward as function of the distance between contact point and the central point the paddle, denoted by d, as opposed to the original Pong in which it is constant (-1 or +1) when the agent the opponent misses the pong. We denote by L the half- of the paddle and formulate two groups of reward functions: (I) Linear reward: rı- L
ad, and (2) Non-linear
aL reward: r d where varies across domains.
compare AdaRL with CAVIA (Zintgraf et al., 2019) and PEARI. (Rakelly et al., 2019). In the POMDP setting, we compare with PNN (Rusu et al., 2016), PSM (Agarwal al., 2021) and MQL (Fakoor al., 2020). We also compare with AdaRL*, version of AdaRL that does not learn the binary masks e and therefore does not use any structural information. All of these methods use the same number of samples Ntarget from the target domain. We also compare with: 1) Non-t, vanilla non non-transfer baseline that pools data from all source domains and learns a fixed model: and 2) an oracle baseline, which is completely trained on the target domain with model-free exploration. For fair comparison, we use the same policy learning algorithm, Double DQN (Van Hasselt et al., 2016), for all methods. opposed to MAML and PNN, AdaRL only uses the Ntarget samples to estimate mmin target> without any policy optimization.
Baselines In the MDP setting,
Results We measure performance by the mean and standard deviation of the final scores over 30 trials with different random seeds, as well as testing the significance with the Wilcoxon signed- rank test (Conover, 1999). As shown in Tables 2 and 2 AdaRL consistently outperforms the baselines across most change factors in the MDP and POMDP case for modified Cartpole, and
In Table 1-3, "e" indicates the haselines for which the improvements of AdaRI. are statistically significant (via Wilcoxon signed-rank test 5% significance level).
8

---

Published as a conference paper at ICLR 2022
Drsele Upper
Nee-t lower hound
PNN
PSM (Agarual al. 2021)
MTQQ (Fakoor 2020)
AdaRI
AdaRt.
(Rusu 2016)
G im
(# 10:12.6)
1930.5
(2:83799 .9)
1031.55
1268.5 (1:69.00) 0)
1439.8 (#:427.6)
(4:66.6)
1517.9
(1-497.5)
1460.6
(4 (+1002.3)
1897.4 4
408.0
69.74
307.9
330.6
298.6
353.4
G_out
(#67.22
(+19.4)
(+11004)
(+92.6)
(+109.8)
(+69.33
(479.6)
2004.9
608.5 *
1600.8
1891.5
1735.6 e
1884.5
1912.6
M_in
(#:404.3)
(#)22)8)
(#-463.6)
(#:638.4)
(#:99.7)
(#:429.7)
(#378.6)
1498.6
216.4
1032.7
862.2
1219.5
1467.5
M_out
(1:625 4)
(#77.3)
(4:368.5 .))
(4:634.0)
(4:300 44)
(#1014.3)
(#:87.2)
N_ia
(+-3086.1)
5640.5
(+207.0)
942.0 m
(+ (+102449)
3952.4
5279.6
(-1999.7)
(4-2464.8)
6927.3
(#2013.6)
55:40.8
(4 (430099.5)
7817,4
Noout
(#:667.)) .3)
4465.2
(#:35.2)
1002.8
(#:344.6)
1137.1
(#511.5)
2740.9
(#637.8)
3298.5
(#:8664)
2018.9
(#841.0)
Table 2: Average final scores on modified Cartpole (POMDP) with Ntarget 50. The best non-oracle results are marked in red. G, M. and N denote the gravity mass, and noise respectively.
Oracle Upper bound
Non-1 lower bound
PNN
PSM
MTQ (Fakour al., 2020)
AdaRL. Ours wio musks
AdaRL
(Rusu et al., 2016)
(Agarwal et al, 2021)
Ours
6.18
9.70 (#:2.09)
15.79 (4:3.26)
(#:2443)
(4:2.43)
(#3385)
(4:1933
(4:2.00)
0
(#1.09)
19.86
(#3.17)
6.40
(#22778)
9,54
(#3322)
10.82 e
(4:4.13)
10.52 e
(42.499)
12.67
(+3380)
15.75
C_in
(50.44)
19.35
(#2.08)
8.53
(#2.37)
14.44
(#1.17)
19.02
(#22002)
16.97 *
(#1.41)
18.52
(+1.05)
19.14
19.78
6.26
14.84
17.66
15:45
17.92
19.03
C_our
(#:0255
(B3345)
(#:1.98)
(#:2.48) 48)
(4:3.30)
(4:1.83
(d:0.97)
S.in
(81.18)
18.32
(12.02)
6,91
(13.25)
11.80
(#3.72)
12.65 *
13.68 * (1:344)
(::3.19)
14.23
(::172)
16:65
S_out
(#1.04)
19.01
(#3.11)
(++4.68)
(+44.61)
(4:2.46)
11.45
(#:2.62)
12.80
(+2.35)
17,82
16.48
5.51
12.73 e
11:30 e
12.67 e (#3.84)
13.78
16,84
N_in
(#1.2
(#3.88)
(#3.67)
(#2.58)
(+2.11)
(#3.13)
18.26
13:24 (2:2.55)
14.85
N amn
(2:1.11
(1:3.19)
(23.15)
(1:2.12)
(1:3.01)
(1:2.24)
Table 3: Average final scores modified Pong (POMDP) with Ntarget 50. The best non-oracle are marked in red. O, C, S, and N denote the orientation, color, size, and noise factors, respectively.
in the POMDP case for Pong for Ntarget = 50. As ablation studies, to see the effect of learning the graphical structure, we also compare with AdaRT., which does not learn the binary masks c', but just assumes everything is fully connected. Learning the graphical structure improves the performances significantly, and without it AdaRL* is generally comparable to baselines. We provide results with Naarget {20, 50, 10000} in Appendix, showing that the performance gains are larger at smaller sample sizes. Furthermore, we consider the change of reward functions (see Table A12-14 14 in Appendix). More detailed experimental results are provided in Appendix, including the average score across different Ntarget. policy learning curves and an analysis of the estimated 02 w.r.t. real change factor. Interestingly, in the Cartpole case, the estimated 6% matches the physical quantities that are being changed across the domains. In particular, the estimated 0g for gravity and noise are linear mappings of the gravity and noise level values. For the mass-varying case, the learned 0% is a nonlinear monotonic function of the mass, which matches the influence of the mass on the dynamics.
5 CONCLUSIONS AND FUTURE WORK
In this paper, we proposed AdaRL, a principled framework for transfer RL. AdaRL learns latent representation with domain-shared and domain-specific components across source domains, uses it learn an optimal policy parameterized by the domain domain-specifie specific parameters, and applies it to a new target domain. It is achieved without any further policy optimization, but just by estimating the values of the with a few target-domain data. As opposed to previous work, AdaRL, can model changes in the state dynamics, observation function and reward function in an unified manner, and exploit the factorization to improve the data efficiency and adapt faster with fewer samples. Further directions include exploiting the target domain to fine-tune the policy and handling the ofitt--iiittibuttion data. Moreover, exploring an altermative to the reconstruction loss, e.g., using the contrastive loss (Laskin et al., 2020), might also improve the training efficiency. Finally, an exciting next step is to transfer knowledge across different tasks, e.g., different Atari games.
parameters in the target domain, which can be accomplished

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Proofs of all our theoretical results are given in Appendix AI, A2, and A3 with disclosure of all assumptions. More details about the model estimation are given in Appendix A4. Appendix A5 provides the complete experimental details and results. All datasets used are publicly available or instructions are provided on how to generate them in Appendix 5 and the hyperparameters and network architectures used is included in Appendix 6. Source code is providing also complete description of our
description of
given at wwwp:luooooo//aaaaeeeei.... experimental environment, configuration files and instructions on the reproduction of our experiments.
ACKNOWLEDGEMENT
KZ would like to acknowledge the support by the National Institutes of Health (NIH) under Contract R0IHL159805, by the NSF-Convergeece Accelerator Track-D award #2134901, by the United States Air Force under Contract No. FA8650- 17-50-177-775,,, and by grant from Apple. BH would like to acknowledge the support by Apple PhD fellowship in AI/ML. Part of the work was performed while BH was interning at the MIT-IBM Watson Al Lab.
REFERENCES
R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare. Contrastive behavioral similar- ity embeddings for generalization in reinforcement learning. In International Conference Learning Representations (ICIR), 2021. URL https://///..c...t.. /owpnrevieeeeeeeeeeee ew net. / fo Forum? iid= qda7-sVg84.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
C. Boutilier, R. Dearden, and M. Goldszmidt. Stochastic dynamic programming with factored representations. Artificial Intelligence, 121(1-2):49-077222 2000.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
I. Clavera, A. Nagabandi, S. Liu, R. S. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt dynamic, real-world environments through meta-reinforcement learning. In International Conference on Learning Representations (ICLR), 2019. URL https: /pwpneeeeee....ee////e//p net/ forum? Beruuiidd-yyzssOC555
W. J. Conover. Practical nonparametric statistics (3rd ed.). John Wiley & Sons, Inc., 1999.
F. Doshi-Velez and G. Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. Proceedings of the Twenty-Fiit International Joint Conference Artificial Intelligence (IJCAI), volume 2016, page 1432. NIH Public Access, 2016.
Y. Duan, J. Schulman, X. Chen, P. I.. Bartlett, I. Sutskever, and P. Abbeel. RI 2 Fast reinforcement learning via slow reinforcement learning. International Conference on Learning Representations (ICLR), 2017. URL /tlttts::/ppeeeeeeeeee
R. Fakoor, P. Chaudhari, S. Soatto, and A. J. Smola. Meta-q-learning. In International Conference on Learning Representations (ICLR), 2020. URL https:/ http:///ppeeevveeeeeeeeeeeeeeeee ew. net. ftrmm/rrmmm id SJJDDCEFPH.
F. Fernández, J. García, and M. Veloso. Probabilistic policy reuse for inter-task transfer learning. Robotics and Autonomous Systems, 58(7):866-871, 2010.
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICMI), pages 1126-1 1135. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2022
A. Ghassami, N. Kiyavash, B. Huang, and K. Zhang. Multi-domain causal structure learning in linear systems. Advances in Neural Information Processing Systems (NewrIPS), 31, 2018.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786::544-077 2006.
Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 983:1735-1770, 1997.
B. Huang, K. Zhang. J. Zhang, J. Ramsey, R. Sanchez-Romero, C. Glymour, and B. Schölkopf. Causal discovery from heterogeneouu/honssaiionnyy data. Journal of Machine Learning Research, 21(89):612-634 6246434, 2020.
S. Katt, F. A. Olichock, and C. Amato. Bayesian reinforcement learning in factored pomdps. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgena Systems (AAMAS), AAMAS '1, page 7-15, Richland, SC, 2019. ISBN 9781 3781450363099.
M. Kearns and D. Koller. Efficient reinforcement learning in factored mdps. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI), volume 16, pages 740-747, 1999.
D. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014. URL https: https::/openree///eee /wpp///pppweew.e///////ee net www.nneefffooii.oommmmm 33x9fd2- 9FyZd.
M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for rein- forcement learning. In International Conference Machine Learning (ICML), pages 5639-5650. PMLR, 2020.
T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning. arXiv preprint (eXiv:1509002977 2015.
D. A. McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual Conference on Computational Learning Theory (COLT), pages 164-170, 1999.
R. Mendonca, A. Gupta, R. Kralev, P. Abbeel, S. Levine, and C. Finn. Guided meta-policy search. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
G. Mesnil, Y. Dauphin, X. Glorot, S. Rifai, Y. Bengio, I. Goodfellow, E. Lavoie, X. Muller, G. Des- jardins, D. Warde-Farley, P. Vincent, A. Courville, and J. Bergstra. Unsupervised and transfer learning challenge: deep learning approach. JMLR W and CP: Proc. of the Unsupervised and Transfer Learning challenge and workshop. 27, 2012.
V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforee- ment learning. Nature, 518(7540):529 533, 2015.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), pages 1928-1937. PMLR, 2016.
K. Murphy. Dynamic bayesian networks: Representation, inference and learning. UC Berkeley, Computer Science Division, 2002.
Pearl and Mackenzie. The Book of Why. Basic Books, New York, 2018. ISBN 978-0-465- 09760-9.
X. B. Peng. E. Coumans, T. Zhang. T.-W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Proceedings of Robotics: Science and Systems (RSS), 07 2020. doi: 10. 00.15677RSS.2000 20.0..00.06...
11

---

Published as a conference paper at ICLR 2022
K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off- polipp meta-reinnorcemmen learning via probabilistic context variables. In International Conference on Machine Learning (ICML), pages 5331-5340. PMLR, 2019.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:160.00460 2016.
B. Schölkopf. Causality for machine learning, arXiv preprint 1r..i::111..100000 2019.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Schulman, P. Moritz, Levine, M. Jordan, and Abbeel. High-dimension continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), 2016.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanetot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, 2016.
A. L. Strehl, C. Diuk, and M. L. Littman. Efficient structure learning in factored-state mdps. In Proceedings of the Twenty-second AAAI Conference on Artificial Intelligence (AAAI), volume pages 645-650, 2007.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
A. Tamar, Y. Glassner, and S. Mannor. Optimizing the evar via sampling. In Proceedings of the Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI), 2015.
M. F. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7). 2009.
M. E. Taylor, P. Stone, and Y. Liu. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 8(1):2225-267,, 2007.
A. Tirinzoni. A. Sessa, M. Pirotta, and M. Restelli. Importance weighted transfer of samples in reinforcement learning. In International Conference Machine Learning (ICML), pages 4936 4945. PMIR, 2018.
A. Tirinzoni, M. Salvini, and M. Restelli. Transfer of samples in policy search via multiple importance sampling. In International conference on machine learning (ICML), pages 6264-6274. PMLR, 2019.
M. Tomar, A. Zhang, R. Calandra, M. E. Taylor, and J. Pineau. Model-invariant state abstractions for basel-based reinforcement learning. In Self-SSupervision for Reinforcement Learning Workshop ICLR 2021, 2021. URL https: /opnnevvii ew net. forum? d=01 w4 8Wvlmic C.
H. Van Hasselt, Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence (AAAI), volume 30, 2016.
W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Proceedings of Robotics: Science and Systems (RSS), Cambridge, Massachusetts, July 2017. doi: 10. 15507/RSS.2017...0 XIIL004.
A. Zhang, C. Lyle, S. Sodhani, A. Filos, M. Kwiatkowska. J. Pineau, Y. Gal, and D. Precup. Invariant causal prediction for block mdps. In International Conference on Machine Learning (ICML), pages 11214-11222 PMLR, 2020a.
A. Zhang, R. T. McAllister. R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Repre- sentations (ICLR), 2021a. URL htttts:://ppereviieeeeeeee
-
12

---

Published as a conference paper at ICLR 2022
A. Zhang, S. Sodhani, K. Khetarpal, and J. Pineau. Learning robust state abstractions for hidden- parameter block (mdp)s. Js. In International Conference on Learning Representations (ICLR), 2021b. URL https: wwtp://ieeereeeeeeeeee.e.. net/ forumm2dd=ff00222333
Zhang and P. Spirtes. Intervention, determinism, and the causal minimality condition. Synthese, 182(3):335-347, 2011.
K. Zhang, M. Gong, P. Stojanov, B. Huang, Q. Liu, and C. Glymour. Domain adaptation problem of inference on graphical models. Advances in Neural Information Processing Systems (NewrIPS), 33:4965-4977, 2020b.
Z. Zhu, K. Lin, and J. Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.0785 2020.
L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta- learning. In International Conference on Machine Learning (ICML), pages 7693-7702, PMLR, 2019.
13

---

Published as a conference paper at ICLR 2022
APPENDIX FOR
"ADARL: WHAT, WHERE, AND How TO ADAPT IN TRANSFER REINFORCEMENT LEARNING"
Appendix organization:
Appendix A1: Proof of Proposition
Appendix A2: Proof of Theorem 1
Appendix A3: Proof of Theorem 2
Appendix A4: More details for model estimation
Appendix A5: Complete experimental results
Appendix A6: Experimental details
A1 PROOF OF PROPOSITION
We first review the definitions of d-separation, the Markov condition, and the faithfulness assumption (Spirtes et al., 1993; Pearl, 2000), which will be used in the proof.
Given directed acyclic graph G (V, where is the set of nodes and E is the set of directed edges, we can define graphical criterion that expresses a set of conditions on the paths.
Definition Al (d-separation (Pearl, 2000)). A path p is said to be d-separated by a set of nodes Z C V if and only (1) contains a chain node m is in Z, or (2) p contains a collider i + m + j such that the middle node m is not in Z and such that descendant of m is in Z.
fork i +- m + j such that the middle
Let X, Y. and Z be disjunct sets of nodes. Z is said to d-separate X from (denoted as X La Y if and only Z blocks every path from node in X node in Y.
Definition A2 (Global Markov Condition (Spirtes et al., 1993; Pearl, 2000)), A distribution P over V satisfies the global Markov condition on graph G for any partition (X, Z,, Y) such that X La VIZ
P(X, Z = P(YI))
In other words, X is conditionally independent of given Z, which we denote as X JL Y!Z. Definition A3 (Faithfulness Assumption (Spirtes et al., 1993; Pearl, 2000)). There no indepen- dencies between variables that are not entailed by the Markov Condition.
If assume both of these assumptions, then the conditional independences from given DAG G. In particular, for any disjoint subset of nodes X,Y,Z V: X IL Y - X La Z
d-separation as criterion to read all of
In our case we can represent the generative model in Eq. as a Dynamic Bayesian (DBNN (Murphy, 2002) over the variables {s; a1-1 Ot-1, Tz, S,, Ox), where the binary masks c represent edges or sets of edges, as shown in Fig. A1. As is typical in DBN we assume that the graph invariant across different timesteps. We assume 0₂ are constant across the different timesteps. We add to the image also the cumulative reward R. In practice we will focus instead on the cumulative future reward Re+1 Ex= 7 - which only considers the contributions of the r, in the future with respect to the current timestep t.
In order to prove Proposition I, we first need to prove that the compact shared representations spain and compact shared representations @min are all the state and change factors dimensions, respectively, that are conditionally independent of a, given the future cumulative reward R even given all other variables:
14

---

Published as a conference paper at ICLR 2022
Domai-- spectic poramaters
%
%
C
p-1
911
chared
Iepresmration
(Latent
roward
Time al0o
Time silee
Figure A1: Graphical representation of the generative model in Eq. 1. In the top of the figure, the square boxes are the domain-specific parameters Or, which are constant in time, while the rectangular boxes represent the binary masks that encode the edges. R represents the cumulative reward
Lemma A1. Under the assumption that the graph G is Markov and faithful the measured data, state dimension Si,e E St is part of sqinn iff:
8i,t V. Oel R 82 v3, S {s+ (iit),
Similarly, change factor dimension 02,0 On is part of Ogin iff:
0g,k V4 orl Ret1,h,, vI, S (st, (0k 0c,)).
Proof. We split the proof in two parts, the "only if" and the "if" part:
"If conditionally dependent a given Ri+1 then in compact representation ":
We first show that s,e satisfies the conditional dependence st, K a Re+1, 3t, va: C {s; sta). then it is part of st smin i.e. $i, either has an edge to the reward in the next time-step 1r+ or, recursively, it has edge to another state component in the next time-step 8j,-1, such that the same component at time step f, $j,t+1 E strin, Note that this recursive definition collects all of the states &t. that have an effect on future reward rt+7, 7 (I,..., ... f), either directly, or through the influence on other state components. Since all of these r+++ are influencing the cumulative future reward Re+1, all of these components have an edge to R&+1 as well. We prove it by contradiction. Suppose that si. JL ar Ri does not have direct or indirect path to Tr+-r i.c. 8i, E min By assumption a, only affects future state S1+1, so 8i,t 1 ar- Then, according to the Markov and faithfulness conditions, $1, is independent of at conditioning
Re+1, since there is path that connects $, - ..... > Tt+7 - R&+1 + Te+1 +- a, on which R&+1 is a collider (i.e. a variable with two incoming edges), which is the only path which could introduce conditional dependence. This contradicts the assumption.
Similarly we show that if if, C (sr, (Ox \ } the change factor dimension 02, Jí. orl Rr|Re+ 1, J.. then 0i E 0min which similarly to previous case means it has direct or indirect effect on TeEr and therefore R441- By contradiction suppose that 0., 11 R441. 1 for all previously defined i. but it is not change parameter for the reward function 0, 0% with ch *r 1. nor is change parameter for the state dynamics 0 E 0i with a direct or indirect path to r for T = 1...., ......,T T t. By assumption of our model, 0g. is never connected to a, directly, nor they might have common cause, so 0, 1 az- Then, according to the Markov condition, 0g is independent of a, conditioning on R441, which contradicts the assumption, since:
if 0i.k & 0% or c.-r = 0, then means there no path 0 - S1,tt + TE+T + R f Tet1 04 for any N that would be open by conditioning R
15

---

Published as a conference paper at ICLR 2022
if 8i E 0% but there is no directed path to retr for any 2 1, i.. then there is also directed path 0e, + .... + r1+T 7 Re+1 f- T+4 + ae that would be open when we condition on R4+1-
"If in compact representation then conditionally dependent on given Re+1"* ":
We next show that if $i, E s, which mean St, has direct or indirect edge re+r: f1...., T - f}, then $₁ satisfies the conditional dependence Si, H ar Re/Re+1 St, V&, S (s, $(,)- We prove it by contradiction. Suppose 81, has directed path to r++r and Si,t is independent on as given R and a subset of other variables & S S, \ 8i,t- Since we assume that there are no instantancous causal relations across the state dimensions, st,t #t Rr|R there can never be an Sjtt such that St,t 1 ag R Sj,t. In this case, this means that 8₁, IL as Re+1 has to hold. Then according to the Markov and faithfulness assumptions, 81, cannot have any directed path any VT+ 2 1, because that any such path create a v-structure in the collider Re+1, which would be open we condition on R contradicting the assumption.
Similarly, suppose 0i,k is dimension in 0₂ that has a directed path to rx+r. We distinguish two cases, and show in neither can 0i. be independent of a, given Re+1 and subset of the other variables:
if 0i, E 0% then it cannot be independent of a, when we condition on Ro+1, which is descendant Tz+1 and therefore opens the collider path 0g 4 Tt+1 + dz;
if 0i, E 0f. then at timestep it is always only connected to the corresponding 8i, So there is directed path T++r, it has go through $1,t- While cannot be blocked by any subset of {Ox J. it can be blocked by conditioning on there are infinite future paths with the same structure, e.g. through Si,++11 I that will not be blocked by conditioning only on variables at timestep L. Under the faithfulness and Markov assumption this means that f,. cannot be independent from a, by conditioning on any subset of state variables at timestep or any other change parameters, which is a contradiction.
We prove our main proposition:
Proposition 1. Under the assumption that the graph G is Markov and faithful the measured data, the union of compact ceimain-speciffe 0% and compact shared representations smin are the minimal and sufficient dimensions for policy learning across domains.
Proof. As shown in the previous lemma, in compact seaaainngeneerlizabble representations spain every dimension is dependent on de given Re+1 and any other variables, and every other dimension is independent of a given R++1 and some other variables. Furthermore, because every dimension that is dependent de is necessary for the policy learning and every dimension that is (conditionally) independent of a, for at least a subset of other variables is not necessary for the policy learning, com- pact domain -spccffic @m and compact shared representations smin contain minimal and sufficient dimensions for policy learning across domains. Note that the agents determine the action under the condition of maximizing cumulative reward, which policy learning aims to achieve, so we always consider the situation when the discounted cumulative future reward R4+1 is given.
A2 PROOF OF THEOREM
Theorem (Structural Identifiability.. Suppose the underlying states sı observed, i.e., Eq. (1) an MDP. Then under the Markov condition and faithfulness assumption, the structural matrices s a ,e Cote Cb and &etr are identifiable.
Proof. We concatenate data from different domains and denote by k be the variable that takes the domain index 1,... n. Since the data distribution changes across domains and the change is due to the unobserved change factors 0₂ that influence the observed variables, can represent the change factors as a function of k. In other words, we use the domain index k as a surrogate variable to characterize the unobserved change factors.
16

---

Published as a conference paper at ICLR 2022
011
1,11
20
0
92
01
$12-1
5/11
$1213
02
621
8
92,+1
$3.
93,41
4-1
4
4++
(a) An example of a ground Bayesian network (unrolled DBN over time).
K
(C) $1,1 is a compact domain -spcctccc represen- tation, since there exists path a, that is d- connected when condition the collider R. Similarly 01 d-connected a when condi- tioning on R.
(b) $2,1 is not compact domain domain-specific represen- tation, since there is no directed path any f*4r. i.c. 82,t ae R. Similarly, there is directed path from 81.6 to
Figure A2: Example model in which s, and $3,r are compact domain-specific representations for policy learning, while 82,, not. This does not that 82, and 02 useful in the model estimation part, especially in estimating og.
17

---

Published as a conference paper at ICLR 2022
We denote the variable set in the system by V, with V = {11,1-11- ................................ 1..--.....,.d.-1111 $1,1........... Sd.1, a: 7 and the variables form a dynamic Bayesian network G. Note that in our particular problem, according to the generative environment model in Eq. (1), the possible edges in G are only those from 8i.-1 E 8 to *j E 8t, from 04-1 to 8j, E 8t, from 8i,-1 E 8 to rt. and from 0t-1 to rt- We further include the domain index k into the system characterize the unobserved change factors.
has been shown that under the Markov condition and faithfulness assumption, for every V, V, E V. V2 and V are not adjacent in if and only if they are independent conditional on some subset of {Vil(= & i, + j} U k (Huang et al., 2020). Thus, we can asymptotically identity the correct graph skeleton over V.
Moreover, since we assume a dynamic Bayesian network, there the direction of an edge between a variable at time to one at time - 1 fixed. Therefore, the structural matrices O ca extr and ea*V, which are parts of the graph & over V, are identifiable
Furthermore, we want show the identifiability of C and ed that is, to identify which distribution modules have changes. Whether a variable V, has a changing module is decided by whether Vi and are independent conditional on some subset of other variables. The justification for one side of this decision is trivial. If V4 module does not change, that means P(V. PA') remains the same for every value of k, and so Vi IL k PA'. Thus, if V, and are not independent conditional on any subset of other variables. Vi's module changes with k which is represented by edge between V; and k. Conversely, we assume that if V,'s module changes, which entails that V; and are not independent given PA' then V/2 and & are not independent given any other subset of (V(Vi. If this assumption does not hold, then we only claim to detect some (but not necessarily all) variables with changing modules.
A3 PROOF OF THEOREM
In this section, we derive the generalization bound under the PAC-Bayes framework (McAllester, 1999; Shalev-Shwartz and Ben-David, 2014), and our formulation follows (Pentina and Lampert, 2014) and (Amit and Meir, 2018). We assume that all domains share the sample space Z, hypothesis space H. and loss function é H x Z - [0, 1j. All domains differ in the unknown sample distribution Es parameterized by m associated with each domain k. We observe the training sets Si...., Sn. corresponding to n different domains. The number of samples in domain is denoted by mg. Each dataset S₁ is assumed to be generated from unknown sample distribution Ex We also assume that the sample distribution E& are generated i.i.d. from an unknown domain distribution т. More specifically, we have S₁ (21,k;- ..., Zi,ks..., Zm,, ,)), where zi,A- = (8.,;; v (st,)), )). Note that, Si,k is the i-th state sampled from k-th domain and v" (8i.a) is its corresponding optimal state-value. For any value function hgme(-) parameterized by 0 we define the loss function E(hggnn ((,k) (hiie (Si,k); u² (si,k)), where Daist is distance function that measures the discrepancy between the learned value and the optimal state-value. We also let le be the prior distribution over H and Q the posterior over H.
Theorem 1. Let Q be arbitrary distribution over 0pin and P the prior distribution over O Then for any E (0, 1), with probability at least 1 - 5, the following inequality holds uniformly for all Q
1
2nmn
(r())=
-
ér Sk) +
DKL(OII)) + +log
2(m - 1)
8
k=1
1
(DKL(O)P) log 2n
+ V 2(n 1)
where er( Q) and ér (Q, S&) are the generalization error and the training error between the estimated value and the optimal value, respectively.
Proof. This proof consists of two steps, both using the classical PAC- AAC-Bayes bound (McAllester, 1999; Shalev-Shwartz and Ben- Den-David, 2014). Therefore, we start by restating the classical PCA-Bayes bound.
18

---

Published as a conference paper at ICLR 2022
Theorem A1 (Classical PAC-Bayes Bound, General Notations). Let X' be sample space, P(X) a distribution over X. a hypothesis space. Given a loss function E(B, X) è x + [0. and a collection of M i.i.d random variables (X1...., XM) sampled from P(X), let be prior distribution over hypothesis in Q Then, for any 8 E (0, 1), the following bound holds uniformly for all posterior distributions p over 6
P X))P(X))
1(0,
Σ
E 1000 0mp
1 V 2(M - 1)
M (DKL(ellm) log
Vp
1 1 6.
Between-domain Generalization Bound First, we bound the between-doman generalization, i.c., relating er er(@) to er( Ex).
We first expand the generalization error as below,
er(2)
E
E Em
E
E k~Q(S.0)
E
((h,2)
(E,m)~r
0~Q
E
E
E 2(0, E)
(E, mEm
Em m
E
E Smm
er (Q, E).
(A1)
(E,m)or
Then we compute the error across the training domains,
E
E
E
llh, 2) = Σ er(@, Ek).
(A2)
Q~
0)
B
Then Theorem A1 says that for any 6o ~ (0, 1}. we have
1 Σ ex) Σ
+ V 1 (DKL(Q1PP) log n
1
") 2 5p,
)
P
er(
(A3)
8p
k=1
where P is a prior distribution over O.
Within-domain Generalization Bound Then, we bound the the i.., relating er(2, F&) to (Q,
generalization,
We first have
er cr(Q,
Q Ex) = Q~
E h=Q(SE,))
E E
E(h, 2).
(A4)
Then compute the empirical error across the training domains,
cr(2, Sk) =
E(h, 21.))
(A5)
j=1 h-)
According to Theorem A1, for any 8D ~ (0. 11, we have
P cr( Ea) ( (2)
V 2(m -) (DKL(OlIT) + log
1
2 1
8z.
(A6)
With the choice of f ((O)Q(SD. 0)d0 and p (()) 0)d0, we have that (Kl(pl(T) & (2 (Yin et al., 2019). Thus, the above inequality can be further written as,
my )
P {
< (e( + 2(ma -
(xi(211)) + log
2 1 - Sr. (A7)
0z
19

---

Published as a conference paper at ICLR 2022
Overall Generalization Bound Combining Eq. (A3) and (A7) using the union bound and choosing that for any 6 > 0, set 60
2 and 85 = 2n for l = 1..... n, then we finally obtain,
P
1 n er( Q) S 11
Σ
ér Sk)) +
n
V 2(mix - 1] (DKL(QIIP) + log
2nmk
+V 2(n - (DiL(QUI)) + log 2n
))
2 1 - 8. (A8)
8
A4 MORE DETAILS FOR MODEL ESTIMATION
A4.1 LOCATING MODEL. CHANGES
In real-world scenarios, it is often the case that changes to the enviromment sparse and localized. Instead of assuming every function to change arbitrarily, which is inefficient and unnecessarily complex, we first identify possible locations of the changes. To this end, we concatenate data from different domains and denote by k the variable that takes distinct values ... n to represent the domain index. Then, we exploit (conditional) andependennies/deeenneneii to locate model changes. These (conditional) independenncis//epeeneeedd independence tests (Zhang et al., 2011), which allows for both linear or nonlinear relationships between variables, Below we show that in some cases, we can identify the location of 0,., by using the conditional independence relationships from concatenated data.
can be tested by kernel-based conditional
Proposition A1. In POMDP, where the underlying states are latent, we can localize the changes by conditional independence relationships from concatenated observed data D in the following cases:
C₁ if or 1 k, then there is neither a change in the observation function nor in the state dynamics for any state that is an imput to the observation function;
C2: if of 1. I and a, V.. k|re+1, then there is only a change in the reward function;
C2: if at 1 klret,, then there is neither a change in the reward function nor in the state dynamics for any state in smin;
C. aa 1 klret1 and or 16 k. then there is a change in the observation function, there exists a state that is not in mii but is an input to the observation function, whose dynamics has a change.
Proof. We first formulate the problem as follows. We concatenate data from different domains and use domain-index variable & to indicate whether the corresponding distribution module has changes across domains. Specifically, by assuming the Markov condition and faithfulness assumption, S;, has an edge with & if and only if p(s..el (PA(s..).. changes across domains, where PA(-) denotes its parents. Similarly, rt has an edge with & if and only if p(r, PA(r:) changes across domains, and o has an edge with A if and only if p(or) [o:]PA((e (o:))) changes across domains. Under this setting, locating changes is equivalent to identify which variables have an edge with & from the data.
We consider the scenario of POMDP, where we only observe {oe, Te, ) and the underlying states st are not observed. Below, we consider each case separately.
Case 1: Show that if or L k, then there is neither change in the observation function nor a change in the state dynamics for any state that is an input to the observation function.
We prove it by contradiction. Suppose that there is change in the observation function and a change in the state dynamics for any state that is an input to the observation function. That is, og has an edge with k. and s,, that has direct edge oR also connects with k. Based on faithfulness assumption, or JL k which contradicts to the assumption. Since we have a contradiction, it must be that there is neither change in the observation function nor change in the state dynamics for any state that is an input to the observation function.
Case 2: Show that if ot 1. k and at H. klret1, then there is only change in the reward function.
20

---

Published as a conference paper at ICLR 2022
If ot 1 l and ac-1 t k/rr, based on the Markov condition and faithfulness assumption, ra has an edge with k, and S1, and or do not have edges with k; that is, there are only changes the reward function.
Case 3: Show that if a, 1 k|r++1, then there is neither a change in the reward function nor a change in the state dynamics for any state in smi.....
By contradiction, suppose that there is a change in the reward function or there exists a state Sj, E 87 that has change in its dynamics. That is, r, has an edge with & or corresponding 8j, has an edge with k. Based on faithfulness assumption, dt M. k|r++1, which contradicts to the assumption.
Case 4: Show that if a L k klr+1 and o J. k, then there a change in the observation function, or there exists state that is not in sminn but is input to the observation function, whose dynamics has a change.
According Case 3, if at 1 A then there is neither change in the reward function nor change in the state dynamics for any state in gminn Furthermore, since or J. k, then based on the Markov condition, either o, has an edge with k. there exists a state that not in smain but is an input to the observation function, whose dynamics has an edge with k. That is, there is a change in the observation function, or there exists state that not sman but is an input to the observation function, whose dynamics has a change.
Based on Theorem and Proposition A1, in MDP, we can fully determine where the changes are, so only need to consider the corresponding of² to capture the changes. In POMDP, in Case only need to involve of and fr model estimation, that is, 0₁ = {0%, 0 }: in Case 2, 0 = { ); and in Case 3 & 4, 0₁ {09. 0% f. For other cases, we involve 0₂ 10% 0% 0% in model estimation.
A4.2 MORE DETAILS FOR ESTIMATION OF DOMAIN- VARYING MODELS IN ONE STEP
We use MiSS- VAE to learn the environment model, which contains three components: the "sequential VAE" component, the "multi-model" component, and the "structure" component. Figure A3 gives the diagram of neural network architecture in model training.
Specifically, for the "sequential VAE" component, we include a Long Short-Term Memory (LSTM (Hochreiter and Schmidhuber, 1997)) encode the sequential information with output he and Mixture Density Network (MDN (Bishop, 1994)) to output the parameters of MoGs, and thus to VI.t,k, 01:1-1,A:
learn the inference model 4s (s 1.K- the output. The generated sample further acts as an input to the decoder, and the decoder outputs ōe+1 and f4+2. Moreover, the state dynamics which satisfies a Markov process modeled with an MLP and MDN.
1 0&) and infer a sample of s from qa as
For the "multi-model component, we include the domain index as an input to LSTT and involve O2 as free parameters in the inference model ga. by assuming that O also characterizes the changes in the inference model. Moreover, we embed 0g in state dynamics py, 0% in observation function and of in reward function in the decoder. With such characterization, except 0g, all other parameters are shared across domains, so that all we need to update in the target domain is the oow-dimensional 0 which greatly improves the sample efficiency and the statistical efficiency in the target domain- -umain- few samples are needed.
For the "structure" component, the latent states are organized with structures, captured by the mask Cs*a, Also, the structural relationships among perceived signals, latent states, the action variable, the reward variable, and the domain-speciff factors are embedded free parameters (structural vectors and scalars " c da cr and ca²r) into MiSS-VAE.
A5 COMPLETE EXPERIMENTAL RESULTS
In this section provide the complete experimental results both of our settings, modified Cartpole and modified Pong in the OpenAl Gym (Brockman et al., 2016). In the POMDP setting, we use images as input, which for Cartpole look like Fig. A4(a) and for Pong look like Fig. A8(a). For
21

---

Published as a conference paper at ICLR 2022
fou (t [oesa Re +2 Preauree 4:2)
Beat
(0
Encoder
for, Te 0
(osss: Tes +
forsz Peeeitruaaseeii
LSTM (Puws)
LSTM (Barr)
LSTM
(Buat)
Encoder
+
for, A+
( Pe R)
(0 Teaz, ac-t, A))
hy
LSTM
(9t)
(0a)
(9L)
Encoder
MDN
MDN
MDN
Dynamios
k+1
MUP 01
MBN
S
+5
0g
Gp-1
Deconv (0g.
Decoder
(8%.9))
k
f141
Gr+1.
02-2: Pr43
Figure A3: Diagram of MiSS-VAE neural network architecture. The "sequential VAE" compo- nent, "multi-model" component, and "structure" component are marked with black, red, and blue, respectively.
Cartpole, we also consider the MDP case, where the true states (cart position and velocity, pole angle and angular velocity) used the input to the model.
We consider changes in the state dynamics (e.g., the change of gravity or cart mass in Cartpole, or the change of orientation in Atari), changes in perceived signals (e.g., different noise levels observed images in Cartpole, as shown in Fig. A4 or colors in Pong) and changes in reward functions (c.g.. different reward functions Pong based on the contact point of the ball), as shown Fig. A8 for Pong. For each of these factors, we take into account both interpolation (where the factor value in the target domain is the support of that source domains), and extrapolation (where it is out of the support w.r.t. the source domains).
We train on n source domains based on the trajectory data generated by random policy. In Cartpole experiments, for each domain we collect 10000 trials with 40 steps. For Pong experiments, each domain contains 40 episodes data and each of them takes maximum of 10000 steps.
A5.1 COMPLETE RESULTS OF MODIFIED CARTPOLE EXPERIMENT
The Cartpole problem consists of a cart and a vertical pendulum attached to the cart using passive pivot joint. The cart can move left or right. The task is to prevent the vertical pendulum from falling by putting force on the cart to move it left or right. The action space consists of two actions: moving left or right.
We introduce two change factors for the state dynamics 0f: varying gravity and varying mass of the cart, and a change factor in the observation function 0 that is the image noise level. Fig. A4 gives visual example of Cartpole game, and the image with Gaussian noise. The images of the varying gravity and mass look exactly like the original image. Specifically, in the gravity case, we consider source domains with gravity g {5, 10, 20, 30, 40J. We take into account both interpolation (where the gravity in the target domain in the support of that in source domains) with = {15). and extrapolation (where it is out of the support w.r.t. the source domains) with g (55). Similarly, we consider source domains where the mass of the cart is m = (0.5, 1.5, 2.5, 3.5, 4.5). while in target domains it is (1.0, 5.5). In terms of changes the observation function 0g. add
22

---

Published as a conference paper at ICLR 2022
(a)
(b)
Left
Right
Figure A4: Visual examples of Cartpole game and change factors. (a) Cartpole game; (b) Modified Cartpole game with Gaussian noise the image. The light blue arrows are added to show the direction in which the agent can move.
A
SERNE
tagan
Eagul
Gruvity
Mass
Noise
Figure A5: The estimated O for the three change factors in Cartpole (POMDP); gravity (A), mass (B), and noise level (C) for Ntarget = 50.
Gaussian noise on the images with variance a = {0.25, 0.75, 1.25, 1.75, 2.25) in source domains, and (0.5, 2.75) in target domains.
We summarize the detailed settings in both source and target domains in Table A1. In particular in each experiment we all source domains for each change factor and one of the target domains at time in either the interpolation and extrapolation set.
Gravity
Mass
Noise
Source domains Interpolation set Extrapolation
{5, 10, 20, 30,40) 40}
(0.5, 1.5. 2.5, 3.5, 4.5)
(0.25, 0.75, 1.25, 1.75, 2.25)
115]
[1.0]
(0.5)
{55)
(5.5)
(2.75)
Table A1: The settings of source and target domains for modified Cartpole experiments.
A5.1.1 LEARNED 0₃ IN CARTPOLE EXPERIMENTS
Fig. A5 shows the estimated Og in the modified Cartpole experiments. For the gravity and mass scenarios, the learned parameters with different sample sizes are close with each other. This phe- nomenon indicates that even with only a few samples (Ntargez change parameters very well. For the noise level factor, the learned curves with different sample sizes have a similar behaviour, but the distance is larger. We can see that the 0g we learn is approximately linear function of the actual perturbation in gravity, while for the mass and noise monotonic functions.
50), AdaRL can estimate these
1..1.2 AVERAGE FINAL SCORES FOR MULTIPLE Ntarget IN CARTPOLE EXPERIMENTS
Tables A2 and A3 show the complete results of the modified Cartpole experiments (POMDP settings)
for Nargget 20 and Ntarget 10000. Table A4 and A5 give the complete results of the modified
23

---

Published as a conference paper at ICLR 2022
Onslce Upper
Nen-l
PNN
PSM
MTQ
AdaRL
AdaRE.
6_in
(1:1042.6)
1830.6 5
(4 (1.509.3) 4)
$28.5 e
(1:71922)
(:14533.0) 6)
1008.5
(4:503.5)
1257.2
(=:589.1
1280.6
(4874.0)
1302.7
408.6
54.0
1
156.8
120.5
173.8
198.4
(#177.2
(+13.6)
(#:24.2)
(#:49.5)
(4:7.4)
(4:99.3
(404.6)
M_in
(#404.3)
2004.8 8
(#:3.6)
447.2
(4348.1)
11200.6
(#:363.2)
9822.
(#274.0)
1246.7
(#521.3)
1056.8
(#:327.3)
1361.0
1498.6
830.6
1082.5
(=025.4)
(#:9.8)
(#:2114)
(#:11722
(#202.5)
(#330.9)
(#236.3)
N
(1:3066.)
86440.5 5
(4:283.5)
679.4
(#:2202.2)
4170.6
(#: (6.1604.)
4936 .
(4:2387.4)
3985.7 e
(4:2627.8)
4954.3 e
(4:2441.5)
$761.2
N_out
(#:667.3) .3)
(4: 429.2)
(4:385.2)
2841.5
(4-4533.6
(#:277.9)
2654.0
(#:47003)
1785.2
(+:293.3)
Table A2: Average final scores in modified Cartpole (POMDP) with Ntarget 20. The best non-oracle results are marked in red. G, M, and N denote the gravity, mass, and noise respectively.
Onclle Upper bound
Non-t Jower bound
PNN
PSM (Agarwal 2021a)
MTQ
AdaRE Ours w/o masks
AdaRT.
(Rusu at.l., 2016)
(Fakoor et al., 2020)
Ours
1930.5 5 (#:1042.6)
1115.2 (1:41.8)
1637.44 (1:378.2)
1838. (d:358.1)
1439.2 (1:666.5)
1884.5 (=199411
1924.6
(1874.0)
G_out
(+:67.2)
408.6
(46659)
161.3
(#-48.9)
329 2293.G
(4:138.5)
457.3
(+-785)
303.2
(#110377
384.2
(+192.3
410.6
2004.8
(=404.3)
(#373.4)
(#:42.9)
(::493.0)
(#:378.2)
(=309-5)
(#:03.44
M out
(2625.4) .4)
1498.6 6
(::14633)
325.6
(1:394.7)
1206 8
(::5200.5
1339 4
(#.773.1)
1296.2 e
(::411:2
1297.4
(::596.2)
1496.3
N_in
(4:3086.1) 1)
8640.6
(4-80.5)
(+-1132.9)
6476.2
(4-1981.0)
(4-2389.0)
7932.9
(4-2915.3)
7382.4
(4-2356.0)
N X_oul
44663.2 2 (=067.3)
(#:341.8)
962.5
(1:00))))
30431.9
(#177233
2987 :2
(#763.0)
3892.4
(=782.2)
4183.0
(#332.4)
4245. 22
Table A3: Average final scores in modified Cartpole (POMDP) with Ntarget = 10000. The best non-oracle results marked in red. G, M, and N denote the gravity, mass, and noise respectively.
Oracle Upper bound
Non-t lower bound
CAVIA (Zintgral et al., 2019)
PEARL (Rakelly et al., 2019)
AdaRL* Ours w/o masks
AdaRL
Ours
2486.1
1651.5 (#623.8)
1720.3 (+589.4)
G_in
(+:369.7
(#368.5)
(=393.6)
(#765.4)
693.9
243.8
356.2
362,
292.4
395.6
G_out
(#:100.6)
(d:15.2)
(#:66.3
(=57.3)
(#:91.8)
(#101.7)
2678.2
480.3
1306.8
1589.4
1624.8
1962.0
(#:630.5
(#136.2)
(#:376.5)
(::682.33
(=531.6)
(#:652.8)
1405.6
306.5
853.2
969.4
984.6
1113.5
(#:66 0)
(= (+162.4)
(#:11.0))
(#:238.5)
(=209.8)
(+:9442)
1984.2
374.9
1174.3
964.3
1209.6
& M_in
(+871.3)
(=126.8)
(#298.2) 2)
(#370.5)
(=425.7)
(+392.6)
GLo
939.4
292.4
494.6
368.4
399.8
531.2
M,
(1.270.5)
(2:127.6)
(1:201.3)
(::259.8)
(=:42.5)
(+:272.5)
Table A4: Average final scores in modified Cartpole (MDP) with Ntarget = 20. The best non-oracle
results are marked in red, while bold indicates statistically significant result w.r.t. all the baselines. G, M. and N denote the gravity, mass, and noise respectively. indicates the baselines for which the improvements of AdaRL statistically significant (via Wilcoxon signed-ram rank 5% significance level).
24

---

Published as a conference paper at ICLR 2022
Oracle Upper bound
Non-4 lower bound
CAVIA (Zinigral et al., 2019)
PEARL
AdaRL Ours w/o musks
AdaRL
(Rakelly et al., 2019)
Ours
2486.1
986.3
1907.4 (#526.8)
2102.3 (#:398.5)
1864.0 (=369.2)
2365.1
G_in
(#:00977 7)
(#392.5)
(+:103.5)
093.9
349.2
502.9
585.7
494.7
604.8
G_out
(+11006)
(#72.00
(#133.2)
(=98.6)
(=151.4)
(#1117.6)
2678.2
643.9
2008.6
2106.2
2148.9
2415.2
M_in
(#:30.5)
(#281.3)
(#-436.2)
(#-436.7)
(=387.2)
(#:591.4)
1405.6
617.4
1182.7 e
1294.5
1207.5
1263.5
M_
(4308.0) 0)
(#145.3)
(#255.8)
(#2100.)
(=251.3)
(#302.9)
G_in
1984.2
452.6
1275.0
1468.7 e
1395.4
1589.4
& M_in
(#871.3)
(#178.3)
(#-132.5)
(#:097.2)
(=387.2 2)
(+:399.5)
939.4
709.5
& 2M out
(4:270.5)
(#: (#:17755)
(#:386.0)
(#:20099)
(=263.8)
(#:208.4)
Table A5: Average final scores in modified Cartpole (MDP) with Norggt = 10000. The best non- oracle results are marked in red, while bold indicates statistically significant result w.r.t. all the baselines. G, M, and N denote the gravity, mass, and noise respectively.
Cartpole experiments (MDP settings with symbolic input) for Niarget 20 and Niarget 10000. We average the scores across 30 trials from different random seeds during the policy learning stage. The results suggest that, most cases, AdaRL can outperform other baselines.
A5.1.3 AVERAGE POLICY LEARNING CURVES IN TERMS STEPS
Gravity: 15
Gravity: 55
Mass: 1.0
5000
3301
1500
2000
200
1000
300
240
Scores
200
3081
-0.1
-002
Mase: 5.5
Noise: 0.5
Noise: 2.75
1000
2004
21003
1000
23403
2000
110
300
2001
430
000
2001
590
Episodes
AdaRL
Oracle
PNN
PSM
MTQ
Non-t
Figure A6: Learning curves for modified Cartpole experiments (POMDP version) with change factors.
The reported scores are averaged across 30 trials.
Fig. A7 and A6 provide the learning curves for modified Cartpole experiments (MDP and POMDP versions) with multiple change factors. In most cases, AdaRL can converge faster than other baselines.
25

---

Published as a conference paper at ICLR 2022
Gravity: 15
Gravity:
Mass: 1.0
3000
2040
2314
2000
-500
15000
.000
1000
600
Scores
370
Mass: 5.5
Gravity: Mass: 1.0
Gravity: 55; Maes: 5.5
2100
-3000
1000
1900
-000
000
1001
202
3.0
Episodes
AdaRL
Oracle
CAVIA
PEARL
Non-t
Figure A7: Learning curves for modified Cartpole experiments (MDP version) with change factors.
The reported scores are averaged across 30 trials.
Original
Orientation
Size
Color
Noise
Reward
Moving diractions
Agent
Hard-coded
Visual example of the original Pong game and the various change factors. The light blue arrows are added to show the direction in which the agent can move.
Figure A8:
A5.2 COMPLETE RESULTS OF THE MODIFIED PONG EXPERIMENT WITH CHANGING DYNAMICS AND OBSERVATIONS
Atari Pong is two-dimensions game that simulates table tennis. The agent controls paddle moving up and down vertically, aiming at hitting the ball. The goal for the agent is reach higher scores, which are earned when the other agent (hard-coded) fails to hit back the ball. We show the example of the original visual inputs and how appears after we have changed each the change factors in Fig, A8.
In source domains, the degrees are chosen from w = {0', 180%) ). and in target domains, they are chosen from w {90", 270"). For the image size, reduce the original image by factor of {2, 4, 6, 8) in source domains and by a factor of (3, 9} in target domains.
We summarize the detailed settings in both source and target domains in Table A6. In particular, in each experiment we use all source domains for each change factor and one the target domains at a time in either the interpolation and extrapolation set.
26

---

Published as a conference paper at ICLR 2022
target
Size
Noise
(a) Source target domains for Naarget
10000.
C
0.0
-05
10,000
-10
size
noise
(b) Different Naargtt in target domains.
Figure A9: Learned 0% for the two change factors, size and noise, modified Pong.
Size
Orientations
Noise
Background colors original, green, red yellow white
Source domains Interpolation set
{2, 4, 6,8
0", 180"
(0.25, 0. 75, 1.25, 1.75, 2.25
(3)
90"
(1.0)
Extrapolation set
(9)
270
(2.75)
Table A6: The settings of source and target domains for modified Pong experiments.
A5.2.1 LEARNED Og IN MODIFIED PONG EXPERIMENTS
Fig. A9a, Table A7, and Table A8 show the learned 0g in modified Pong experiments across different change factors.
Table A7: The learned 0% across different orientation angles with different Ntarget in modified Pong. The bold columns represent the target domains.
Ntarget
Orientations
0"
90º
180
270
g=l
-2.32
-178
1.69
0.44
10000
0
-2.94
1.86
1.47
1.59
ga1
-2.01
- 0.87
1.85
0.69
50
0-2
-2.59
- -184
1.42
1.07
-169
1.23
0.79
1.38
20
0
-198
- 0.56
0.82
1.20
We can find that each dimension of the learned 0% is a nonlinear monotonic function of the change factors. Table A7, Table A8 and Fig. A9b also give the learned &h with different sample sizes Ntarget target domains. Similarly, the learned curves with different sample sizes are homologous. Even with few samples, AdaRI. can still capture the model changes well.
27

---

Published as a conference paper at ICLR 2022
Table A8: The learned &g across different colors with different Marget in modified Pong. The bold columns represent the target domains.
Narget
Colors
Original
Red
Green
Yellow
White
0g
1.36
1.47
1.04
1.58
0.91
10000
0g:
0.72
1.15
1.17
0.96
1.33
0
0.93
- -1288
1.31
0.65
-1.09
0g
0.96
1.13
0.82
1.26
0.73
50
0.59
-0.46
0.75
1.32
0.59
0
0.61
-1002
0.91
-0.18
0.49
0g
1.09
1.38
0.65
1.30
0.46
0g
0.58
-0.72
0.39
1.60
0.27
00:
0.38
0..99
0.63
0.24
0.33
A5.2.2 AVERAGE FINAL SCORES FOR MULTIPLE Ntarget
Table A2 and A3 provides the complete results of the modified Pong experiments with Narggt = 20 and 10000, respectively. The details of both source and target domains are listed in Table A6. Similar to the results of Cartpole, AdaRL can perform the best among all baselines in Pong experiments. As shown in the results of the main paper, AdaRL consistently outperforms the other methods.
A..22.3 AVERAGE POLICY LEARNING CURVES IN TERMS OF STEPS
Fig. A12 gives the learning curves for modified Pong experiments with multiple change factors. From the results, we can find that AdaRL can converge faster than other baselines.
A5.3 COMPLETE RESULTS OF THE MODIFIED PONG EXPERIMENT WITH CHANGING REWARD FUNCTIONS
Table A9 summarizes the detailed change factors both linear and non-linear reward groups.
Linear reward (ks) (0.1, 0.1, 0.3, 0.4, 0.6,0.060777.33.
Non-linear reward (kg) (2.0, 3.0, 5.0, 6.0, 7.0, 8.0, 9.0)
Source domains
Interpolation set
[0.5)
[1.0]
Extrapolation set
(..9)
[1.0}
Table A9: The settings of source and target domains for modified Pong experiments.
We denote with L the half-length of the paddle and then formulate the two groups of reward functions kyd (1) Linear reward functions: re 2 where kn (0.1, 0.3, 0.4, 0.6, 0.7, 0.8) in source domains and k; {0.5,0.9 target domains; and (2) Non-linear reward functions: r 2432
k,
where b2 E (2.0,3. 3.0, 5.0, 6.0, 7.0, 8.0, 9.0} in source domains and A: E (100,4.0) in target domains.
A5.3.1 LEARNED 0, IN MODIFIED PONG EXPERIMENTS
Fig. A10 and All give the learned 0, with both linear and non-linear rewards. In both groups, the learned 0, is linearly or monotonically correlated with the change factor and there no significant gap between the learned 6, with different Ntarget-
A5.3.2 AVERAGE FINAL SCORES FOR MULTIPLE Ntarget
Table A12, A13 and A14 shows the average final scores for Niarget 50, 10000 and 50000 in modified Pong experiments with changing rewards. AdaRL consistently outperforms the other methods across different Narget-
28

---

Published as a conference paper at ICLR 2022
15
10
0.0
10
50,000
0.0
10,000
-1.9
k1
K1
(a) Learnt 0, in each domain.
(b) Different Nearget in target domains.
Figure A10: Learned 0g for the linear changing rewards modified Pong.
00
target
target
k2
k2
(a) Learnt 0, in each domain.
$0,000
50,000
100000
10,000
k2
k2
(b) Different Ntarget target domains.
Figure A11: Learned 0% for the non-linear changing rewards in modified Pong.
A5.3.3 AVERAGE POLICY LEARNING CURVES IN TERMS OF STEPS
Fig. A12 (last two rows) gives the learning curves for modified Pong experiments with changing rewards.
A5.4 RESULTS ON MUJoCO BENCHMARKS
also apply AdaRL on MuJoCo benchmarks (Todorov et al., 2012), including Cheetah and Ant with set of target velocities. We follow the setup in MAML (Finn et al., 2017) and CAVIA (Zintgraf et al., 2019). model estimation stage, we choose 20 tasks for each of the game. The goal velocity of each task is sampled between 0.0 and 2.0 for the cheetah and between 0.0 and 3.0 for ant. The reward the negative absolute value between agent's 's current and the goal velocity. In policy optimization stage, utilize Trust Region Policy Optimization (TRPO Schulman al. (2015)). The results
29

---

Published as a conference paper at ICLR 2022
Oracle Upper bound
Non-t
PNN (Rusa 2016)
PSM LAguraal 2021a)
MTQ
AdaRL- Ous
AdaRL
2020)
18.65
4.30
5,88 e
9.60 e (#33119
14.30
15.05 e
16.79
(#2.43)
(#2.95)
(#5.78)
(#2.002
(=3.19)
(#1.84)
19.88
O_out
(#11.09)
(#2.41)
(#:.26)
(#66228)
(#2.46)
(=3.09)
(#b4.88)
C_in
(+0.440)
19.35
(#22.68)
7.72
(#:4.16)
13.75
(+00115)
10.87
(+a.007)
14.80
(=2.86)
16.07
(#13330)
16.29
12.59
(40.25)
(#3.21)
(#:442)
#3.80)
(#3.22)
(=3.10)
(#2.00)
S_i
(#1.18)
18.32
(#3.42)
6.25
(#2.72)
12.93
(+1.85)
10.67
(#3.46)
12.78
(=2.96)
13.86
(+44.48)
14.02
S_out
(41.04)
19.01
(#2.76)
(#:.27)
(43.15)
13.50
(#2.411
(=2.13)
(#3.72)
15.88
N_in
(#1.25)
18.48
(#2.22)
4.29
13.85
(2.83)
(#2.221)
13.69
(#3.27)
10:96
(=3.07)
13.51
(#2.95)
15.57
14.07
N_out
(4:1.11)
(4:2.47)
(1:3.82)
(1:2.56)
(4:3.18)
(=3.10)
(d:2.72)
Table A10: Average final scores modified Pong (POMDP) with Ntrrgets 20. The best non- oracle results are marked red, while bold indicates a statistically significant result w.r.t. all the baselines. O, C, S, and N denote the orientation, color, size, and noise factors, respectively.
Upper bound
lower bound
(Rusu al., 2016)
(Agarwal et 202
(Fakoor al., 2020)
Oum wio masks
Ours
Q
(4:2.43)
18.65
(4:1.78)
8.04
(43.07)
12.19
(4:2.92)
12.37
(4:3.01)
14.64 e
(=2.20)
17.42
(411663)
18.85
O.
(#:1.09)
19.
(#1.888
(1:3100
(#:2:20)
(#44.93)
(=1.86)
(#2.41)
Cjn
(#:0.45)
19.35
(13.111)
8.00
(1:3-19)
15.89
(1:2.38)
16.70
(:2.16)
17.85
(=2.01)
17.73
18.93
19.78
7.48
16.85
16.29
17.93
16.49
19.
C_oun
(#:0225)
(#22009)
(1:.117)
(1:2.64)
(22.35)
(=2.04)
(=1136)
18.32
7.45
12.89
13.84
15.33
17.49
(d:1.18)
(1:3.15)
(4:2.04)
(1:3.27)
(4:2.03)
(=2.62)
(1:2.18)
S_out
(#11.04)
19.01
(#/2.36)
7.04
(#2.003
14.69 e
(+:2300)
17.25
(+1.36)
18.48
(=1.98)
17.82
(#:0663
19.21
16.80
15.95
(#11255)
(#:2.09)
(122.82
(d:1.73)
(#2.19)
(=3.68)
(#1.81)
N_out
(d:1.11)
18.
(1:2.46)
7.82
(1:2.98)
14.89
(1.3.94)
16. 16.85
(42.36)
17.03
(=3.25)
18.49
(1:2.16)
17.85
Table All: Average final scores on modified Pong (POMDP) with Ntargeta = 10000. The best non-oracle results are marked in red, while bold indicates a statistically significant result w.r.t all the baselines. O, C, S, and N denote the orientation, color, size, and noise factors, respectively.
Oracle Upper bound
Non lower bound
PNN (Rusu al., 2016)
PSM (Agirwal 2021a)
MTQ (Fikoor 2020)
AdaRL* wlo masks
AdaR
Ours
RLin
(#:3.8) 81)
7.98
(#:2.27)
4.95 (+-1,08)
(#:2.11)
(#22110
4.78
(+1.97)
3.49
(#2.06)
5.81
RI
(#:4.
9.61
(#:2.80)
5.19
(#11933
5.89
(#:2.71)
6.03
(#3.
6.21
(#2.59)
5.64
(#3.45)
6.12
Rilin
(22.16)
(#1171)
(::2,78)
5.31
(1:3.88)
5.06
(1:3.40
5.52
(2:3003
5.79
(#3.17)
5.84
41.36
21.73
27.19
23.27
25.49
26.33
29.92
(45.770)
(#8.544)
(.28)2)
(#88.01)
(#6.18)
(#7.944
(#6.39)
Table A12: Results on modified Pong game with Ntorgets = 50. The best non-oracle results are marked in red, while bold indicates statistically significant result w.r.t. all the baselines. Rl and Ro denote the linear reward and nonlinear reward- soward-changing cases, respectively.
30

---

Published as a conference paper at ICLR 2022
Orientation: 90"
Orientation: 270"
Color: yellow
-20
307
1000
30:0
2300
1330
1330
3000
Color: white
Size:
Size:
10
Scores
-30
200
1000
1500
3000
3000
500
1000
1330
3000
100
15.0
3000
Noise: 1.0
Nnise: 2.75
Kj =.55
50.0
K1-0.9
Kg 110
Ky K,0
-33
Episodes
AdaRL
Oracle
PNN
PSM
MTQ
-No-t
Figure A12: Learning curves for modified Pong experiments with change factors. The reported scores are averaged across 30 trials.
Oraele Upper hound
Non-1 lower bound
PNN (Rasu et 2016)
PSM (Agarwal ol., 20216)
MTQ (Fäkknor 20200)
AdeR1, Ours wko mrakka
AdoRI.
Churs
5.17 (1:1.98)
(1:3. 81)
(21.70)
(:: (::1.82)
(::2.45) 45)
(23.19)
(22.00)
Rl_uul
(+-4.75)
9.61
(#22.01)
5.82
(4:2.79)
6.10
(#+1.98)
7.30
(422.14)
8.42
(+2.622
7.04
(#2.36)
8.41
RiLin
(#2.11)
7.62
(#2.47)
3.13
(#11.42)
(#:3.31)
6.42
(#3.119)
6.30
(21.00)
5.52
(#1.24)
6.57
41.36
27.70
31,28
33.60
29.77
36.52
Ru
(25..70)
(::4.09)
(::3.552
(1:385)
(15.02)
(1:.118)
Table A13: Average final scores on modified Pong (POMDP) with Ntarget 10000. The best non-oracle results are marked in red, while bold indicates statistically significant result all the baselines. RI and Ro denote the linear and nonlinear reward changes, respectively.
the target domains are specified in Table A15, suggesting that AdaRL can achieve better performance than the meta-learning approaches (i.e., MAML and CAVIA).
31

---

Published as a conference paper at ICLR 2022
Ocacle Upper bound
Nozi lower bound
PNN (Rusu 2016)
PSM (Apurwal 202
MTQ (Takooer 2020)
AdaRL"
AdaRL
wlo masks
Ours
RLin
(#:3.8) 81)
7.98
(#22.03)
(4t4.87)
(#b..47)
6.90
(#3.18)
7.34
(#3.12)
(#2.09)
7.93
Rl_uul
(2:4.78)
9.61
(#2.16)
3.89
(.:2)8
7.85
(#:3.75)
7.37
(#22.613
8.77
(#3.10)
7.78
(#2.02)
8.94
Ra
(4:2.
7.62
(#:1.09)
(4:2.85)
6.81
(::2.46)
7.01
(4:3.44
6.28
(4:2,63)
6.30
(#1.94)
7.57
Rn
(43.70)
41.36
(#3.002)
29.98
(=10.33)
36,08
(#:11.20)
37.26
(4-12.59)
38.48
(4:9.36)
34.19
(+6.92)
41.25
Table 14: Results on modified Pong game with Ntargets = 50000. The best non-oracle results are marked in red, while bold indicates statistically significant result w.r.t all the baselines. RI and Ro denote the linear reward and nonlinear reward-changee cases, respectively.
MAML at. 2017)
CAVIA (Zintrrai et 2019)
AdaRL
Cheetáh (vel)
(1:4.)
-89.
-86.5 (4:2.0)
(#3.2)
-81.7
100.4
95.7
106.8
Aat (vel)
(#10.9)
(#66. 92)
(#8.4)
Table A15: Results on MuloCo benchmarks (Cheetah and Ant experiments with different target velocities, with trials each) with Ntargets = 50, 000. The best results are marked red.
A5.5 EFFECT OF THE POLICY USED FOR DATA COLLECTION DURING MODEL ESTIMATION
In our framework, use the random policy generate trajectories for each domain. The generated trajectories are further used to conduct the model estimation. To study whether the random policy will affect the effectiveness of model estimation, we compare the learnt parameters of latent space in model estimation with the trajectories generated via (1) the random policy in our framework, and (2) the optimal policies learnt on the source domains. Here we show the case with changing gravity in the modified Cartpole game. In Fig. A13, we give the learnt parameters (μ and log or) of the first and second components in the Gaussian mixtures the 10-th latent state 20-th epoch. The results demonstrate that the difference between the two sets of learnt parameters is limited.
A5.6 MORE STATISTICAL EVALUATION PROTOCOLS ON THE PERFORMANCE
In this section, we provide more detailed and comprehensive comparison on the performances of AdaRL and other baseline methods. Following the recent published work on reliable evaluation for RL (Agarwal et al., 2021b), we utilize 4 evaluation metrics: median, interquartile mean (IQM), mean, and optimality gap. Median and mean are the sample median and mean, respectively. IQM discards the top and bottom 25% samples and then computes the mean value of the remaining ones. Thus, this factor is insensitive to outliers. Optimality gap is quantified via the gap between the performance of each method and the mean score obtained by the oracle agent. Therefore. higher mean, median and IQM and lower optimality are the indications of better methods. All metrics with 95% bootstrap confidence intervals (CIs) (Efron, 1992).
Fig. A14, A15, A16. and A17 give the comparison on AdaRL and baselines using the set of evaluation protocols in different games and settings. The results suggest that in most cases, AdaRL performs consistently better than the baselines across all evaluation metrics.
32

---

Published as a conference paper at ICLR 2022
Learnt mean (u)
/1
M22
Learnt std (log d)
log Gj-
log dzi
Random policy
Optimal policy
Figure A13: Learnt a and log model estimation with both random policy (left), and optimal
policy (right).
33

---

Published as a conference paper at ICLR 2022
G in
Median
QM
Mcon
Optimality Cap
CAVA
MORL
G oot
Median
IQM
Mcan
Optimality Gap
PEARL
CAV
AdoRL
400
450
500
400
450
5:00
4000
450
500
200
250
300
M__nn
Medion
IOM
Mean
Optimalty Gap
CAN
AdWRI
Mouut
Optimality
PEARL
AIRRL"
AdaRL
750
900
1055
750
900
1050
750
900
1050
450
000
750
G In M_in
Median
IQM
Mean
Opt maily Gap
CAV
AGDRL-
1200 1500
7200
1500
500
1000 1250
G out
M
Median
IOM
Mean
Ostimalty Gap
CAN
AdaRL
600
700
800
600
700
800
600
700
800
160
240
320
400
Figure A14: Evaluation on the results of modified CartPole game under MDP settings for N Ntorget 50.
34

---

Published as a conference paper at ICLR 2022
G_in
Modian
Optimality
PSM
PNN
MTQ
AdaRL*
1200
1600
2000
1200
1600
2000
1200
1600
2000
400
600
800
1600
o_u
Median
IQM
Mean
Optimality
PSM
PNN
MTQ
AdaRL*
AdaRL
360
160
M in
Median
IQM
Mean
Optimality Gap
AdaRL
1800
1600 2000
2000
150
600
M out
Median
IQM
Mean
Optimality Gap
900
1200
1500
1800 800
1200 1600
900
1200
1500
1800
200
400
600
in
Median
IQM
Mean
Optimality Gap
PSM
PNN
MTQ
AdaRL*
AdaRL
4000
6000
6000
4000
6000
6000
4000
6000
8000
1500
3000
4500
N out
Optimality
PSM
PNN
MTQ
AdaRL*
AdaRL
2400
2000 3000
1600 2400
2400
Figure A15: Evaluation on the results modified CartPole game under POMDP settings for 50.
Nearget
35

---

Published as a conference paper at ICLR 2022
O in
Nedan
IQM
Mean
Optimality Gup
PSM
PMN
AaRL.
AcaRL.
18
20
12
15
18
O_out
IQM
Optimaityy
MIQ
AdaRL
C in
Median
IQM
Mcan
Optimality Gop
PMN
NTQ
ACaRL-
15.0 165 18.0
19.5
14
16
15.0
16.5
18.0
19.5
1.5 3.0
C out
Ned
IQM
Mean
Optimality Gap
PSM
PMN
AdaRL"
AcaRL-
15
S in
Nedian
I0M
Optimalny
PSM
AdaftL
AcaRL
16
S_out
Median
I0M
Optimalty
PSM
MTQ-
ASRRL".
N in
Med
I0M
Mean
Optimelity Gap
PSM
PMN
AdaRL
AcaRt
15.0
12.5 15,0
17.5
10.0
12. 15.0
17,510.0
12.5
35.6
37.5
out
Nedian
10M
Mean
Optimally
PSM
PHI
MSRLY.
ACaRL-
125
150
17.5
125
15.0
17.5
12.5
15.0
175
36
Figure A16: Evaluation on the results of modified Atari Pong game under POMDP settings with changing orientation, color, size, and noise levels for Ntarget 50.

---

Published as a conference paper at ICLR 2022
RI in
Median
IQM
Mean
Optimality
PNN
MTQ
AdaRL
4.5
6.0
7.5
4.5
6.0
7.5
4.5
6.0
7.5
RI out
Optimality
PSM
AdaRL*
AdaRL:
Rn in
Median
IQM
Mean
Optimality Gap
PNN
MTQ
AdeRL-
1.6
2.A
3.2
Rn_out
Median
Oatimality
MTQ
AdaRL
30.0 32.5 35.0
37.5
30
33
36
30.0
32.5
35.0
37.5
5.0
7.5
10.0
12.5
Figure A17: Evaluation the results of modified Atari Pong game under POMDP settings with changing reward functions for Ntorget = 10000.
37

---

Published as a conference paper at ICLR 2022
A5.7 PSEUDO CODE OF MIss- VAE
Algorithm A1 Pseudo code of Miss- VAE,
s: latent statez theta_e, thete_s, theta_r: changing factore;
CLap, CLSE, CLaz, CLas, CLas, C_sheta_oo, C_taeta_ Cutheta_sss structure masks)
t o: observations a: action; E: zeward;
nun_mix: the number of componenta mixture Gauasian model
5
6
7 #87459EE084444B44EEE
loader: lead batch of Enooder
domains
8
conv (o)
9
conv (a)
10
conv
11
(theta__o, theta_s, theta_r)
12
consat [ho, he, hd]
13
LSTM(A) output lognix, mean, logstd state output, (ample( sample
14
15
16
Decoder (
17
4 Reconstruct o_[t)
multiplyls, 19 thate_o_e multipyyttheta_a (_theta_oo)
18
20
h_a_theta concst (tota_o), to(chtta_o_ol)
21
(a observation of current step
22
23
i Reconstruat [_(++])
24
multiply (_eE)
25
maltiply CLas)
26 theta_r_r (t c_theta_r:)
27 hass [elconeat a_x])
28
l_theta_r - fcitheta___e)
29
r_t+1 to(concat (h_as, h_theta_r]) $ reward of current step
30
31
i Predics 0_1t111
32
a_t+i corv_sranaposess, theta_e, thete_s) & reward of next stop
33
34
i Predics e_(=+2]
35
=_t+2 feconncat (fe(s), fe(a), fclthers_o), (ttetta_s)) + reward in the next step
36
Markovian transition
38
s.s multiply c_as]
39
thsta_s_s (t::iplyytheta_ss, C_theta_ss)
40
ass multiply CLas)
41
h_dyn fe [e [s_8, theta_s_a, s_8]
42
s_output 1inear (h_dyn)
s_ Loomixx s_mean, log_lo a_output.aplit & state in next step
****
Loss
# Reconscruction loss
Rec_ loss mean (M o_tl MSR Y_t+1)]
* Predicsion loss Prad_lass mean (o] :] a_t+l) + (st( r_t+2))
# EL less
53
for idx in range wwwm_nnummnneeee inmmmiixuure::
54
g_logmix, q_mean, g_logatd idkk), a_mean[:,,dx), loggiogst
Loas 1- lolg_logmix, g_mean, g_logstd]
56
loss mean (log(1 (Rt_toss 1e-10) + 1e-10))
57
Sparsity constraiats
59
Reg_lose mean (C_5o) maan (C_ar) mosn (C_ar) mean (C_ss) mean (C_theta_odd (an[[thetta_r) theta_r)
mean (theta_o) + mear. (theta_s) + meant
60
61
4 optimization atep
loss Rec_loas Pred_losa + 1oss Reg_losa 63
62
lcss ())
64
38

---

Published as a conference paper at ICLR 2022
A6
EXPERIMENTAL DETAILS
A6.1 HYPERPARAMETERS SELECTIONS
Model estimation We use a random policy collect sequence data from source domains. For both modified Cartpole and Pong experiments, the sequence length is 40 and the number of sequence is 10000 for each domain. The sampling resolution set to be 0.02. Other details are summarized in Table A16. Parts of the implementation of model estimation modules follows the open-source framework the world model (Ha and Schmidhuber, 2018).
Settings
Caripole
Pong
Dimenssoos latent space
20
25
# Dimensions of Epochs
Size & noise: I, orientation: 2, color: 3, Reward: (lincar), 2 (non-linear)
roward-varying: others:
Batch size
20
80
Initial leamiing rate
0.01
0.01
Learnnngg rate docay rate
0.999
0,999
Dropout
0.90
0.90
KL-tolerance
0.50
0.50
Table A16: Experimental details on the model estimation part.
Policy learning We adopt Double DQN (Van Hasselt al., 2016) during policy learning stage.
The detailed hyper-parameters are summarized in Table A17. For fair comparison, we use the same set of hyperparameters for training other baseline methods.
Settings Discount factor Exploration rate
Cartpole
Pong
0.99
0.99
1.0
1.0
Initial learning rate
0.01
0.01
Learning rate decay rate
0.999
0.999
Dropout
0.10
0.10
Table A17: Experimental details on the policy learning part.
A6.2
EXPERIMENTAL PLATFORMS
For the model estimation, Cartpole and Pong experiments are implemented on 1 NVIDIA P100 GPUs and NVidia V100 GPUs, respectively. The policy learning stages in both experiments implemented on 8 Nvidia RTX 1080Ti GPUs.
A6.3 LICENSES
In our code, we have used the following libraries which are covered by the corresponding licenses:
Tensorflow (Apache License 2.0),
Pytorch (BSD 3-Clause "New" or "Revised" License),
OpenAl Gym (MIT License),
OpenC' (Apache 2 License),
Numpy (BSD 3-Clause "New" or "Revised" License)
Keras (Apache License).
We released our code under the MIT License.
39

---

Published as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization reinforcement learning. In International Conference Learning Representations (ICLR), 2021a. URL https: opwpreveeeeeeeeee//e/ee net/ forum?id=qda? sVg84.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In International Conference on Machine Learning (ICML), pages 205-214, 2018.
C. M. Bishop. Mixture density networks. In Technical Report NCRG/4288, Aston University, Birmingham, UK, 1994.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint (rXiv:1606.01540, 2016.
Bradley Efron. Bootstrap methods: another look the jackknife. In Breakthroughs in statistics, pages 569-593. Springer, 1992.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning. In Inter- national Conference on Learning Representations (ICLR), 2020. URL Utt.htttsss//ppeereee
net/ / Eorum Saw.uumndddSDeeeEeeeee
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), pages 1126-1135. PMLR, 2017.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 13777777 1997.
B. Huang, K. Zhang, J. Zhang. J. Ramsey, R. Manchez-Romero, C. Glymour, and B. Schölkopf. Causal discovery from heterogencous//oonsattonnny data. Journal of Machine Learning Research, 21(89):612-634 2020.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual Conference Computational Learning Theory (COLT), pages 164-170, 1999.
Kevin Murphy. Dynamic bayesian networks: Representation, inference and learning. UC Berkeley, Computer Science Division, 2002.
Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning. In Interna- tional Conference on Machine Learning (ICML), pages 991-999, 2014.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning (ICML), pages 5331-5340. PMLR, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.0666..7, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pages 1889-1897. PMLR, 2015.
40

---

Published as a conference paper at ICLR 2022
Shai Chalev-Shwarde and Shai Ben-Davice Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. Spring-Verlag Lectures in Statistics, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026 5026-5033. IEEE, 2012.
Hado Van Hasselt Arthur Guez, and David Silver. Deep reinforeement lcarning with double q- learning. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence (AAAI), volume 30, 2016.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-lcarning without memorization. arXiv preprint arXiv:1912.03820. 2019.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Seventh Conference on Uncertainty Artificial Intelligence, UAI'11, pages 804-813, Arlington, Virginia, USA, 2011. AUAI Press. ISBN 9780974903972.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning (ICML), pages 7693-7702. PMLR, 2019.
41


Reviews:
[{"review_id": "swv4nKBlwh", "reviewer": "Reviewer_awgy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- I think the explicit factorization is great and that the results are impressive.\n- The ablation on the masks is good and helps demonstrate that all the components of the method are needed.\n\nImprovements and questions:\n- The particular architecture i.e. the masks and factorization, seem like they might be broadly \n- What happens if you get the number of parameters that vary between domains wrong i.e. you say there are two but there are actually three or vice versa?\n- How does this scale with the number of source domains?\n\nClarity: \n- In the evaluation section, you have $N_\\text{target}=20, 50,10,000$ and I am pretty sure you mean $20$ and $50$ and $10000$ but it looks a little confusing. You might want to remove the comma from the 10000.\n- In evaluation case, you state the you are evaluating in the POMDP case since the inputs are high dimensional. Are you stacking frames? If so, this isn’t necessarily a POMDP. (TODO check appendix)\n- It seems odd to call this method interpretable. It seems to be interpretable because you know what the factors of variation are a-priori; would it still be interpretable if you didn’t? As in, does your method allow you to post-de-facto construct the factors of variation by examining the masks or something fo the sort?\n- On page 2 you state that the noise factor does not affect the optimal policy for rotated + white noise pong? Is this obvious? It seems possible to construct scenarios where the noise factor very much affects the optimal policy. For example, one could imagine a game of adversarial pong where collision with the ball receives a huge negative reward but the opposing agent in pong stays the same. In such a case, it might be optimal to hide in one of the corners when the noise is high and you frequently can’t see your paddle; in the case where the noise is low you might want to actively dodge the ball. Perhaps I’m missing something and it’s obvious that noise doesn’t change the optimal policy. \n- On page 2 in the final paragraph you use the notation s_{2,t} but this $2$ index is not defined at this point in time.\n- Am I correct in observing the Algorithm 1 does not contain the actual inference procedure of the target value of $\\theta$? Where in the main text is the inference procedure discussed? \n", "summary_of_the_paper": "Instead of implicitly updating the policy using data from the source domain, learn a particularly structured latent model and the elements of variation and learn a policy that performs pretty well on some set of the elements of variation. At test time, estimate the elements of variation and provide them as input to the policy. I think this is a good paper but could benefit from some clearer writing as discussed in the clarity section below.", "summary_of_the_review": "Small issues with the writing but the ideas seem new and to work quite well!", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dEqD3PxhIkr", "reviewer": "Reviewer_6qyY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I quite liked this paper as it introduces a well-motivated idea that can be quite useful to the research community. However, the proposed algorithm is rather involved and the empirical gains seem to be quite marginal relative to other methods in the literature (which are arguably easier to explain/implement).\n\nMy two main concerns are clarity and the quality of the empirical evaluations. More details below.\n\n**Clarity**: The idea is somewhat involved and there are _lots_ of parameters and moving parts. While the authors have made a good effort to make it clear, there are still some parts that are hard to follow. Some more details below.\n\n1.  Although Figure 1 helps somewhat, it is still rather difficult to follow. I would suggest moving Figure A3 from the appendix to the main paper.\n1.  In the definitions of $\\mathcal{L}^{rec}$ and $\\mathcal{L}^{pred}$ the $\\beta$ terms need to be explained more. Both $\\beta_1$ and $\\beta_2$ produce $o$s and $r$s, yes? Why are they conditioned on different parameters? I _think_ one is the Dynamics and the other is the Decoder model (as illustrated in Figure A3), but it's not clear. Indeed, the sentence immediately after says that \"$p_{\\beta_1}$ and $p_{\\beta_2}$ denote  the generative model with parameters $\\beta_1$ and $\\beta_2$\". The use of the singular \"the\" suggests that it's the same model with different parameters, which I don't think is the case.\n1. In Algorithm 1, are $Q$ and $Q'$ initialized independently or in the same way? Does it matter?\n1. The rightmost panel in Figure 2 is difficult to understand. Maybe show more context of the Pong game so it's clear that this is a paddle and a ball.\n\n**Empirical evaluations:**\n1.  Although I can appreciate the environment modifications the authors used to showcase their algorithm, there are already existing benchmarks (such as [ProcGen](https://openai.com/blog/procgen-benchmark/), [Jumping Task](https://github.com/Maluuba/jumping-task), and [Atari modes](https://arxiv.org/abs/1810.00123)). Why were none of these used to evaluate the proposed method?\n1.  The bold red fonts in the tables at the end are misleading, as there is _major_ overlap between the AdaRL confidence intervals with many of the other methods, which means that the gains are often not statistically significant. This is also evident in the appendix figures (e.g. Figures A6 and A7). There seem to be no statistically significant gains to be had from AdaRL.\n\n---\n\nSome questions for the authors:\n1. In the definition of $\\mathcal{L}^{rec}$, what is the range of $q_{\\phi}(\\cdot |\\theta_k)$? $S$? Writing the expectation subscript as $s_{t,k}\\sim q_{\\phi}(\\cdot | \\theta_k)$ would help clarify things.\n1.  In the last sentence of the **Modified Cartpole setting**, does $\\theta_k=\\{\\theta_k^o\\}$ mean the only variations are in observations?\n1. In page 8, right above the **Baselines** section, it says $r_t=\\frac{kL}{d+3L}$. Is $k$ the actual index used in the computation of $r_t$? If not, use a different letter.\n\n---\n\nSome other minor points:\n\n1. The authors use the word \"parsimonious\" a lot, which seems strange. The Oxford Dictionary says its definition is _\"extremely unwilling to spend money\"_, which seems like a strange adjective to apply to an RL algorithm.\n1.  In the Introduction the authors should also reference [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://arxiv.org/abs/2101.05265). I know it is referenced later in the paper, but it should also be discussed in the introduction when other representation learning methods (such as (Zhang et al. 2021)) are being discussed.\n1.  PNN in the introduction is not defined.\n1. At the bottom of page two it says \"We learn an optimal policy\", but to be precise it should be \"a near-optimal policy\".\n1.  In Equation 1 the index $k$ is used. Later in the paper it becomes clear that it indexes domain, but this should be clarified before equation (1).\n1.  At the bottom of page 4 the term ${\\bf y}_{1:t,k}$ is used, but this variable has not been introduced yet (it is only introduced at the top of page 5).\n1. Right above section 3.2 it says \"In this way, except $\\theta_k$\", it's missing a \"for\" before the $\\theta$.\n1. In Algorithm 1 (e.g. lines 7 and 13) use `\\cdot` in the posterior (e.g. $q(\\cdot | o_{\\leq t+1, ...})$), as the posterior at this point is a distribution, not a materialized probability.\n1. In page 2 of the appendix there's a missing \"}\" in the first line of the last paragraph.\n1. In the first bullet of page 2 of the appendix it should say \"then it ***means*** there is no path\".", "summary_of_the_paper": "This paper proposes a method that learns structural relationships between variables of the RL system so as to be able to adapt to changes across domains, and learn a new policy in a new domain with few samples. The authors provide some theoretical results on their method, and evaluate their proposal on some standard environments that have been modified to evaluate this type of idea.", "summary_of_the_review": "I think this is an interesting paper, but it falls short in the empirical evaluations. I think the idea is interesting and the theoretical results may be of interest to others working in this space.\nHowever, I don't feel the empirical performance justifies the complexity of the algorithm, which is why I'm somewhat borderline, but leaning towards an accept.\nI look forward to reading what the other reviewers think, as well as the authors' response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SRduG9jWKR", "reviewer": "Reviewer_DmF1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is proposing an interesting method for an important problem. Transfer with minial number of steps (and no training) is a very valuable aspect for reinforcement learning. The authors do a great job at presenting the problem and the current approaches to the problem, giving enough context to the reader.\n\nThe proposed method is novel, it has significant differences compared to previous methods. The authors explain the rationale and the theory behind it in details.  For evaluation, the authors give a pretty detailed comparison with different types of target domains and through comparison with other methods. On the other hand, the selection of domains could be more complex such as locomotion environments where contact dynamics is a primary concern and transfer learning is a natural problem to tackle. Or the authors could also test changing dynamics of the pong game in addition to the representation. The results show that the AdaRL outperform other methods, but the error bounds are very large (both for AdaRL and other methods) with respect to the difference between the methods. \n\nThe writing and the presentation of the paper is very well. The paper is easy to follow and understand.\nFew small details: When the authors write 10000 as 10,000 it causes weird side effects such as Ntarget = {20,50,10,000}. This should be corrected.I believe that in page 7,  0^k_o should be 0^o_k.\n", "summary_of_the_paper": "The paper proposes the algorithm \"AdaRL\" a transfer method for reinforcement learning for different domains. The method is based on learning a latent representation with \"domain shared\" and \"domain specific\" components. A policy that is parameterized by \"domain specific\" components is learned. The transfer is done by collecting some data in target domain and estimating \"domain specific\" variable of the target domain. The authors show the algorithms mechanism for POMDP and MDP settings. The method is evaluated on modified versions of Cart-Pole and Pong domains. The authors test different settings where parameters of the environment as well as rewards functions are changed within source domains and the target domain. The authors also evaluate interpolation vs extrapolation cases. The method is compared with latest transfer learning methods and the results show that AdaRL performs better compared to other methods ", "summary_of_the_review": "The paper proposes a complex and interesting approach to transfer for RL. The authors do a throughout comparison and analysis of the method. The evaluated domains could be more complex, otherwise the results support the claims of the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Eolei_L4RsD", "reviewer": "Reviewer_i8j9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper investigates the question of transfer via identifying a small number of changing variables to maximise data efficiency. It is overall clearly written and easy to follow for most parts. \n\nThe three step training framework with a split between model estimation from random data and policy learning can lead to a set of challenges which should be mentioned and discussed. First, random data might not cover enough of an environment's state space such that the estimated model is not accurate for later stages of policy training when a different part of the state space is visited. Second, splitting model estimation (focused on reconstruction and prediction) and policy training can lead to model features which are suboptimal for the policy. Third, the policy will be trained with the feature distributions of a set of source domains. Any function approximator used for the policy will need to handle out of distribution data on target domains which can differ from these features. In particular the second point could be investigated by iterating between model estimation and policy training.\n\nThe evaluation focuses on new variations of known domains opening the question why no existing benchmarks e.g. from the metalearning literature are used (e.g. benchmarks from the set of baseline papers such as the cheetah or ant domains from MAML). Further, it is unclear why e.g. MAML, Progressive Networks, etc (which are used in the MDP setting) are not applied to the POMDP setting. Vice versa for many POMDP methods which are not constrained to the use of POMDPs.\n\nThe use of image based domains does not need to lead to a POMDP. In both cartpole and Pong, occlusions do not occur. If the agent stacks multiple frames, velocity as part of the state is included. If not, it will be helpful to emphasise this aspect to describe which aspects are unobserved.\n\nWhile overall well written, parts about equations and the general model are hard to trace and a visual representation of various parts would be helpful. See minor points for details.\n\nMinor:\n- description of meaning for masking parameters (c’s) should be move closer to Eq. 1\n- a graphical representation for Eq. 1 (definition 1 with respect to compactness would benefit from an example graph either in the appendix or, if space can be found, in the main paper)\n- Section 3.1. describes the training of the VAE via SGD. Including ‘in one step’ in the title seems inaccurate here. I could not find any justification for this term in the section.\n- Explicit pseudocode for algorithm 1 is very helpful!\n- The related work section provides a broad background but work on learning an embedding space for system dynamics to enable quick adaptation for sim2real transfer should be added [1,2].\n\n\n[1] Yu, Wenhao, et al. \"Preparing for the unknown: Learning a universal policy with online system identification.\" arXiv preprint arXiv:1702.02453 (2017).\n[2] Peng, Xue Bin, et al. \"Learning agile robotic locomotion skills by imitating animals.\" arXiv preprint arXiv:2004.00784 (2020).\n", "summary_of_the_paper": "The submission proposes a method for transfer in reinforcement learning building on estimating a small set of factors describing a system and modelling the agent as a dynamic bayesian network. In detail, the method splits into domain shared and domain dependent factors which are modelled via a new form of structured sequential VAE. The VAE is trained to enable dynamics, observation and reward prediction (including reconstruction, prediction, KL and sparsity regularisation losses).\n\nData for VAE training is collected from random policies and model fitting is followed by policy training on source domains to be transferred to target domains by identifying domain specific features for the target domains.\n\nThe method is evaluated on variations of a cartpole and a pong domain (both from images) and evaluated against a set of recent, competitive baselines.\n", "summary_of_the_review": "A strong submission on improving transfer learning which leaves a couple of open questions regarding its evaluation. However, these questions should be easy to address in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method called "AdaRL" that learns a structured latent representation that characterizes relationships between different variables in an RL system. The method is evaluated on modified Pong and Cart-Pole domains and it is shown to outperform other transfer learning baselines. The reviewers agree that the method makes sense and addresses an important problem of transfer in RL. The authors did a good job in the rebuttal to empirically validate their claims and provided extra experiments. The reviewers also point out that the evaluated domains are rather simple and the paper would benefit from evaluations in a more complex environment as well as better writing. Please focus on improving these aspects in the final version of the paper.
Decision: Accept (Spotlight)