Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the algorithm "AdaRL" a transfer method for reinforcement learning for different domains. The method is based on learning a latent representation with "domain shared" and "domain specific" components. A policy that is parameterized by "domain specific" components is learned. The transfer is done by collecting some data in target domain and estimating "domain specific" variable of the target domain. The authors show the algorithms mechanism for POMDP and MDP settings. The method is evaluated on modified versions of Cart-Pole and Pong domains. The authors test different settings where parameters of the environment as well as rewards functions are changed within source domains and the target domain. The authors also evaluate interpolation vs extrapolation cases. The method is compared with latest transfer learning methods and the results show that AdaRL performs better compared to other methods 
main_review: The paper is proposing an interesting method for an important problem. Transfer with minial number of steps (and no training) is a very valuable aspect for reinforcement learning. The authors do a great job at presenting the problem and the current approaches to the problem, giving enough context to the reader.

The proposed method is novel, it has significant differences compared to previous methods. The authors explain the rationale and the theory behind it in details.  For evaluation, the authors give a pretty detailed comparison with different types of target domains and through comparison with other methods. On the other hand, the selection of domains could be more complex such as locomotion environments where contact dynamics is a primary concern and transfer learning is a natural problem to tackle. Or the authors could also test changing dynamics of the pong game in addition to the representation. The results show that the AdaRL outperform other methods, but the error bounds are very large (both for AdaRL and other methods) with respect to the difference between the methods. 

The writing and the presentation of the paper is very well. The paper is easy to follow and understand.
Few small details: When the authors write 10000 as 10,000 it causes weird side effects such as Ntarget = {20,50,10,000}. This should be corrected.I believe that in page 7,  0^k_o should be 0^o_k.

summary_of_the_review: The paper proposes a complex and interesting approach to transfer for RL. The authors do a throughout comparison and analysis of the method. The evaluated domains could be more complex, otherwise the results support the claims of the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Instead of implicitly updating the policy using data from the source domain, learn a particularly structured latent model and the elements of variation and learn a policy that performs pretty well on some set of the elements of variation. At test time, estimate the elements of variation and provide them as input to the policy. I think this is a good paper but could benefit from some clearer writing as discussed in the clarity section below.
main_review: Pros:
- I think the explicit factorization is great and that the results are impressive.
- The ablation on the masks is good and helps demonstrate that all the components of the method are needed.

Improvements and questions:
- The particular architecture i.e. the masks and factorization, seem like they might be broadly 
- What happens if you get the number of parameters that vary between domains wrong i.e. you say there are two but there are actually three or vice versa?
- How does this scale with the number of source domains?

Clarity: 
- In the evaluation section, you have $N_\text{target}=20, 50,10,000$ and I am pretty sure you mean $20$ and $50$ and $10000$ but it looks a little confusing. You might want to remove the comma from the 10000.
- In evaluation case, you state the you are evaluating in the POMDP case since the inputs are high dimensional. Are you stacking frames? If so, this isn’t necessarily a POMDP. (TODO check appendix)
- It seems odd to call this method interpretable. It seems to be interpretable because you know what the factors of variation are a-priori; would it still be interpretable if you didn’t? As in, does your method allow you to post-de-facto construct the factors of variation by examining the masks or something fo the sort?
- On page 2 you state that the noise factor does not affect the optimal policy for rotated + white noise pong? Is this obvious? It seems possible to construct scenarios where the noise factor very much affects the optimal policy. For example, one could imagine a game of adversarial pong where collision with the ball receives a huge negative reward but the opposing agent in pong stays the same. In such a case, it might be optimal to hide in one of the corners when the noise is high and you frequently can’t see your paddle; in the case where the noise is low you might want to actively dodge the ball. Perhaps I’m missing something and it’s obvious that noise doesn’t change the optimal policy. 
- On page 2 in the final paragraph you use the notation s_{2,t} but this $2$ index is not defined at this point in time.
- Am I correct in observing the Algorithm 1 does not contain the actual inference procedure of the target value of $\theta$? Where in the main text is the inference procedure discussed? 

summary_of_the_review: Small issues with the writing but the ideas seem new and to work quite well!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method that learns structural relationships between variables of the RL system so as to be able to adapt to changes across domains, and learn a new policy in a new domain with few samples. The authors provide some theoretical results on their method, and evaluate their proposal on some standard environments that have been modified to evaluate this type of idea.
main_review: Overall I quite liked this paper as it introduces a well-motivated idea that can be quite useful to the research community. However, the proposed algorithm is rather involved and the empirical gains seem to be quite marginal relative to other methods in the literature (which are arguably easier to explain/implement).

My two main concerns are clarity and the quality of the empirical evaluations. More details below.

**Clarity**: The idea is somewhat involved and there are _lots_ of parameters and moving parts. While the authors have made a good effort to make it clear, there are still some parts that are hard to follow. Some more details below.

1.  Although Figure 1 helps somewhat, it is still rather difficult to follow. I would suggest moving Figure A3 from the appendix to the main paper.
1.  In the definitions of $\mathcal{L}^{rec}$ and $\mathcal{L}^{pred}$ the $\beta$ terms need to be explained more. Both $\beta_1$ and $\beta_2$ produce $o$s and $r$s, yes? Why are they conditioned on different parameters? I _think_ one is the Dynamics and the other is the Decoder model (as illustrated in Figure A3), but it's not clear. Indeed, the sentence immediately after says that "$p_{\beta_1}$ and $p_{\beta_2}$ denote  the generative model with parameters $\beta_1$ and $\beta_2$". The use of the singular "the" suggests that it's the same model with different parameters, which I don't think is the case.
1. In Algorithm 1, are $Q$ and $Q'$ initialized independently or in the same way? Does it matter?
1. The rightmost panel in Figure 2 is difficult to understand. Maybe show more context of the Pong game so it's clear that this is a paddle and a ball.

**Empirical evaluations:**
1.  Although I can appreciate the environment modifications the authors used to showcase their algorithm, there are already existing benchmarks (such as [ProcGen](https://openai.com/blog/procgen-benchmark/), [Jumping Task](https://github.com/Maluuba/jumping-task), and [Atari modes](https://arxiv.org/abs/1810.00123)). Why were none of these used to evaluate the proposed method?
1.  The bold red fonts in the tables at the end are misleading, as there is _major_ overlap between the AdaRL confidence intervals with many of the other methods, which means that the gains are often not statistically significant. This is also evident in the appendix figures (e.g. Figures A6 and A7). There seem to be no statistically significant gains to be had from AdaRL.

---

Some questions for the authors:
1. In the definition of $\mathcal{L}^{rec}$, what is the range of $q_{\phi}(\cdot |\theta_k)$? $S$? Writing the expectation subscript as $s_{t,k}\sim q_{\phi}(\cdot | \theta_k)$ would help clarify things.
1.  In the last sentence of the **Modified Cartpole setting**, does $\theta_k=\{\theta_k^o\}$ mean the only variations are in observations?
1. In page 8, right above the **Baselines** section, it says $r_t=\frac{kL}{d+3L}$. Is $k$ the actual index used in the computation of $r_t$? If not, use a different letter.

---

Some other minor points:

1. The authors use the word "parsimonious" a lot, which seems strange. The Oxford Dictionary says its definition is _"extremely unwilling to spend money"_, which seems like a strange adjective to apply to an RL algorithm.
1.  In the Introduction the authors should also reference [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://arxiv.org/abs/2101.05265). I know it is referenced later in the paper, but it should also be discussed in the introduction when other representation learning methods (such as (Zhang et al. 2021)) are being discussed.
1.  PNN in the introduction is not defined.
1. At the bottom of page two it says "We learn an optimal policy", but to be precise it should be "a near-optimal policy".
1.  In Equation 1 the index $k$ is used. Later in the paper it becomes clear that it indexes domain, but this should be clarified before equation (1).
1.  At the bottom of page 4 the term ${\bf y}_{1:t,k}$ is used, but this variable has not been introduced yet (it is only introduced at the top of page 5).
1. Right above section 3.2 it says "In this way, except $\theta_k$", it's missing a "for" before the $\theta$.
1. In Algorithm 1 (e.g. lines 7 and 13) use `\cdot` in the posterior (e.g. $q(\cdot | o_{\leq t+1, ...})$), as the posterior at this point is a distribution, not a materialized probability.
1. In page 2 of the appendix there's a missing "}" in the first line of the last paragraph.
1. In the first bullet of page 2 of the appendix it should say "then it ***means*** there is no path".
summary_of_the_review: I think this is an interesting paper, but it falls short in the empirical evaluations. I think the idea is interesting and the theoretical results may be of interest to others working in this space.
However, I don't feel the empirical performance justifies the complexity of the algorithm, which is why I'm somewhat borderline, but leaning towards an accept.
I look forward to reading what the other reviewers think, as well as the authors' response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The submission proposes a method for transfer in reinforcement learning building on estimating a small set of factors describing a system and modelling the agent as a dynamic bayesian network. In detail, the method splits into domain shared and domain dependent factors which are modelled via a new form of structured sequential VAE. The VAE is trained to enable dynamics, observation and reward prediction (including reconstruction, prediction, KL and sparsity regularisation losses).

Data for VAE training is collected from random policies and model fitting is followed by policy training on source domains to be transferred to target domains by identifying domain specific features for the target domains.

The method is evaluated on variations of a cartpole and a pong domain (both from images) and evaluated against a set of recent, competitive baselines.

main_review: The paper investigates the question of transfer via identifying a small number of changing variables to maximise data efficiency. It is overall clearly written and easy to follow for most parts. 

The three step training framework with a split between model estimation from random data and policy learning can lead to a set of challenges which should be mentioned and discussed. First, random data might not cover enough of an environment's state space such that the estimated model is not accurate for later stages of policy training when a different part of the state space is visited. Second, splitting model estimation (focused on reconstruction and prediction) and policy training can lead to model features which are suboptimal for the policy. Third, the policy will be trained with the feature distributions of a set of source domains. Any function approximator used for the policy will need to handle out of distribution data on target domains which can differ from these features. In particular the second point could be investigated by iterating between model estimation and policy training.

The evaluation focuses on new variations of known domains opening the question why no existing benchmarks e.g. from the metalearning literature are used (e.g. benchmarks from the set of baseline papers such as the cheetah or ant domains from MAML). Further, it is unclear why e.g. MAML, Progressive Networks, etc (which are used in the MDP setting) are not applied to the POMDP setting. Vice versa for many POMDP methods which are not constrained to the use of POMDPs.

The use of image based domains does not need to lead to a POMDP. In both cartpole and Pong, occlusions do not occur. If the agent stacks multiple frames, velocity as part of the state is included. If not, it will be helpful to emphasise this aspect to describe which aspects are unobserved.

While overall well written, parts about equations and the general model are hard to trace and a visual representation of various parts would be helpful. See minor points for details.

Minor:
- description of meaning for masking parameters (c’s) should be move closer to Eq. 1
- a graphical representation for Eq. 1 (definition 1 with respect to compactness would benefit from an example graph either in the appendix or, if space can be found, in the main paper)
- Section 3.1. describes the training of the VAE via SGD. Including ‘in one step’ in the title seems inaccurate here. I could not find any justification for this term in the section.
- Explicit pseudocode for algorithm 1 is very helpful!
- The related work section provides a broad background but work on learning an embedding space for system dynamics to enable quick adaptation for sim2real transfer should be added [1,2].


[1] Yu, Wenhao, et al. "Preparing for the unknown: Learning a universal policy with online system identification." arXiv preprint arXiv:1702.02453 (2017).
[2] Peng, Xue Bin, et al. "Learning agile robotic locomotion skills by imitating animals." arXiv preprint arXiv:2004.00784 (2020).

summary_of_the_review: A strong submission on improving transfer learning which leaves a couple of open questions regarding its evaluation. However, these questions should be easy to address in the rebuttal.
review_title: null

Meta-review: The authors present a method called "AdaRL" that learns a structured latent representation that characterizes relationships between different variables in an RL system. The method is evaluated on modified Pong and Cart-Pole domains and it is shown to outperform other transfer learning baselines. The reviewers agree that the method makes sense and addresses an important problem of transfer in RL. The authors did a good job in the rebuttal to empirically validate their claims and provided extra experiments. The reviewers also point out that the evaluated domains are rather simple and the paper would benefit from evaluations in a more complex environment as well as better writing. Please focus on improving these aspects in the final version of the paper.
Decision: Accept (Spotlight)