Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the problem of learning generalizable context in RL. In particular, it suggests learning disentangled context representation of each confounding in the environment using the proposed model, DOMINO, which optimizes decomposed MI objectives. It adopts the contrastive learning method when learning the disentangled context representation, regarding trajectories sampled from the setting of the same confounding as positive pair and of different confounding as negative pair. The authors also provide a theoretical basis for how optimizing their decomposed MI objective can make $I_{NCE}$ a tighter lower bound by alleviating the underestimation of MI. By learning policy conditioning on the learned context vector, DOMINO can achieve higher generalization performance compared to both model-based and model-free baselines.
strengths_and_weaknesses: Strengths:

The paper is well written and clear to understand. Using contrastive loss when learning disentangled representation of each confounding is novel and intuitive. And it is intriguing to get an idea of sampling negative pairs from different episodes. The experiments are comprehensive and the results are impressive.


Weaknesses:

However, the proof of Lemma 1 and Theorem 1 lacks mathematical rigor. Also, there is some missing specific information about notations in the proof, thereby undermining the clarity and soundness of the paper (e.g., $w_y$ and $E$). Visualization of the learned context embeddings does not show how effectively each confounding is encoded.

questions: - In Appendix A,
  - How does the first equility ($q(y | x, y_{2:K}) = p(y) K w_y$) hold?
  - What is $w_y$?
  - What does $E$ mean in $I_{NCE}(x;y | E, K)$?
- In Theorem 1,
  - Even if the number of confounders increases, the true mutual information $I(c; \mathcal T)$ does not. Therefore it shows inconsistency to regard $I_{NCE}(c; \mathcal T)$ and $I_{NCE}(c_i; \mathcal T)$ as having the same upper bound.
  - The following inequality also holds in the same setting. $\sum I_{NCE}(c_i;\mathcal T | K) = I_{NCE}(c;\mathcal T| K) \le log K$ 
- In Figure 6,
  - DOMINO encodes the context vector of each confounder mass and damping. However, Figure 6 does not show how effectively each confounder was encoded.
  - For example, Setup0 and Setup4 have the same damping condition and Setup0 and Setup3 have different mass and damping conditions. Then shouldn't Setup0 and Setup4 be located closer than Setup0 and Setup3?
limitations: Yes, the authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a contextual reinforcement learning (RL) setting where the environment dynamics are parameterized by independent factors, which the authors refer to as “confounders.” In each episode, the underlying factors can vary. They present a method for contextual meta-reinforcement learning (RL) called DOMINO, which learns to encode the RL agent’s current trajectory into a set of independent context vectors. These independent context vectors can then be used as inputs to the transition model in model-based RL (MBRL) and as an input to the policy in model-free RL, thereby providing the agent with an inferred context for the underlying environment factors in any given episode. Importantly, their method assumes the underlying environment factors are similarly independent. The main contributions of the paper are the method, DOMINO, for learning independent context vectors from the trajectory and their analysis and experimental results demonstrating the favorable properties of this method (including improved empirical performance against baselines learning entangled context vectors), when the underlying independence assumptions are valid.
strengths_and_weaknesses: Strengths

- The paper provides a simple method for improving context-aware meta RL in an environment with multiple independent factors of variation that impact the transition dynamics. The method itself is clearly described. This seems to be the first method to directly exploit an explicit assumption of independence among the underlying environment factors of variation.
- The method performs well against sensible baselines. Importantly the method performs well against an ablation that does not learn disentangled context vectors.

Weaknesses

- The reported results in the Table 1 and 2 have high overlap between the authors’ DOMINO and MINO methods and the baselines. The signficance of these results could be made clearer by reporting the results of a Welch t-test between the proposed method and the baselines.
- Similarly, the performance comparison plot in Figure 1b should have error bars. It should also state what method of averaging was used for the plotted values
- The paper can benefit from a full pass to improve the clarity of the writing. There are numerous missing details about basic figures, such as what measure of uncertainty is represented by the error bars for each plot and table. There are also several ambiguous phrasings and sentences with confusing wording. For example
- A key aspect of this paper is the analysis of InfoNCE as a “loose bound” of the mutual information. However, the authors never define whether this bound is an upper or lower bound. While this detail can be inferred from context, I think it is important to make this point clearer to the reader. Relatedly, the definition of “MI underestimation” in L45 is unclear.
- Given that the independence assumption is core to this work, it is unclear how significant this setting will be in practice and for future work.
- Moreover, it seems important for the experiments to assess how valid such an independence assumption is in practice, and crucially, what is the price in performance one might expect to pay for making this assumption. An experiment assessing the performance of DOMINO and MINO on a more complex environment whose underlying factors of variation are not mutually independent would improve this paper by providing a more complete picture of the effectiveness of this method.
- There seems to be an underlying assumption that the N independent context vectors aim to encode information about the underlying factors of variation in the environment. However, this connection is actually never explicitly made in the writing, making the jump from discussing MI in terms of environment factors to context vectors (4.1 to 4.2) unclear.
- It seems that DOMINO requires setting the number of context vectors N equal to the number of environment factors of variation. In general, we may not know this value exactly. Adding a sensitivity analysis to how dependent the performance is on setting N to this exact value would provide important information on how applicable this method is in practice.

Minor comments:

- L22: “mythologies” should be “morphologies”.
- L47-48: “First the context encoder embeds the past state-action pairs into disentangled context vectors” is an inaccurate description, as it must first be optimized to do so (as next described in L48-49).
- This paper could consider citing related work in unsupervised environment design [1,2,3,4] and more generally, RL work in procedurally-generated environments [5,6]. These works are deeply related as they effectively perform meta-RL over a space of environment variations with an implicitly learned context. Ignoring this line of work seems like a significant oversight.

References

[1] Dennis et al, 2020. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design.

[2] Jiang et al, 2021. Prioritized Level Replay.

[3] Jiang et al, 2021. Replay-Guided Adversarial Environment Design.

[4] Parker-Holder et al, 2022. Evolving Curricula with Regret-Based Environment Design.

[5] Raileanu et al, 2021. Decoupling Value and Policy for Generalization in Reinforcement Learning.

[6] Cobbe et al, 2019. Leveraging Procedural Generation to Benchmark Reinforcement Learning.
questions: - Could the authors elaborate on their choice of T-MCL as the sole baseline in their MBRL experiments? Further details on this choice would benefit the clarity of the experiments section.
- Further, since the goal of the method is to perform efficient meta-test adaptation via context vectors, why was PEARL chosen over Varibad, which has been shown to provide much more efficient within-episode adaptation compared to PEARL?
- It seems that the number of context vectors N must be set to the number of environment factors. Is this understanding correct?
- This paper assumes the environment confounders impact the transition function, but not the reward function. How do the authors view the role of reward generalization in their work? Can DOMINO be expected to work in settings where the environment confounders also impact the reward function?
limitations: The core assumption of this work also acts as its primary limitation: The environment factors of variation are assumed to be independent, and their number known a priori. The authors should make an effort to emphasize this limitation and to what extent they believe such an assumption of independence may be applicable in practice.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of generalization in MDPs where the dynamics changes are assumed to be caused by multiple independent factors, denoted as context. The proposed framework (DOMINO) learns a context encoder that maps trajectories to a latent context via decomposed mutual information using noise-contrastive estimation (InfoNCE). The authors combine DOMINO with model-free and model-based RL algorithms, and perform experiments in classic environments, as well as in the Mujoco benchmark, in settings where multiple confounders change simultaneously. Additionally, qualitative visualizations of the latent context vectors are presented using t-SNE.
strengths_and_weaknesses: Strengths:
- The idea of capturing the different confounders that may affect the dynamics of the MDP into different latent contexts is novel and interesting.
- The experimental results show that the proposed method can, in general, achieve better performance than the state-of-the-art.

Weaknesses:
- The paper needs improvement regarding the clarity of the mathematical definitions, such as the objective functions.
- It is not clear whether the improvements are because of the decomposed mutual information framework, or because of other algorithmic improvements (see below).

questions: Furthermore, I have the following questions and constructive criticisms:

- “Our goal is to learn a generalized context encoder, which is learned in training process by maximizing the expected rewards R_train in seen environments, and zero-shot generalized to unseen environments for a high expected rewards R_test” 
This sentence is hard to follow. The objective is also not very clearly defined in Eq. 1. For instance, how is the context $c$ generated? How do the confounders $u$ affect the expectation?

- In Section 4.2, it is not clear the difference between the two context encoders, $g_\phi$ and $h_w$? Importantly, why is $h_w$ necessary? Notice that $h_w$ does not appear in the pseudo-code of the algorithms in the appendix.

- It is not clear what are the parameters being optimized through the loss functions in Eq. 5, 8, and 9. I suggest specifying which parameters ($\psi$, $\phi$, etc.) are involved in the gradients of each loss function.

- “The critic function $\psi(·, ·)$ measures the similarity between inputs by dot product.” Why was dot product chosen as the similarity measure? Can you elaborate on this decision?

- “The results illustrate that DOMINO learns the policy more efficiently and stably than the baselines.” 
It is not possible to infer that DOMINO learns more “stably” than the baselines. In fact, sometimes it shows more variance (see Fig. 3 - Halfcheetah-m-d) than the baselines.

- Why does MINO perform better than T-MCL? It seems the improvements of MINO (in comparison to T-MCL) are more important than the improvements of DOMINO (in comparison to MINO).

- In Figure 6, it is not clear what each point represents. Given a trajectory, the context encoder outputs N different context vectors. In this figure, did you concatenate all vectors before applying t-SNE? 

- The key characteristic of the proposed method is the fact that it should learn disentangled and independent latent contexts. However, the visualizations using t-SNE were not able to show that. An important result would be to show that indeed each latent context captures one of the confounders of the environment. This could be shown by varying only one of the confounders and observing whether only one of the latent contexts changes accordingly.

- The authors state in the checklist that they have included the code, data, and instructions needed to reproduce the main experimental results in Appendix D. However, only pseudocode of the algorithms and code for the environment were made available. The actual code of the proposed methods is not available.

- In the paper’s abstract, it is said that the open-sourced code and videos are released on their anonymous homepage. However, there are no videos on this page, and only code for the environments is available, not the algorithms/training code.

limitations: The paper could benefit from a discussion regarding the assumption of independent confounders. For instance, how difficult it would be to adapt the algorithm to the case where we have co-related confounders?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a decomposed mutual information method to learn disentangled context information, which can generalize reinforcement learning algorithms into unseen environments. The experimental experiments demonstrate that the proposed method can achieve better performance than the previous methods.
strengths_and_weaknesses: Strengths:
1. The writing of this paper is pretty well, and the idea of it is easy to follow.
2. The figures in this paper are very clear and very well.
3. The extensive experiments show the effectiveness of the proposed method.

Weakness: 
1. Based on the title, I assume that this study focuses on the meta-reinforcement learning problem. The conventional meta-reinforcement learning methods include an adaptation process, but this paper makes no mention of this process. Additionally, the paper states that it intends to train a general context-encoder to solve the adaptation problem, indicating that the paper's context is the dynamics generalization in reinforcement learning (this paper also mentions it in line 84), which is in contrast to the title of the paper, which refers to meta-reinforcement learning.

2. The second problem of this paper is the novelty. The paper aims to maximize the mutual information between contexts extracted from historical information and the historical trajectories. However, this paper does not make clear the relationship with [1,2,3] which also attempt to maximize the MI between context vector and historical trajectories. Furthermore, this work does not compare the performance with [3] and even does not acknowledge it, despite the fact that [3] focuses on a similar problem to this paper. As a result of the missing contribution and experimental comparisons with [1,2,3], I believe this paper's uniqueness is somewhat limited.

3. The number of learned context vectors $c$ is set as the number of environments in the study, which is the primary hyperparameter of the suggested technique. However, in a real-world setting, the number of environments is not available, making it unfair to compare it to the baseline TMCL, which doesn't rely on such prior information. This increases my concerns about the technical soundness of this paper.

In conclusion, while the writing and experimental results are excellent, this paper suffers from the aforementioned clarity and novelty issues. If the authors address my concerns in their response, I will consider raising my score.


------------------------------------------------- After Rebuttal ------------------------------------------------

I think that the additional experimental results and discussion in the revision resolve my concerns about the clarity problem of the submission, so I increase my score from 4 to 6 accordingly.

Minors: I believe that RIA considers context information and constructs confounder sets with multiple confounders, so I believe that RIA should be discussed in the introduction's confounder discussion (Line 42).

[1] Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: an approach based on contrastive learning. 

[2] Li, L., Huang, Y., Chen, M., Luo, S., Luo, D., & Huang, J. (2021). Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning. arXiv preprint arXiv:2102.10774.

[3] Guo J, Gong M, Tao D. A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning[C]//International Conference on Learning Representations. 2022.
questions: Please refer to the "Weakness" listed above.
limitations: Please refer to the "Weakness" listed above.
review_title: null

Meta-review: metareview: This paper proposes DOMINO, an optimization framework, for contextual meta reinforcement learning. The reviewers generally agree that the paper is well written, the idea is novel and interesting, the evaluation is comprehensive and the results are impressive. Reviewers also raised a few concerns in the initial reviews, such as the proof of Lemma 1 and Theorem 1, and the mathematical definitions. Throughout the discussion phase, most of these concerns were sufficiently addressed, and the review scores were increased accordingly. Overall, the quality of the revised paper has improved significantly during the rebuttal. Thus, I recommend accepting this paper. Please incorporate the remaining reviewers' suggestions in the future version of this paper.
confidence: Certain
Decision: Accept