Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to learn a language-conditioned dynamics model (without action labels), and shows that the representation learned by this dynamics model can improve downstream policy training, when combined with necessary techniques such as distillation. The method is evaluated on the SILG benchmark, which includes several distinct environments. Compared to baselines such as pure RL, inverse RL, reward-shaped RL, the proposed method is shown to outperform or on par with the best of all baselines across the benchmark. It is also ablated across several variants to show dynamics modeling with language outperforms without language, and that using expert trajectories for dynamics modeling is important.

--------------
Post Author Response Update

The authors have addressed my major concern for the paper. I'm updating my review score to 6.
strengths_and_weaknesses: Strengths:

1. The paper text is written with clarity.
2. The method appears to be novel for the reviewer.
3. The paper investigates important question in the field: how can language modeling help decision-making problems?
4. The evaluations include several distinct environments and tasks.

Weaknesses:

My concern is mostly around the significance of the paper. While it’s shown to be better than various baselines, the paper is not contextualized well enough for various prior works in representation learning for RL. While many prior works in this domain may not involve language modeling, given the nature of the proposed method, it should be discussed how it differs from other representation learning methods for RL and positions the paper from this perspective. Moreover, while the paper shows that the proposed method outperforms various baselines, it is susceptible that the improvement mostly comes from the good representation modeling. Therefore, this can only be a meaningful contribution if the paper could show comparisons to prior works in representation learning for RL. For example, what if the expert trajectories are used to train a representation module that is not a dynamics model, using some objectives? Would this representation work well in these tasks? If not, is it because they don’t model language?

Minor improvement include:

1. While the text is relatively clear in the paper, Figure 1 is poorly made, with blurry environment pictures and unclear method visualization. Potential improvement, for instance, can be separating the “traditional training loop” from the proposed method.
2. Fig 6b seems to be missing several learning curves.
questions: Please refer to the comments in the weakness section.

To summarize:

1. It is suggested to better contextualize the work around representation learning for RL, while advocating the importance of language modeling from this perspective.
2. In this regard, any comparison to a well-known or state-of-the-art method in “representation for RL” could be very useful to evaluate the significance of the contribution of this work.
limitations: The authors did not adequately address the limitations of this work; it is only stated that the limitation is that the environments are simulated. This is an insignificant limitation and not useful to evaluate the significance of the contribution of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes to ground policy representations on representation learned from language conditioned dynamic modeling. The paper argues that this is a cheap approach to obtain langauge grounding from observations, and illustrates across a set of different textual domains how such an approach can lead to improved RL performance.
strengths_and_weaknesses: Strengths:

The paper is relatively clear to read, and the evaluation makes sense to me

Weaknesses:

The underlying proposed method seems a bit trivial to me -- the author simply proposed to finetune on a dynamics model.

The benefit of using expert data to train the dynamics model compared random data in Figure 7 seems to be minimal.

The underlying rewards curves of different models have very high variance. For example see the ablations in Figure 5.

The performance of the proposed method also appears to similar to other methods (see Figure 3)

Intuitively -- why does such a dynamic modeling objective lead to grounded language representations? Can the authors illustrate this more convincingly with more qualitative visualizations or quantitative grounding results? Also illustrating this qualitative difference with using random frames would be helpful.
questions: See above
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a new method, Language Dynamics Distillation(LDD), which pre-trains the model to predict environment dynamics with given demonstrations and language descriptions, then fine-tunes the final policy via reinforcement learning on pre-trained representations. The main contribution claimed by the authors is that this method can learn the RL policy more efficiently. It is especially effective when the reward is sparse or delayed, and it is a realistic setting because it assumes a cheap additional expert demonstration (ex. video) without action labeling.
strengths_and_weaknesses: Strengths
- It considers the realistic setting that assumes only cheap additional expert demonstration without action labeling.

Weaknesses
- It seems that the paper does not provide a clear explanation of the problem setting and the details of the proposed algorithm.
- It seems that the baseline algorithms provided in the experiment are not sufficient. It would be better if it was compared with several algorithms which are proposed to address the settings of sparse and delayed rewards. (ex. RND [1], ICM [2], …)

[1] Yuri Burda et al, Exploration by random network distillation, ICLR 2019

[2] Deepak Pathak et al, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017

questions: - I think the authors should have clearly defined and explained the problem setting (ex. about language description)  in Chapter 3. Does the defined s_t include a language description? If not, a detailed explanation of exactly how the language description is used seems necessary.
- Among the baseline algorithms, how was ‘reward shaping with expert’ (line 218-219) applied specifically?
- Can the proposed method be used with general intrinsic reward-related methods (ex. RND, ICM, …)? If so, I wonder if it still works effectively when applied with those methods.

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Sparse reward environments can be challenging for language-conditioned policies as credit assignment to the language description can prove difficult. The authors pretrain on a dataset of language-conditioned experts’ observations and distill this into the RL policy to demonstrate strong performance gains on a set of language-guided environments. I recommend accept for this paper, but would like to see a few things (mainly regarding ablations) addressed.
strengths_and_weaknesses: ## Strengths

**Writing:** The motivation/method are presented clearly.

**Novelty:** The method is seemingly novel yet simple.

**Results:** LDD seems to work pretty well and provide strong improvements over the baselines in the paper across the some of the environments.

## Weaknesses

**Writing:** The last paragraph of the intro is unnecessarily long and reads more like an experiments analysis section. The intro would be more readable by greatly shortening this paragraph and leaving the detailed experiments results to the experiments section.

**Missing Ablation:** While I am pretty sure this baseline would be outperformed by LDD, the authors should test LDD trained without language guidance (just predicting next states). I see appendix Figure H does this partly, but this only compares with/without language inputs for dynamics modeling accuracy, not downstream RL performance.

Additionally, the authors should also include at least one other baseline that uses the same dataset in some way. For example, perhaps an LDD type of model that is pretrained with contrastive learning or some other representation learning objective (predict whether  observations are from the same trajectory given the language description), to verify that the dynamics loss is the best commonly used objective to apply here.

**Minor issues:** 

- L220: “expert policy instead of for”
- Hyperparams: It’s useful for papers to at least have a small table/description of how hyperparameters were selected for all baselines and the main method, and how much tuning was done.
questions: None
limitations: None
review_title: null

Meta-review: metareview: This work proposes to learn better representations for language description-based tasks like navigation, via first pretraining a dynamics model on sequences of observations without action labels and using this model to aid RL-based policy learning. Good empirical results are presented and the work has been well-received. The dynamics policy learnt helps policy learning especially with longer horizons. The authors are encouraged to incorporate the reviewer feedback into account and especially the VAE experiments and add discussion.
confidence: Less certain
Decision: Accept