Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a framework to use language models like transformers to be used for decision making in interactive environments. The authors also present the features of this approach and present experiments to verify the same. The key contributions are identifying that language models for decision making in RL-type settings improves combinatorial generalization, an active data gathering approach to account for less than ideal pre-collected expert data and how sequentiality is very important in all these encoding techniques for planning in the environments discussed in the paper.
strengths_and_weaknesses: Strengths: The paper presents the hypotheses very well and also lays out a rigorous enough framework to test the hypotheses. It lays out a framework to use a general large language model to be used in place of existing non-transformer based policy networks. The authors touch on the importance (or lack thereof) of the encoding scheme, how to convert structured pieces of instructions into language and testing the framework for achieving the goal state. Finally the piece about active data gathering is very significant and can be expanded in general to a lot of other RL methods to compensate for lack of expert data in RL environments

Weaknesses: Even though there are experiments in section 7 trying to give an intuitive feel for the reason this method is working, there can be more discussion from a model-structure perspective on how the internal structure of the transformer is creating these embeddings for policy networks. 
questions: Q1: In Table 2, VirtualHome seems to be a very tough environment for any policy that isn't LID-ADG. I'd like to see comparison against other methods which have significance performance on this: even though they may not be using ADG, it would be nice to see how much can ADG push a method against a better performing model which might have more expert data.

Q2: Can you provide more intuition/experiments on how different layers of GPT-2 in terms of self-attention are lighting up when a task is being performed? This question is aimed to poke at the structure of the transformer and how different layers are being used for interactive decision making

limitations: Authors don't discuss the limitations enough of the model in the environment of VirtualHome and BabyAI. I'd like to see a discussion of failure modes of the model and how that traces back to the encoded embeddings.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to utilize the pre-trained language model (PLM) for solving interactive decision-making problem, named LID. Specifically, the policy network consists of a PLM and tailored task modules. The policy information (goal, history, information) will be described as the natural text and encoded by GPT-2 PLM. During the fine-tuning process, the PLM and task-specific modules will be jointly optimized (the authors also study the different training paradigms). To alleviate the scarcity problem of expert data, the authors also propose an active data gathering method to make the agent learn in a self-supervised way. The BabyAI platform and VirtualHome platform are used to evaluate the performance of the LID framework. Several empirical experiments and the corresponding analysis show the rationality and generalization ability of the proposed LID framework. 

The contributions of this paper are two-folded: 1) proposing a PLM-based framework for interactive decision-making task, i.e., LID in this paper; 2) proposing a self-supervised data gathering method, which provides a stable RL training process. The authors also analyze the source of the generalization ability of LID framework, which can provide a clear direction for future research.

strengths_and_weaknesses: **Strengths**:
1. ***Novelty***: It is not surprising that the "pre-training LM then fine-tuning" paradigm is widely used in natural language processing tasks. Although the authors also follow this paradigm, it is interesting to see that the sequence modeling ability of PLM can be used in policy learning.
2. ***Significance***: The proposed framework is well supported by empirical studies and evaluated on two commonly used benchmarks. This new framework powered by PLM can be beneficial to the research community of policy learning.
3. ***Quality***: This paper is technically sound, and the authors provide enough technical details in the Appendix including network architecture, encoding methods, etc.

**Weaknesses**:
1. ***Scalability***: The proposed encoding method is templated-based (Line 155-156). Although the input encoding scheme (Section 7.1) may be a trivial problem, the encoding scheme may still affect the performance. Searching for the optimal encoding scheme is an expensive process, which may bring a high cost of hand-crafted engineering.  Besides, the data gathering method also relies on hand-designed templates (Line 220).
2. ***Presentation***: The related work of PLM is adequately cited. But the authors should also introduce the background of policy learning so that the significance of this work can be highlighted.
3. ***Performance***: Compared to the work that uses traditional networks like DQN, the integration of PLM may affect the inference speed. 
4. ***Clarity***: Most parts of this paper are well written. However, there are some typos in the paper:
    - Line 53: pretrained LMs -> pre-trained LMs
    - Line 104: language -> language. (missing full stop mark)
    - Some papers should be cited in a proper way: Line 108: [23], Line 109: [36], Line 285:[15], Line 287 [15]. For example, in Line 108, "[23] show that" needs to be rewritten as "Frozen Pretrained Transformer (FPT) [23] show that". 

[Rebuttal Updates] The authors provided the additional experiments for addressing my concern of scalability. The authors also revised the typos and added the related works.


questions: 1. Pre-trained language models limit the length of input text. For instance, the maximum input length of GPT-2 [1] is 1024 [2]. Will the length of encoded text in your experiments exceed this number? If not, can you figure out a solution for handling long-text input? Because the environment and policy description could be too long, we may not neglect this situation.
2. In the exploration stage (Section 4.2.2), what is the exact sampling method for the goal and initial state?
3. The BabyAI 1.1 baselines [3] report the mean ± std in their papers. However, the evaluation of the BabyAI in this paper (Table 1 and 4) did not use different random runs. I did not find any justification in Section 5.2. Could you please offer an explanation?  

[Rebuttal Updates] I confirmed the authors' answers.

Reference:

[1] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). [Language models are unsupervised multitask learners.](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf).

[2] https://huggingface.co/gpt2/blob/main/config.json

[3]  Hui, D. Y. T., Chevalier-Boisvert, M., Bahdanau, D., & Bengio, Y. (2020). [BabyAI 1.1.](https://arxiv.org/abs/2007.12770)
limitations: - Societal Impact: No potential negative societal impact. The authors provide a new perspective to aid policy learning with a pre-trained language model. 
- Limitation: 1) Building text descriptions for each task still requires human labor. We do not know what textual format is optimal for policy learning. It varies from task to task, model to model. On the other hand, as I stated in Question 1, the long-text input could restrict the scalability of this framework. 2) The proposed methods also need humans to design some templates/rules, as the authors mentioned in the conclusion part.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces LID, a framework that encodes the goal, history, and observation as the tokens and fine-tunes a pre-trained GPT-2 on the next action prediction task. The authors show empirically that such a pre-training mechanism significantly improves performance on both VirtualHome and BabyAI tasks. The proposed method is especially useful when only a limited number of demos are available. Their LID-ADG is able to make meaningful predictions on VirtualHome with any expert data (unlike baselines which get about 0% accuracy.

strengths_and_weaknesses: ### Strengths
- The proposed method is novel, simple, and quite effective.
- The authors conduct a deep analysis in Section 7 to provide deeper understanding of various design choices in LID.
- The paper is easy to read.

### Weaknesses
- One concern is about the reproducibility of the paper. The authors do not provide the detailed architecture and hyperparameters of the method. Although there is a short description in Appendix D.2. I don't think it is sufficient for reproduction.
- It would be better if the authors can provide standard deviations in Tables 1 and 4. With a small number of demos, I believe that the standard deviation would be large.

### Summary
Overall, the proposed method is sound and effective, so I recommend accepting it. However, the tasks studied in this paper are not in my research area, so I am not confident with my judgment.

### Violating the formatting instruction
Notably, the authors include the appendix at the end of the main paper submission instead of putting it in the supplementary material which violates the NeurIPS format. I wonder if it is acceptable.
questions: - Are all the tokens in brackets (such as [grab] and <apple>)  treated as individual special tokens and added to the tokenizer or they are just treated as normal takes?
- I don't see the reason why the authors only use autoregressive language model. The next action prediction task doesn't seem to require an autoregressive model since it's not a sequence prediction task. I wonder how bidirectional Transformers such as BERT and RoBERTa perform.

limitations: Yes, the authors discuss them in Section 9.
review_title: null

Meta-review: metareview: This paper adapts the "pretrain-then-finetune" framework to policy learning using large language models and demonstrates its effectiveness. They also develops an active expert data gathering approach for settings where no expert data is available. All reviewers find the empirical findings in the paper interesting and the work technically solid. This paper may spur more work in using pretrained language models in RL settings. I recommend acceptance.
confidence: Certain
Decision: Accept