Pre-Trained Language Models for Interactive Decision-Making
Authors: Shuang Li
Xavier Puig
Chris Paxton
Yilun Du
Clinton Wang
Linxi Fan
Tao Chen
De-An Huang
Ekin Akyürek
Anima Anandkumar
Jacob Andreas
Igor Mordatch
Antonio Torralba
Yuke Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Decision-Making
Language Models
Combinatorial Generalization

Abstract: Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learning problems? We propose an approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented as a sequence of embeddings, and a policy network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to a set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the VirtualHome environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past "failed" experiences with new goals, and update their policies in a self-supervised loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1%. Finally, we explain these results by investigating three possible factors underlying the effectiveness of the LM-based policy. We find that sequential input representations (vs. fixed-dimensional feature vectors) and LM-based weight initialization are both important for generalization. Surprisingly, however, the format of the policy inputs encoding (e.g. as a natural language string vs. an arbitrary sequential encoding) has little influence. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.

Pre-Trained Language Models for Interactive Decision-Making
Shuang Li I+, Xavier Puig', Chris Paxton², Yilun Du', Clinton Wang', Linxi Fan', Tao Chen', De-An Huang', Ekin Akyürek', Anima Anandkumar Jacob Andreas Igor Mordatch*1, Antonio Torralba Yuke Zhu
MIT, NNidia, CCaltech, *Google Brain, 'UT Austin Junior authors are ordered based on contributions and senior authors ordered alphabetically.
Abstract
Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learn- ing problems? We propose approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the Virtual- Home environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past "failed" experiences with new goals, and update their policies in self sufe-suueeeieed loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1 %. Finally, we explain these results by investigating three possible factors underlying the effectiveness of the LM-based policy. We find that sequen- tial input representations (vs. dimenssiaa feature vectors) and LM-based weight initialization are both important for generalization. Surprisingly, however, the format of the policy inputs encoding (c.g. as natural language string vs. an arbitrary sequential encoding) has little influence. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.
sequence of embeddings, and policy
1 Introduction
Language models (LMs) play a key role in machine learning approaches to natural language processing tasks (9) This includes tasks that not purely linguistic, and require nontrivial planning and reasoning capabilities 24 [3: for example, instruction following, vision-language navigation, and visual question answering. Indeed, some of these tasks
distant from language modeling that one can ask whether pre-trained LMs can be used as a general framework even for tasks that involve no language at all. If so, how might these capabilities be accessed model trained only to process and generate natural language strings?
"oorrespondence to: Shuang Li <lishuang mit.cdux edu>>
Prrojcct page: https://shuangli http:///hhangll-rojjcee wwt.eett.ttbuuiieeeee github.io/Pre io/Pre-Trained Sraiiedd-Lnguaaad Language- -odeaa/-eoot--foo--tttttiieeeeee Making Part of this work was done during Shuang's 's internship NVIDIA.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

VirtualHome
Graphi parttal oleervationn
Goal
History
Observation o
Tolecnkanton
Tokenbaation
Tokrnization
Goal prodicadens Iustieiaannakesttanni...
appert
Language Mudel
BabyAI
Grid portiol olservation:
Language pool:
aprnt
Put the green next to the purple bex
Teckspecificc Model
Next action a
Figure 1: Environments (left): Different environments have different types of observations and goals. Our approach (right): We pre-trained LMs general framework for interactive decision-making by converting policy inputs into sequential data. Such method enables effective combinatorial generalization to novel tasks.
In this paper, study these questions through the lens of embodied decision-making. investigating the effectiveness of LM pre-training as a general framework for learning policies across a variety of environments. We propose LID, framework that uses Pre-Trained Language Models for Interactive Decision-Making As shown in Figure [ (right), we encode the inputs to a policy-including observations. goals, and history- as a sequence of embeddings. These embeddings are passed to a policy network initialized with the parameters of pre-trained LM, which is fine-tuned to predict actions. This framework is broadly applicable, accommodating goals and environment states represented as natural language strings, image patches, or scene graphs.
We find that imitation learning using pre-trained LMS as policy initializers improves in-domain performance and enables strong generalization over novel tasks. For i.i.d. training and evaluation tasks, this approach yields 20% more successful policies than other baseline methods in Virtual- Home 31 For combinatorial generalization to out-of-distribution tasks, i.e. tasks involving new combinations of goals, states or objects, LM pre-training confers even more benefits: it improves task completion rates by 43.6% for novel tasks (see Figure 3). These results hold for a variety of environment representations: encoding states as natural language strings, when possible, improves the data- data-efficiency of training, but even LMs fine-tuned on random environment encodings generalize combinatorially to new goals and states when trained on large enough datasets.
We further examine how our method may be used in environments where expert data is not available, and agents must instead actively gather data. To do this, we integrate Active Data Gathering (ADG) procedure into pre-trained LMs as shown in Figure 2 Our proposed approach to ADG consists of three parts. First, exploration collects trajectories using a mix of random actions and actions generated by the current policy. Exploration is insufficient in this high dimensional problem and most of the trajectories will likely fail achieve the end goal. A key insight that even the failed trajectories contain useful sub- sub-trajectories that solve certain sub-goals, and we relabel these goals in a hindsight relabeling stage. The relabeled goal describes what was achieved in the extracted sub-trajectory. The policy update stage samples relabeled trajectories to update the policy. The active data gathering procedure allows us train the LM-policy without pre-collected expert data. also outperforms reinforcement learning (RL) methods on embodied decision decision-making tasks and enables more effective generalization to novel tasks.
Finally, we investigate why LID contributes to generalization. We hypothesize three possible causes for the effectiveness of LM-based policy initialization: (1) the use of language-basee input encodings, and more generally LMs' ability to reason about natural language strings; (2) the sequential structure of transformer inputs, in contrast to the fixed-sized observations used by most policy architectures. and (3) kusk-generdl inductive bias conferred by weight initialization with LM pretraining. We investigate (1) by encoding the policy inputs as different types of sequences. Different input encoding schemes have only negligible impact on the performance: the effectiveness of language modeling not limited to utilizing natural strings, but in fact extends arbitrary sequential encodings. We study (2) by encoding observations with single vector embedding, thereby removing its sequential structure. This operation significantly degrades the model's performance on novel tasks. Finally, we investigate (3) by learning the parameters of the policy from scratch. The success rate after removing the pre-trained LM weights drops by 11.20 11.2%, indicating that LM pretraining provides useful inductive bias for sequence processing even when sequences not natural language strings.

---

To summarize, our work has four main contributions:
First, we propose to use pre- preitedined LMs as a general scaffold for interactive decision-making across a variety of environments by converting all policy inputs into sequential data.
Second, we demonstrate that language modeling improves combinatorial generalization pol- icy learning: initializing a policy with pre-trained L.M substantially improves performance on novel tasks.
Third, we integrate an active data gathering procedure into the proposed approach to further enable policy learning on environments without using pre-collected expert data.
Finally, we perform several analyses to explain the generalization capabilities of pre-trained LMs, finding that natural strings are not needed to benefit from LM pre-training, but the sequential input encoding and weight pre-training are important.
These results point to the effectiveness the proposed framework with pre-trained LMs as a general- purpose framework promote structured generalization in interactive decision-makinn makinng.
2
Related Work
In recent years, word and sentence representations from pre-trained LMs 29 9 + have become ubiquitous natural language processing 49 30 Some of the most successful applications of pre-training lie at the boundary of natural language processing and other domains, as in instruction following [3] and language- gegguagg-gudeddd image retrieval [22].
Learning representations of language. From nearly the earliest days of the field, natural language processing researchers observed that representations of words derived from distributional statistics in large text corpora serve as useful features for downstream tasks 8 11 The earliest versions of these representation learning schemes focused on isolated word forms seen a number of techniques for training (masked autoregressive) language models to produce contextualized word representations (which incorporate information neighboring words in sentences and paragraphs) via variety of masked-wond prediction objectives 1 47
28 However, recent years have
Applications of pre-trained LMs. LMs can be fine-tuned perform language processing tasks other than language modeling by casting those tasks as word-prediction problems. Successful uses of representations from pre-trained models include syntactic parsing [19 and language-to-code
translation 45 successful adaptations of LM prediction heads include machine translation [49], sentiment classification (6] and style transfer [8] A number of tasks integrate language and other modalities, including visual question answering and image captioning 48 Recent works find that
image representations can be injected directly into LMs' embedding layers
12
Policy learning and LM. Traditional policy learning methods, such as PPO 3
DQN 27
DDPG (T)) A3C (26), perform well on playing tasks Atari, OpenAl gym [5] and MuJoCo
Some of them might fail to solve more challenging tasks on embodied environments B9 Several recent papers
[5] propose use LM for policy learning. Frozen Pretrained Transformer (FPT) 2 demonstrates that pre-trained LMs require very little fine-tuning to match the performance of fask-specific models on several image classification and numerical sequence processing tasks. Semi-SSuerrised Skill Learning with Latent Language ( 3X shows that LMs can serve as an effective backbone for hierarchical policies that express plans as natural language strings 2 4. In this paper, we focus on building a general framework for maaiing tasks using pre-trained LMs, even when language is not provided as an input or output.
3 Decision-Making and Language Modeling
3.1 POMDPs and Policy Learning
We explore the application of LMs to general sequential decision-maker tasks in partially observed environments. These tasks may formalized as partially observable Markov decision processes (POMDPs). A POMDP is defined by set of states, set of observations, set of actions, and transition model T(*+41 8e, at) that maps the current state and action to the next state. Importantly, in POMDP setting, the observation o only captures portion of the underlying state st, and an
3

---

optimal cerision-making strategy (a policy) must incorporate both the current observation and the history of previous observations and actions. In our experiments, policies are parametric models To g.. hi, 0t) that output the probability of an action given the goals y, history information hę {01, 01," ... Ot-1, a,-1)) ). and partial observations Og the current state 8t-
In Figure D (right), we show a high-level overview of the proposed method. We first convert all policy inputs into sequence and provide them input a transformer encoder. Representations from this encoder model are then passed to a task-specific decoder that predicts actions. We collect a dataset of N training trajectories D {d J where each trajectory consists of goal and sequence of observations and actions: d' = {g o ai.-.. T a'r, J. where T, is the length of the trajectory. We then train the policy maximize the probability of actions want to achieve a' {ai..... oiy across trajectories using the cross-entropy loss:
Φ² arg min
In To (atlal hoollo,n,... ot.
(1)
3.2 Language models as policy initializers
Our experiments focus on autoregressive, transformer-based LMs 43 These models are trained to fit distribution over a text sequence y {yr ) via the chain rule p(y) = p(y1) I[= P(y, .1..... 1)-1). ). Each term on the right hand side is parameterized by a transformer network, which accepts the conditioned tokens input. Each token passes through learned embedding layer Fe. then the full conditioned sequence is fed into the LMM. In our work, we use standard LM, GPT-2, to process the input sequence rather than predict future tokens.
Both POMDP maiiin and language modeling are naturally framed as sequence prediction tasks, where successive words or actions/observation are predicted based on a sequence of previous words or actions/observatioos.. This suggests that pre-trained LMs can be used to initialize POMDP policies by fine-tuning them to model high-reward expert trajectories, described below.
4 Approach
We evaluate the effectiveness of pre-trained LMs in solving decision-making tasks across envi- ronments. We use BabyAT [66] and VirtualHome (31 to evaluate the proposed method. While both environments feature complex goals, the nature of these goals, as well as the state and action sequences that accomplish them, differ substantially across environments (Figure I (left)).
4.1 Policy Network
We first examine whether pre-trained LMs provide effective initializers when states and action historics are represented as natural language strings. We encode the inputs to the policy-including observations, goals, and action histories- sequences of words. These word sequences are passed to the LM (using its pre-trained word embedding layer Fø) and used to obtain contextualized token representations. Token representations averaged and used to predict actions. We design a policy network following the general policy framework proposed in Figure] !
Environment encodings in VirtualHome. In VirtualHome, each goal consists of a sequence of pred- icates and multiplicities, and translated into a templated English sentence (e.g. "Inside (apple, fridge) :2" becomes "put two apples inside the fridge"). To encode the agent's partial observation, we extract a list of currently visible objects, their states (e.g. "open, clean"), ",) and 3D world coordinates. We use a fully-connected layer encode the 3D information and generate a feature representation of each object in the observation. To encode history, we store information about all previous actions and convert them into templated English sentences (e.g. "I have put the plate on the kitchen table and the apple inside the fridge").
Environment encodings in Ba The observation by default is a 7 7 grid. We convert the observation into 7 x 7 text deseriptions, e.g. "purple ball", "grey wall", "open door", and combine them into long sentence. We then convert the history actions into text descriptions, egg. "turn left" and "go forward"'. We combine the language instruction (without modification) with the observation and history text descriptions, and feed them to the pre-trained LMM

---

We note that the policy network described above does not strictly require that these encodings take the form of natural language strings-own other encodings of the environment sequence also work (see Section 7 This framework could be also generalized to support pixel-based observations using discretization schemes like the one employed in the Vision Transformer
[T0]
Action prediction. We pool LM outputs into a "context representation" that is used to predict the next action. In training, we maximize the probabilities of demonstrated actions. In inference, we select the valid action with the highest probability. See Appendix C.I for details.
VortualHome and Baby AI have quite different observation spaces, action spaces, and goal spaces; however, we show that embedding policy inputs as sequences and utilizing the pre-trained LM as a policy initializer, enables effective generalization novel tasks both environments. We note that LID is not limited to VirtualHome and BabyAI, but is straightforwardly applicable to other embodied environments, such ALFRED 40
and iGibson 39
4.2 Training
We first examine LID through imitation learning data collected by experts in Section 4.2.1 We then show that integrating an active data gathering procedure into LID enables policy learning without using expert data in Section 4.2.2
We use VirtualHome as an example to explain the data gathering.
4.2.1 Policy Learning with Expert Data
The policy model is first initialized from a pre-trained LM and then fine-tuned on data collected by experts. We build on the VirtualHome environment to collect a set of expert trajectories using regression planning 20 and create vertualHome-Imiaation Learning dataset. Given task described by goal predicates, the planner generates an action sequence to accomplish this task (See Appendix E. I.. The planner has access to privileged information, such as information about the pre-conditions and effects of each action, allowing agent to robustly perform tasks in partially observable environments and generate expert trajectories for training and evaluation.
4.2.2 Policy Learning with Active Data Gathering
Collecting expert data is sometimes challenging. It may require privileged information of the environment or hu- man annotations, which can be time- consuming and difficult to scale. A promising way to scale up supervi- sion is Hindsight Experience Replay (HER) B. which allows agents to learn from orders of magnitude more data without supervision. However, existing HER methods [ focus on simple tasks with small state/action space and full observability. They can- not tackle more complicated embod- ied decision-making tasks, requiring nontrivial planning and reasoning or natural language understanding. LID with the active data gathering (LID- ADG) can be used in solving tasks in such environments.
Navigation trajectory
1. Exploration
Interacted objects
Sample a random goals: Inside (milk, fridge)
Useful sub-trajectory for hindsight relabeling
Actions generated by the current policy or random exploration:
3. Pollicy update
[open] <kitchen cabinet>
[pur] <apple> <kitchentable>
2. Hindsight Relabeling
Relabel a task goal for the useful sub- trajectory: On (apple, kitchen table): 1
Figure 2: LID with the active data gathering procedure. By iteratively repeating the exploration, hindsight relabeling, and policy update, L.ID with active data gathering learn an effective policy without using pre-collected expert data.
As shown in Figure LID-ADG consists of three stages, iee. exploration, hindsight relabeling. and policy update. The key idea to gradually improve the task success rate by asking the agent to iteratively explore the environment, relabel failure samples, and update its policy using imitation learning. In the exploration stage, we first randomly sample a goal and an initial state. We then use mix of random actions and actions generated by the current policy To (g, hi, or) to obtain the next action. We repeat this process until this episode ends. We collect M trajectories and store them in the replay buffers. The generated actions in the early stages rarely complete the given task.
5

---

However, even the failed trajectories contain useful sub-trajectories that solve certain sub-goals. In the hindsight relabeling stage, we extract useful sub-trajectories and relabel a goal g' for each of them. We design a goal relabel function fi that generates a goal based on the sequence of observations and actions using hand-designed templates.
practice, we implement the goal relahel function as program (see Appendix F..2. The hindsight relabeling stage allows sample-efficient learning by reusing the failure cases. During policy update, the agent samples the data from the replay buffers and updates its policy network T..
By interleaving the exploration, hindsight relabeling, and policy update, LID-ADG can gradually improve the policy without requiring pre-collected expert data. In embodied environments with large action spaces, sparse rewards, and Long-horizon planning. RL methods often struggle to obtain stable policy gradients during training. Our method enables sample-efficient learning from the sparse rewards by relabeling new goals for the bad samples that the agent fails to achieve. In addition, LID-ADG leverages the stability of supervised learning in the policy update stage, enabling it to outperform RI. approaches on a wide range of decision-male making tasks.
5
Experiment Setup
We evaluate the proposed method and baselines on VirtualHome and Baby
5.1 VirtualHome
VirtualHome is 3D embodied environment featuring partial observability, large action spaces, and long time horizons. We evaluate policies' performance from three aspects: (I) performance on in-distribution tasks; (2) generalization to novel scenes; and (3) generalization to novel tasks.
In-Distribution. The predicate types and their counts in the goal are randomly sampled from the same distribution as the training data. The objects are initially placed in the environment according to common-sense layouts (e.g. plates appear inside the kitchen cabinets rather than the bathtub).
Novel Scenes. The objects are placed random positions in the initial environment without common- sense constraints (e.g. apples may appear inside the dishwasher).
Novel Tasks. The components of all goal predicates are never seen together during training (e.g. both plates and fridges appear in training goals, but Inside (plate, fridge) only appears in the test set. (See Appendix for more details.)
We evaluate the success rates of different methods on each test set. A given episode is scored as successful if the policy completes its entire goal within the maximum allowed steps of the environment. On each of the test subsets, we use different random seeds and test 100 tasks under each seed. Thus there are 1500 examples used evaluate each model.
5.2 BabyAI
BabyAl is a 2D grid world environment for instruction following. Observations BabyAl are 7 x 7 x 3 grids describing partial and local egocentric view of the state of the environment. We evaluate the methods on four representative tasks: GoToRedBall, GoToLocal, PickupLoc, and PutNextLocal. Performing well on the test set requires the models to generalize to new environment layouts and goals, resulting in new combinations tasks not seen in training. For each method, we compute success rates over 500 episodes on each task.
6
Experiments
We first show results of the proposed method and baselines for embodied decision decision-making tasks using expert data Section 6. We then show our results when using actively gathered data in Section 6.2
6.1
Embodied Decision Making with Pre-trained Language Model (LID)
6.1.1 Results on VirtualHome
We evaluate the following methods:

---

100
MLP-1
LSTM
Tasks
Methods
Number of Demos
Success Rate
75
MLP
LID-Text (Ours)
100
500 IK
10K
BabyAl-Ori [16] 81.0 96.0 99.0 99.5
99.9
GoToRedBall
50
L.ID-Text (Ours)
93.9
99.4 99.7
100.0
100.0
GoTolLocaa
Al-yAA-Ori [16] 55.9
84.3 98.6
99.9
99.8
25
1.ID-Text (Ours)
64.6 97.9 99.0
99.5
99.5
BabyAI-Ori [16] 28.0 58.0 93.3
97.9
99.8
PickuplIoc
0
LID-Text (Ours) 28.7 73.4 99.0
99.6
99.8
In Distiiibbbuton Novel Scenes Novel Tasks
PutNextLocal
BabyAI-Ori [IG] 14.3 16.8 43.4
81.2
97.7
Figure 3: Comparisons of the proposed method and baselines on VirtualHome. All the methods are trained on expert data using imitation learning. MIP-1, MLP,, and LSTM are baselines without using the pre-trained LM. The proposed method, LID-Text (Ours), outper- forms all baselines.
LID-Text (Ours) 11.1 93.0 93.2
98.9
99.9
Table
Success rafes on BabyAI tasks. All the methods are trained on offline expert data using imita- tion learning. LID- Text (Ours) outperforms Baby Al- Ori, the method used in the original paper [[6].
LID-Text (Ours) is the proposed method that converts all environments inputs into text descriptions. The pre-trained LM is fine-tuned for decision- decision-making making (conditioned on goals, observations, and histories) described in Section
Recurrent Network. We compare our method with a recurrent baseline using an LSTM [4! to encode the history information. The hidden representation from the last timestep, together with the goal and current observation, are used to predict the next action.
MLP and MLP-1. We perform additional comparisons with baselines that do not use recurrent networks or pre-trained LMs. MLP and MLP-/ take the goal, histories, and the current observation as input and send them to the multilayer perceptron neural network (MLP) to predict actions. MLP-/ has three more weeaae-ppolling layers than MLP that average the features of tokens the goal, history actions, and the current observation, respectively, before sending them to the MLP layer.
Quantitative results. Each method is trained on 20K demos from the Mattualoom-mmmmtttiin Learning dataset, and then evaluated on the three test subsets: In-Distribution, Novel Scenes, and Novel Tasks. In Figure 3 LID-Text (Ours). which initializes the policy with a pre-trained LM. has higher success rates than other methods. This difference is most pronounced in the Novel Tasks setting, where test tasks require combinatorial generalization across goals that are never seen during training. Here, LID-Text (Ours) dramatically (43.6%) improves upon all baselines. Such combinatorial generalization necessary to construct general purpose agents, but is often difficult for existing approaches. Our results suggest that pre-trained LMs can serve as a computational backbone for combinatorial generalization.
6.1.2 Results on BabyAl
We use the standard training and test data provided by [6. In BabyAl, performing well on unseen test tasks with new environment layouts and goals requires combinatorial reasoning. In Table D we report the success rate of models trained on different number of demos. BabyAI-Ori [I6] the method used in the original paper. LID-Text (Ours) is the proposed method that converts policy inputs into text sequence. Given enough training data, i.e. 10K demos, both methods achieve high success rates, but LID-Text (Ours) outperforms BabyAI-On with less training data, indicating the proposed method improves sample efficiency when generalizing novel tasks.
6.2 Pre-trained Language Model with Active Data Gathering (LID-ADG)
We compare LID-ADG, the proposed LM framework for decision decision-making using actively gathered data (Section 4.2.2). to variety of baselines that do not use pre-collected expert data VirtualHome.
Random. The agent selects the next action randomly from the valid action space at that state. Goal-Object. The agent randomly selects an object that in the goal and in the valid action space to interact with. For example, given a goal of "Inside (apple, fridge): 1",, this baseline might choose "grab apple", "open fridge", or other actions containing "apple" or "fridge". Online RL. We compare with PPO 37,, one of the most commonly used online RI. methods. For fair comparison, equip PPO with the same main policy network as the proposed method, Our implementation is
7

---

In-Distribution Novel Scenes Novel Tasks
In-Distribution Novel Scenes Novel Tasks
Random
0.0
0.0 1 0.0
0,0 0.0
Goal-Ohject
0,8 1881:00 0.5
0.0 1: 0.0
0.4
LID-ADG (Ours)
2.7
32.2 = 3,3
25.5
+ 4.1
PPO
0.0 + 0.0
0.0 0.0
0.0 0.0
PPO (LID-ADG Init)
53.7 3.5
30.2 4: 3.4
27.8 1: 2.7
DQN+HER
0.0
0.0 0.0
0.0
DT (LID-ADG Data)
42.4 + 1.5
21.6 2.48
16.8 + 1.0
LID-ADG (Ours)
46.7 +: 2.7
32.2 +.2++33. 3.3
25.5
35.5+:4.11
Table 3: The proposed method with active data gather- ing, LID-ADG (Ours), can be used as an policy initial- izer for online RL a data provider for offline RL.
Table 2: Comparisons of methods without using
expert data on VirtualHome. LID-ADG (Ours) is the only successful approach.
based on Stable Baselines3 in and modify its main policy network to be the same as the proposed method.
35 Hindsight Experience Replay. We compare with DQN+HER used
Quantitative results. We compare LID-ADG with baselines on VirtualHome in Table Each experiment is performed 5 times with different random seeds. The Random bascline is always 0, indicating the tasks in VirtualHome cannot be easily solved by a random policy. Goal-Object is better than Random because Goal-Ohject has access to objects in the goal and it samples actions from a much smaller action space. The online RL haseline, PPO, fails to solve tasks in VirtualHome featured by partially observation, large state/action space, and long-term horizon. DQN+HER works well on simple tasks on 2D environments, but they cannot tackle VirtualHome tasks neither, requiring nontrivial planning and reasoning. LID-ADG does not require expert data and can solve the complicated tasks in 3D embodied environments which cannot be easily achieved using RL. |
Policy initializer and data provider. LID-ADG further be used initialize the weights for fine- tuning RL policies and to gather data for offline learning. As shown in Table Z e.g. PPO, fails to solve tasks in VirtualHome. However, after using the policy trained by LID-ADG to initialize the PPO policy, we may effectively learn an interactive policy with good performance. In T PPO (LID-ADG Init) is initialized from LID-ADG and further fine-tuned to solve the tasks in VirtualHome. After initialization, PPO improves its success rate by 53.7% on the In-Distribution setting (See PPO results in Table 2 and Table 3.. In addition, LID-ADG can provide data for offline learning. LID-ADG saves the relabeled data in replay buffers. We train Decision Transformer (DT) using the data collected by LID-ADG. See DT (LID-ADG Data) Table
directly training RL,
7 Analysis: Understanding the Sources of Generalization
The pre-trained LM policy, fine-tuned on either expert data or actively gathered data, exhibits effective combinatorial generalization. Is this simply because LMs are effective models of relations between natural language descriptions of states and actions (. or because they provide a more general framework for combinatorial generalization in decision- maaiing? We hypothesize and investigate three possible factors to understand the sources of such combinatorial generalization. We use policies trained on the expert data as an example to explain the experiments.
7.1 Input Encoding Scheme
We first hypothesize that converting environment inputs into natural language contributes to the combinatorial generalization as the LMs are trained on language data. We explore the role of natural language by investigating three alternative ways of encoding policy inputs to our model without using natural language strings: two in VirtualHome, and one in BabyAl. BabyAl results are in Appendix A
Index encoding in VirtualHome. Rather than natural language strings, LID-Index (Ours) converts policy inputs into integer indices. LID-Index (Ours) retains the discrete, serial format of the goal, history, and observation, but replaces each word with an integer, and replaces the embedding layer from the pre-trained LM with a new embedding layer trained from scratch. For example, grab apple mapped to (5,3) based on the positions of grab and apple in the vocabulary set.
Unnatural string encoding in VirtualHome. LID-Urmatural (Ours) replaces the natural language tokens (e.g. converting the goal "On "foorkr,, table) 1 to put one fork on the lable) with random ones (e.g. converting On (fork, table) to brought wise character trees fine yet). This is done by
NNote that the results of LID-Text in Figure because the difficulty level of the evaluated tasks different. See Appendix F for more details.
B and results of LID-ADG in Table 2 are not directly comparable
8

---

Table 4: Success rates of policies trained with different input encodings in the Novel Tasks setting on VirtualHome. The text encoding is the most sample-eflicient, but all models converge to similar performance given sufficient training data.
Methods
Number of Demos
100
500
1K
5K
10K
20K
LID-Text (Ours)
8.8 :: 1,4
22.2 +: 1,7
26.8 + 1.0
46.0 + 1.0
58.2
1.2
58.2 1.6
LID-Index (Ours)
6.4 + 0,6
18.0 + 3.8
18.8 + 1.0
45.5 2.1
54.6 +: 0.8
57.8
0.9
L.ID-Unnatural (Ours)
6.8 + 1.3
18.6 + 2.1
27.0 + 1.1
47.2 +: 1.7
5588 + 0.8
58.8 0.9
randomly permuting the entire vocabulary, mapping each token to new token. Such permutation breaks the semantic information in natural strings.
LID-Index (Ours) and LID-Unnature (Ours) have the same policy network as LID-Text (Ours). All are fine-tuned on the expert data. The averaged results using different random seeds on the Novel Tasks setting are reported in Table Given few training data, e.g. 100 demos, all the models perform poorly, with success rates lower than 10%. LID-Text (Ours) achieves higher success rates than LID-Indez (Ours) and -DD-Unaaural (Ours) when dataset size increases, e.g. LID- (Ours) is around 4% higher than LID-Indea (Ours) and -UDnnuaala (Ours) with 500 training demos. When the training dataset is further enlarged, e.g. 20K demos, success rates of all approaches reach similar performance. This result indicates that the effectiveness of pre-trained LMs in compositional generalization is not unique to natural language strings, but can be leveraged from arbitrary encodings, although adapting the model to arbitrary encodings may require more training data.
7.2 Sequential Input Representation
Next, explore whether generalization re- quires the sequential processing mechanisms in transformer-based LMs. We investigate whether the LM pre-trained policy will still be effective when the input encoding not sequential. No-Seq en- codes the goal as a single vector by averaging all goal embeddings, History and observation features are obtained in the same way. All features are then sent to the pre-trained LM to predict actions. As shown in Table 5 removing sequential structure significantly hurts performance on Novel Tasks. No- Seq achieves good performance on test tasks that are closer to training tasks. but cannot generalize well to more challenging unseen tasks. Thus, combinatorial generalization in pre-trained LMs may be attributed in part to transformers' ability to process sequential input representations effectively.
Table 5: Experiments on sequential inputs and weight initialization. Fine-tuning the pre-trained weights and the usage of sequential encoding are important for combinatorial generalization.
In-Distribution Novel Tasks
LID-Text (Ours)
87.6 + 1.9
58.2 + 2.3
No-Seq No-Pretrain No-FT
74.0 2.3
2.0 + 0.6
90.8 + 2.0
47.0 2.8
51.2 + 4.5
17.0 + 2.9
7.3 Favorable Weight Initialization
Finally, we investigate the favorable weight initialization from LM pre- pre-training enables effective generalization of the proposed model. No-Pretrain does not initialize the policy using the pre- praitrined LM, but instead training the policy on the expert data from scratch. In Table []] we find that removing the pre-trained weights can fit the in-domain data and thus performs well on the In-Distribution setting. However, its success rate is 11.2% lower than the proposed model on the Novel Tasks setting, indicating the pre-trained weights are important for effective generalization, but not necessary for effective data fitting. We further test a baseline, No-FT, that keeps the pre-trained weights of the language model but freezes them while training the rest model our expert data. Freezing the pre- trained weights without fine-tuning significantly hurts the performance on both settings, suggesting that fine-tuning of the transformer weights is essential for effective combinatorial generalization.
Together, these results suggest that sequential input representations (vs. dimendiieesionnl feature vectors) and favorable weight initialization are both important for generalization, however, the input encoding schemes (e.g. as a natural language string vs. an arbitrary encoding scheme) has little influence. These results point to the potential broader applicability of pre-trained LMs as computational backbone for compositional embodied decision making, where arbitrary inputs, such as language, images, or grids, may be converted to sequential encodings.

---

Go Novel In-Detributton
(pancake, stove): I; Close (stovel; (stove)
Walk Didwwader
Open Dallwasber
Kitcken Cubiict
Walk
Upsn Kacchnn
Callect
Grah Cuthery Pack
Walk Frider
Frilas
Put Cutlers Fork.
imsi the Fridgo
Goal: Insäde (cutlery fork, fork, fridge):
Start
Right
Pueward
Right
Right
Pickup
Goal: go the grey
Goul: up purple right
Figure 4: Qualitative results of our model on VirtualHome and BabyAL. We only show a sub-trajectory in each example to save space. The interacted objects are labelled by green bounding boxes.
Grounding
insie
Walk Comner Trewer
Put Apple Kitchenlfiblle
Goal: Inside (salmon, fridge): 1; Inside (sundae,, fridge):
Goal: (fork, kitchen table):
Policy
Walk
Walk
Walk Bestraco
Calinet
Goal: Tnside (chleken, oven): I; Close (oven); TurnON (oven)
(pillow, bed ):
Figure 5: Failure cases. We show failure cases caused by the grounding and policy error. The interacted objects are labelled by green bounding boxes.
8 Qualitative Results
In Figure f we show examples of LID-Texi (Ours) completing tasks in VirtualHome and Baby AI. We show successful examples from VirtualHome the In- In-Distribution and Novel Tasks settings, and two successful examples from Al on solving the GoToLocal and Pickuploc tasks. We only show short trajectories extract a sub-trajectory for saving space.
we show some failure cases of the proposed method. We observed two main types of failure cases: grounding error and policy error. For failures caused by the grounding error, the agent interacts with a wrong object that is not related to the given goal, e.g. the agent puts cutlets instead of the salmon inside the fridge. For failures caused by the policy error, the agent cannot find the target objects or does not interact with them. The proposed method that converts policy inputs into sequential encodings and feeds them to the general L.M framework can accomplish decision-making tasks efficiently. however, there are still challenging tasks that the policy fails to accomplish. Larger LMs, e.g. GPTT- G may improve the success rate of those challenging tasks.
Failure case analysis. In Figure 5
Conclusion and Broader Impact
In this paper, we introduced L.DD, a general approach to sequential decision-making that converts goals, histories, and observations into sequences and processes them using policy initialized with pre-trained LM. We integrated an active data gathering procedure into the proposed method to enable policy learning without using expert data. Our analysis showed that input representation and favorable weight initialization both contribute to the generalization while the input encoding scheme has little influence. One drawback of the active data gathering is that relies on hand-designa designed rules for task relabeling. More generally, potential disadvantage of the proposed approach is that biases further study of LID-based models' bias
of the pre-trained LMs may influence its behavior, required before they may be deployed in sensitive downstream applications. Nevertheless, our results demonstrate that LID enables effective combinatorial generalization across different environments, and highlight the promise of LM pre-training for more general decision makiing problems.
10

---

References
[1] P. Ammanabrolu and M. O. Riedl. Playing text-adventure games with graph-based deep reinforcement learning. arXiv preprint arXiv:/8/2.01628, 2018.
[2] Andreas and D. Klein. Learning with latent language. In North American Association for Computational Linguistics, 2022.
[3] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin,
P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint TrXXiv/707.01425
2017.
[4] M. L. Athul Paul Jacob and J. Andreas. Multitasking inhibits semantic drift. In North American Association for Computational Linguistics, 2021.
[5] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et Language models are few-shot learners, arXiv preprint arXiv:2005. 14165, 2020.
[7] L. Chen, K. Lu, A. Rajeswaran, K. Lee, Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.011 013401 2021.
[8] Deerwester, T. Dumais, W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391-407, 1990.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXhv:1810.04805, 2018
[10] A. Desovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[11] S. T. Dumais. Latent semantic analysis. Annual review of information science and technology. 38(1):188 230, 2004.
[12] D. Ghosh, A. Gupta, A. Reddy, Fu, C. Devin, B. Eysenbach, and Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv: 1912.0609/ 2019.
[13] Hill, S. Mokra, N. Wong, and T. Harley. Human mnstruction-follooing with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09938, 2020.
[14] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 18): 1780, 1997.
[15] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
[16] D. Y.TT. Hui, M. Chevalier- Chevalier-Boisvert,, D. Bahdanau, and Bengio. Babyai 1.1, 2020.
[17] F. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 99121002 PMLR, 2022.
[18] N. S. Keskar, B. McCann, I. R. Varshney, C. Xiong, and R. Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: 1909.05858 2019.
[19] N. Kitaev, S. Cao, and D. Klein. Multilingual constituency parsing with self-attention and pre-training, arXiv preprint 11160 2018.

---

[20] R. E. Korf. Planning as search: A quantitative approach. Artificial intelligence, 33(1):65- 88, 1987.
[21] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (XXX:0/9000099// 2015.
[22] Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and vision-and-language and-language tasks. arXiv preprint arXiv:1908.02233 2019.
[23] K. Lu, A. Grover, Abbeel, and Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv: (tt.:::213300550 052477247 2021.
[24] A. Majumdar, A. Shrivastava, Lee, Anderson, D. Parikh, and D. Batra. Improving vision- and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pages 259-274. Springer, 2020.
[25] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 311 1-11,9, 2013.
[26] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[27] V. Mnih, K. Kavukcuoglu, Silver, Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
(28) Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[29] M. E. Peters, M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 05365, 2018.
[30] E. Platanios, A. Pauls, Roy, Y. Zhang, Kyte, A. Guo, Thomson, Krishnamurthy,
J. Wolfe, J. Andreas, et al. Value-agnostic conversational semantic parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3666-3681 2021.
[31] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494-851 2018.
[32] X. Puig, Shu, S. Li, Z. Wang, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09880,, 2020.
[33] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training. 2018.
[34] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[35] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[36] M. Reid, Y. Yamada, and S. S. Gu. Can wikipedia help offline reinforcement learninn arXiv preprint 2rXiv:2201.12/22 2022.
[37] Schulman, Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06333 2017.
12

---

[38] P. Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language. In Association for Computational Linguisties, 2022.
[39] B. Shen, F. Xia, C. Li, R. Martin-Martín, L. Fan, G. Wang, S. Buch, C. D'Arino,, Srivas- tava, L. P. Tchapmi, et al. igibson, simulation environment for interactive tasks in large realisticscenes. arXiv preprint arXiv:2012.02924, 2020.
[40] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[41] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5000 6033. IEEE, 2012.
[42] M. Tsimpoukelli, Menick, S. Cabi, Eslami, O. Vinyals, and F. Hill. Multimodal few-shot learning with frozen language models. arXiv preprint arXiv:2106.1388 2021.
[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. arXiv preprint arXiv:1706.03766 2017.
[44] J. Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, pages 37-42, Florence, Italy, July 2019. Association for Computational Linguistics.
[45] B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson. Rat-sql: Relation- aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04943 2019.
[46] Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint arXiv: 1IIvv:1910.03771, 2019.
[47] Z. Yang, Z. Dai, Y. Yang. J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xlnet: Gener- alized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buee Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[48] Z. Yang. N. Garcia, C. Chu, M. Otani, Y. Nakashima, and H. Takemura. Bert representations for video question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1556-1565, 2020.
[49] Zhu, Y. Xia, I. WU, D. He, T. Qin, W. Zhou, H. Li, and T.-Y. Liu. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002. 06823, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
-
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section g and Appendix C.2
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Section e
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "bbkEQg8WQhZ", "reviewer": "Reviewer_5cMv", "summary": "This paper presents a framework to use language models like transformers to be used for decision making in interactive environments. The authors also present the features of this approach and present experiments to verify the same. The key contributions are identifying that language models for decision making in RL-type settings improves combinatorial generalization, an active data gathering approach to account for less than ideal pre-collected expert data and how sequentiality is very important in all these encoding techniques for planning in the environments discussed in the paper.", "questions": "Q1: In Table 2, VirtualHome seems to be a very tough environment for any policy that isn't LID-ADG. I'd like to see comparison against other methods which have significance performance on this: even though they may not be using ADG, it would be nice to see how much can ADG push a method against a better performing model which might have more expert data.\n\nQ2: Can you provide more intuition/experiments on how different layers of GPT-2 in terms of self-attention are lighting up when a task is being performed? This question is aimed to poke at the structure of the transformer and how different layers are being used for interactive decision making\n", "limitations": "Authors don't discuss the limitations enough of the model in the environment of VirtualHome and BabyAI. I'd like to see a discussion of failure modes of the model and how that traces back to the encoded embeddings.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper presents the hypotheses very well and also lays out a rigorous enough framework to test the hypotheses. It lays out a framework to use a general large language model to be used in place of existing non-transformer based policy networks. The authors touch on the importance (or lack thereof) of the encoding scheme, how to convert structured pieces of instructions into language and testing the framework for achieving the goal state. Finally the piece about active data gathering is very significant and can be expanded in general to a lot of other RL methods to compensate for lack of expert data in RL environments\n\nWeaknesses: Even though there are experiments in section 7 trying to give an intuitive feel for the reason this method is working, there can be more discussion from a model-structure perspective on how the internal structure of the transformer is creating these embeddings for policy networks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L3h79bdrBSG", "reviewer": "Reviewer_EQi5", "summary": "This paper introduces LID, a framework that encodes the goal, history, and observation as the tokens and fine-tunes a pre-trained GPT-2 on the next action prediction task. The authors show empirically that such a pre-training mechanism significantly improves performance on both VirtualHome and BabyAI tasks. The proposed method is especially useful when only a limited number of demos are available. Their LID-ADG is able to make meaningful predictions on VirtualHome with any expert data (unlike baselines which get about 0% accuracy.\n", "questions": "- Are all the tokens in brackets (such as [grab] and <apple>)  treated as individual special tokens and added to the tokenizer or they are just treated as normal takes?\n- I don't see the reason why the authors only use autoregressive language model. The next action prediction task doesn't seem to require an autoregressive model since it's not a sequence prediction task. I wonder how bidirectional Transformers such as BERT and RoBERTa perform.\n", "limitations": "Yes, the authors discuss them in Section 9.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The proposed method is novel, simple, and quite effective.\n- The authors conduct a deep analysis in Section 7 to provide deeper understanding of various design choices in LID.\n- The paper is easy to read.\n\n### Weaknesses\n- One concern is about the reproducibility of the paper. The authors do not provide the detailed architecture and hyperparameters of the method. Although there is a short description in Appendix D.2. I don't think it is sufficient for reproduction.\n- It would be better if the authors can provide standard deviations in Tables 1 and 4. With a small number of demos, I believe that the standard deviation would be large.\n\n### Summary\nOverall, the proposed method is sound and effective, so I recommend accepting it. However, the tasks studied in this paper are not in my research area, so I am not confident with my judgment.\n\n### Violating the formatting instruction\nNotably, the authors include the appendix at the end of the main paper submission instead of putting it in the supplementary material which violates the NeurIPS format. I wonder if it is acceptable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DdNtH_2SWce", "reviewer": "Reviewer_uuzZ", "summary": "This paper proposes to utilize the pre-trained language model (PLM) for solving interactive decision-making problem, named LID. Specifically, the policy network consists of a PLM and tailored task modules. The policy information (goal, history, information) will be described as the natural text and encoded by GPT-2 PLM. During the fine-tuning process, the PLM and task-specific modules will be jointly optimized (the authors also study the different training paradigms). To alleviate the scarcity problem of expert data, the authors also propose an active data gathering method to make the agent learn in a self-supervised way. The BabyAI platform and VirtualHome platform are used to evaluate the performance of the LID framework. Several empirical experiments and the corresponding analysis show the rationality and generalization ability of the proposed LID framework. \n\nThe contributions of this paper are two-folded: 1) proposing a PLM-based framework for interactive decision-making task, i.e., LID in this paper; 2) proposing a self-supervised data gathering method, which provides a stable RL training process. The authors also analyze the source of the generalization ability of LID framework, which can provide a clear direction for future research.\n", "questions": "1. Pre-trained language models limit the length of input text. For instance, the maximum input length of GPT-2 [1] is 1024 [2]. Will the length of encoded text in your experiments exceed this number? If not, can you figure out a solution for handling long-text input? Because the environment and policy description could be too long, we may not neglect this situation.\n2. In the exploration stage (Section 4.2.2), what is the exact sampling method for the goal and initial state?\n3. The BabyAI 1.1 baselines [3] report the mean ± std in their papers. However, the evaluation of the BabyAI in this paper (Table 1 and 4) did not use different random runs. I did not find any justification in Section 5.2. Could you please offer an explanation?  \n\n[Rebuttal Updates] I confirmed the authors' answers.\n\nReference:\n\n[1] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). [Language models are unsupervised multitask learners.](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf).\n\n[2] https://huggingface.co/gpt2/blob/main/config.json\n\n[3]  Hui, D. Y. T., Chevalier-Boisvert, M., Bahdanau, D., & Bengio, Y. (2020). [BabyAI 1.1.](https://arxiv.org/abs/2007.12770)", "limitations": "- Societal Impact: No potential negative societal impact. The authors provide a new perspective to aid policy learning with a pre-trained language model. \n- Limitation: 1) Building text descriptions for each task still requires human labor. We do not know what textual format is optimal for policy learning. It varies from task to task, model to model. On the other hand, as I stated in Question 1, the long-text input could restrict the scalability of this framework. 2) The proposed methods also need humans to design some templates/rules, as the authors mentioned in the conclusion part. \n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. ***Novelty***: It is not surprising that the \"pre-training LM then fine-tuning\" paradigm is widely used in natural language processing tasks. Although the authors also follow this paradigm, it is interesting to see that the sequence modeling ability of PLM can be used in policy learning.\n2. ***Significance***: The proposed framework is well supported by empirical studies and evaluated on two commonly used benchmarks. This new framework powered by PLM can be beneficial to the research community of policy learning.\n3. ***Quality***: This paper is technically sound, and the authors provide enough technical details in the Appendix including network architecture, encoding methods, etc.\n\n**Weaknesses**:\n1. ***Scalability***: The proposed encoding method is templated-based (Line 155-156). Although the input encoding scheme (Section 7.1) may be a trivial problem, the encoding scheme may still affect the performance. Searching for the optimal encoding scheme is an expensive process, which may bring a high cost of hand-crafted engineering.  Besides, the data gathering method also relies on hand-designed templates (Line 220).\n2. ***Presentation***: The related work of PLM is adequately cited. But the authors should also introduce the background of policy learning so that the significance of this work can be highlighted.\n3. ***Performance***: Compared to the work that uses traditional networks like DQN, the integration of PLM may affect the inference speed. \n4. ***Clarity***: Most parts of this paper are well written. However, there are some typos in the paper:\n    - Line 53: pretrained LMs -> pre-trained LMs\n    - Line 104: language -> language. (missing full stop mark)\n    - Some papers should be cited in a proper way: Line 108: [23], Line 109: [36], Line 285:[15], Line 287 [15]. For example, in Line 108, \"[23] show that\" needs to be rewritten as \"Frozen Pretrained Transformer (FPT) [23] show that\". \n\n[Rebuttal Updates] The authors provided the additional experiments for addressing my concern of scalability. The authors also revised the typos and added the related works.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept