Language Model Pre-training Improves Generalization in Policy Learning
Authors: Shuang Li
Xavier Puig
Yilun Du
Ekin Akyürek
Antonio Torralba
Jacob Andreas
Igor Mordatch
Venue: ICLR 2022 Submitted (2022)
Keywords: Large Language model
Imitation learning
Interactive tasks
policy learning

Abstract: Language model (LM) pre-training has proven useful for a wide variety of language processing tasks, including tasks that require nontrivial planning and reasoning capabilities. Can these capabilities be leveraged for more general machine learning problems? We investigate the effectiveness of LM pretraining to scaffold learning and generalization in autonomous decision-making. We use a pre-trained GPT-2 LM to initialize an interactive policy, which we fine-tune via imitation learning to perform interactive tasks in a simulated household environment featuring partial observability, large action spaces, and long time horizons. To leverage pre-training, we first encode observations, goals, and history information as templated English strings, and train the policy to predict the next action. We find that this form of pre-training enables generalization in policy learning: for test tasks involving novel goals or environment states, initializing policies with language models improves task completion rates by nearly 20%. Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effective initializer even for tasks with no language as input or output. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but natural goals and plans; these representations can aid learning and generalization even outside of language processing.

Under review as a conference paper at ICLR 2022
LANGUAGE MODEL PRE-TRAINING IMPROVES GENERALIZATION IN POLICY LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Language model (LM) pre-training has proven useful for a wide variety of lan- guage processing tasks, including tasks that require nontrivial planning and rea- soning capabilities. Can these capabilities be leveraged for more general machine learning problems? We investigate the effectiveness of LM pretraining to scaffold learning and generalization in autonomous pacision-mmking We use a prainedi GPT-2 LM to initialize an interactive policy, which we fine-tune via imitation learning to perform interactive tasks in a simulated household environment featur- ing partial observability, large action spaces, and long time horizons. To leverage pre-training, we first encode observations, goals, and history information as tem- plated English strings, and train the policy predict the next action. We find that this form of pre-training enables generalization in policy learning: for test tasks involving novel goals environment states, initializing policies with language models improves task completion rates by nearly 20%. Additional experiments explore the role of language-based encodings in these results; we find that it is possible to train a simple adapter layer that maps from observations and action histories to LM embeddings, and thus that language modeling provides an effec- tive initializer even for tasks with no language as input or output. Together. these results suggest that language modeling induces representations that are useful for modeling not just language, but natural goals and plans; these representations can aid learning and generalization even outside of language processing.
INTRODUCTION
In recent years, language models (LMs) trained on open-domain text corpora have come to play a central role in machine learning approaches to natural language processing tasks (Devlin et al., 2018). This includes tasks that are not purely linguistic, and additionally require nontrivial planning and reasoning capabilities: examples include as vision-language navigation (Majumdar et al., 2020; Fried et al., 2018; Suglia et al., 2021), instruction following (Zhang & Chai, 2021; Hill et al., 2020), and visual question answering (Tsimpoukelli et al., 2021). Indeed, some these tasks are so re- motely connected language modeling that it is natural to ask whether the capabilities that result from LM pre-training might extend to tasks that involve no language at all-and if so, how these ca- pabilities might be accessed in model trained only to process and generate natural language strings. In this paper, we study these questions through the lens of embodied decision- decision-making, investigat- ing the effectiveness of LM pretraining as scaffold for learning control policies for interactive tasks featuring partial observability, large action spaces, complex states, and complex dynamics. We describe a series of experiments in the VirtualHome environment (Puig et al., 2018; 2020) in which LMs are used to initialize policies, and show that LM pre-training substantially improves generalization across common-place tasks in household environments.
In Experiment 1 (Section 6), we encode the inputs to a policy-including observations, goals, and action histories-as templated English phrases (e.g. representing the goal on(fork, table) as There is a fork on the table.)
shown in Figure 1. A pretrained LM is then fined-tuned to produce representations of these phrases that can be used to predict subsequent actions. For i.i.d. training and evaluation tasks, we find that this approach completes tasks at rate comparable to the same transformer- policy trained from scratch. For generalization to disto--dist tasks, however, LM pretraining confers substantial benefits: it improves task completion rates by

---

Under review as a conference paper at ICLR 2022
Input
Serialized Input
Embedding layer
Tranoeee from scrach
Goal:
VirtualHomme
cacoding)
INipancake, fridge:::: ONguiee. lable;:
"Pui two puncakes inside fridge...
Prumrited tondoemen (ramred receding
Prumited tondorne trando anoaiing
Obj name: "upple"
Partial observation
Obj state: [0, I.....0) City coard: O0gz Payse
coallo]pancskej- i
History:
pancake inside the tridge.... ..."
Output action: <walio-juicel
-grato-juiiee....
Zero- cert-hhet Combiration
(a) Overview of the policy initialized from a pre pre-trained Innguage model
(b) Results
Figure I: (a) Overview of the proposed method. We enoode the inputs the policy -incldding observations, goals, and action histories - as templated English phrases. The phrases are sent an embedding layer and a pre-trained language model to predict the subsequent action. The embedding layer fo can either be a embed- ding layer of pre-trained language model (Experiment 1) be learned embedding layer (Experiment 2). (b) Results. We summarize the main results of Experiment 1 ("Transformer from scratch" and "Pretrained trans- former (natural encoding)"), Experiment 2A ("Pretrained transformer (random encoding)") and Experiment ("Pretrained transformer (learned encoding)").
nearly 20% for tasks involving novel initial environment states and goals (Figure 1 "Transformer from scratch" and "Pretrained transformer (natural encoding)")
Next, we conduct two experiments aimed at clarifying the role of this string-based encoding. We design Experiment 2A that uses random strings instead of natural language inputs and Experiment 2B that uses non string- based encodings to study different ways of building interfaces between input encodings and LMs. In Experiment 2A (Section 7), we replace the "natural" string encodings of Ex- periment with an arbitrary mapping between logical goals and tokens (e.g. scrializing on on (fork, table) as brought wise character trees fine order yer). This random encoding substantially (by roughly 12%) degrades performance on out-o--distii tasks, indicating that LM encoders are sensitive to the form of string encodings even when fine-tuned (Figure 1 "Pretrained transformer (natural encoding)" and "Pretrained transformer (random encoding)"). In Experiment 2B, we inves- tigate whether string-based encodings are necessary at all. We replace the model's word embedding layer with a randomly initialized embedding layer that maps from discretized environment obser- vations, goal, and history actions sequence of LM input vectors, and fine-tune this embedding layer jointly with the LM itself. This learned encoder performs almost the same as the encoding of Experiment 1, indicating that effective encodings for non-linguistic tasks can be learned from scratch (Figure "Pretrained transformer (natural encoding)" and "Pretrained transformer (learned encoding)").
These experiments offer two main conclusions. First, they show that language modeling improves generalization in policy learning: initializing policy with a neural LM (pre-trained a next- word prediction task with a large text corpus) substantially improves out-of out-of-distribution performance on (non-line moon-linguistic) tasks in an interactive environment. Second, they show that language- based environment encodings are not needed to benefit from LM pretraining: it is instead possible to learn an interface between observations, actions, and model-internal representations derived from text corpora. These results point the possible effectiveness of language modeling as a general- purpose pre-training scheme to promote structured generalization in broader machine learning applications.
2 RELATED WORK
In recent years, word and sentence representations from pre-trained LMs (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2018) have become ubiquitious in natural language processing, playing a key role in state-of-the-art models for tasks ranging from machine translation (Zhu et al., 2020) to task-oriented dialog (Platanios et al., 2021). Some of the most successful applications of pre- training lie at the boundary of natural language processing and other domains, as in instruction following (Hill et al., 2020) and language-guidee past work, our experiments in this paper aim to explain whether these successes result entirely from improved processing of text, or instead from domain-gentt -eeiral representational abilities. Below, briefly survey existing applications of pretraining that motivate the current study.
image retrieval (Lu et al., 2019). Building on this
Learning representations of language From nearly the earliest days of the field, natural language processing researchers have observed that representations of words derived from distributional statis-
2

---

Under review as a conference paper at ICLR 2022
ties in large text corpora serve as useful features for downstream tasks (Deerwester et al., 1990; Du- mais, 2004). The earliest versions of these representation learning schemes focused on isolated word forms (Mikolov et al., 2013; Pennington et al., 2014). However, recent years have seen number of techniques for training (masked or autoregressive) language models to produce contextualized word representations (which incorporate information neighhoring words in sentences and paragraphs) via a variety of masked-word prediction objectives (Devlin et al., 2018; Yang et al., 2019).
Applications In addition to producing useful representations, these language models can be fine- tuned to perform language processing tasks other than language modeling by casting those tasks as purd-prediction problems. Successful uses of representations from pretrained models include syn- tactic parsing (Kitaev et al., 2018) and language-to-code translation (Wang et al., 2019); successful adaptations of LM prediction heads include machine translation (Zhu et al., 2020), sentiment clas- sification (Brown et al., 2020) and style transfer (Keskar et al., 2019). Text-based games (Yao et al., 2021; Yuan et al., 2018; Ammanabrolu & Riedl, 2018; Côté al., 2018) inherently involve text as both the input and the output, Recent works (Yao et al., 2020) in text-based games use GPT-2 to solve the text-based games and get significant performance improvements. However, it is hard to describe 3D information using text and most of their experiments are in 2D environments, Included in these successes are a number of tasks that integrate language and other modalities, including visual question answering and image captioning (Yang et al., 2020). In models that condition on both text and image data, several previous approaches have found that image representations can be injected directly into language models' embedding layers (Tsimpoukelli et al., 2021) using a similar mechanism to the one we describe in Experiment 2B. One of our main contributions is to show that approach works even for tasks in which only non-linguistic information is relevant to model predictions.
What do LMs encode? The possibility that LMS might encode non-linguistic information useful for other downstream tasks is suggested by a number of recent "probing" studies aimed at under- standing their predictions and the structure of their internal representations. Pre-trained LMs can answer a non-trivial fraction of queries about both factual and common-sense knowledge (Roberts al., 2020). Their representations encode information about perceptual relations among concepts, including visual similarity among object classes (Ilharco et al., 2020) and the structure of color spaces (Abdou et al., 2021). Finally, they appear be capable of basic simulation, modeling changes in entity states and relations described by text (Li et al., 2021).
LM pretraining beyond language Two recent papers consider questions closely related to the ones investigated here: (Brown et al., 2020) show that the GPT-3 model capable of performing a limited set of arithmetic and string manipulation tasks; (Lu al., 2021) show that pretrained LMs require very little fine-tuning to match the performance of task-specific models on several image classification and numerical sequence processing tasks. In this paper, we focus on non-linguis- tic tasks where the inputs and outputs do not involve language. To the best of our knowledge, the current study is the first to demonstrate improved generalization in a non- linguiigci problem over a standard neural-network baseline using pre-trained language model.
LANGUAGE MODELING AND POLICY LEARNING
We begin with brief review of the ingredients of language modeling and policy learning tasks used in our experiments.
3.1 LANGUAGE MODELING
Our experiments in this paper focus on autoregressive, transformer-basedd language models (Vaswani et al., 2017). These models are trained to fit a distribution po(y) over a text sequence y by decomposing it into sequence of tokens y (VI, $2......... ..., . via the chain rule:
log (1y))
Pogso( I yi, ........ 1)-1) 1).
Fach conditional distribution po (o (V1, ......, 1) is parameterized by a transformer neural net- work fe(y1, U/. Ii-1). This network encodes each conditioned token y into a continuous em- bedding e = g(y)) which is then fed into the transformer architecture and encoded into a categorical distribution over token values of y:- Our experiments utilize a standard language model, GPT-2, that is trained on Webtext dataset (Radford et al., 2018) using Huggingface library (Wolf et al., 2019).
3

---

Under review as a conference paper at ICLR 2022
3.2 POMDPS AND POLICY LEARNING
Our experiments explore the application of LMs to general sequential decision-making tasks in partially observed environments. These tasks may be formalized as partially observable Markov decision processes (POMDPs). A POMDP is defined by set of states S, a of observations O, a set of actions A, and a transition model T( 88: at) that predicts the next state 81+1 based on the current state 9t and an action ag. Importantly, in a POMDP setting, the observation or only captures a portion of the underlying state St, and an optimal pecision-makingg must incorporate both the current observation and the previous history of observations and actions. For experiments in this paper, policies are parametric models T g.. h, 0z) that select actions given the goals g. history information h, and partial observations o, of the current state 8t.
strategy (a policy)
All our experiments use imitation learning (Santara et al., 2017; Ng et al., 2000; Peng et al., 2018), specifically behavior cloning (Pomerleau, 1991; 1989; Torabi et al., 2018), to train Te. We collect where each individual trajectory
a dataset of N expert training trajectories D = {di.... -.d.,, dg). consists of set of goal, observations, and actions, i.e. d, (01,a1.". .0r,9). where T is the length of an expert trajectory. We then train a policy Th (arlg, he, or) which maximizes the likelihood Py (a) of the expert actions a {a1.-.. .rr taken in a trajectory using supervised learning.
log (a)
Σ log (a Ig or),
(2)
where hy consists of all history in the environment up timestep 1.
3.3 LANGUAGE MODELS AS POLICY INITIALIZERS
Ultimately, both language modeling and POMDP decision-making involve a sequence predictions (over words or actions) given a sequence of previous observations describing the context for that prediction; it is thus straightfowwar to use LM encoders to initialize POMDP policies (Chen et al., 2021; Janner et al., 2021). Our experiments use pre-trained GPT-2 (Radford et al., 2019) to initialize an interactive policy LLM. which we fine-tune via imitation learning to perform interactive tasks. Different from the pure linguistic tasks that predict next words given the input sentence, we instead utilize the language models as policy by predicting the probability of the next action a,, i.e. ar LgM(g, he, or). We note that such a formulation is broadly applicable to a variety of interactive tasks.
4 VIRTUALHOME FOR INTERACTIVE BEHAVIOR IMITATION
We consider embodied environments with 3D realistic scenes as the platform to test policies ini- tialized from the pre-trained language model, as their goals and actions can be naturally repre- sented as language, such as "put two plates on the kitchen table" and "grab apple". We further consider VirtualHome (Puig et al., 2018; 2020) for our experiments we can easily get its sym- bolic representations that can be further serialized as English sentences. We note that the proposed approach is broadly applicable to other embodied environments with 3D realistic scenes, such as ALFRED (Shridhar et al., 2020) and iGibson (Shen et al., 2020).
VirtualHome is 3D interactive environment simulate household activities. Agents are repre- sented as humanoid avatars. To investigate the contribution of language model pre-training on in- teractive behavior, we create VirtualHome-Iiitaii with more challenging tasks and more diverse scenes and objects. Formally, the goal of an agent to finish some household activities which is defined by set of goal predicates. The agent takes an action based on the partial observation ot, goal g. and history he at each step. Next, we will describe the goals, observations, and actions provided in VirtualHome- Imitatioo Learning.
Learning, an improved version of VirtualHome
Goal Space. For each task, we define the goal as set of predicates and multiplicities. For example, (apple, fridge): : 2; inside (pancake, fridge) 1 means "put two apples and one pan- cake inside the fridge". In each task, the initial environment (including initial object locations). the goal predicates, and their orders and multiplicities are randomly sampled (see Appendix Section D). At training time, these environment configurations and goals reflect common-sense environment lay- outs (e.g. moving food from the refrigerator to the table). To further test the zero-shot generalization ability of different models, several test splits (discussed more in Section 5.1) feature other, less nat- ural goals e.g. putting milk inside the dishwasher. In total, there are 59 different types of predicates. Each predicate appear in a goal with a multiplicity between O and (see Appendix D).
4

---

Under review as a conference paper at ICLR 2022
tridae
character
Ohject Interaction
fridge- openittneaes
kdichen
sollt goblput/wsl
table.fi pullwak
table
Navigation
walt kincher//vttroom.
Environment
Partial observation
Symbollc representation
Action space at single step
Figure 2: VirtualHome is a 3D interactive environment simulating houschold activities. provides symbolic graph representation of the partial observation. The action space changes over time based on the observation.
Observation Space. VirtualHome provides both graph-based and visual observations shown in Figure 2; our experiments focused on graph-based observations. These represent each agent state as scene graph, with nodes representing objects and edges describing their spatial relationships. The scene graph only contains object nodes appearing in the current partial observation.
Action Space. Agents can navigate in the environment and interact with objects. To interact with an object, the agent must predict an action name and the index of the interested object, e-g open (5) opening the object with index (5). The agent can only interact with objects that are in the current observation or execute the navigation actions, such as walk (bathroom). For some actions, such as open, the agent must be close to the object. There are also strict preconditions for actions, e.g. the agent must grab an object before it can put the object on a target position. As a result of these constraints, the subset of actions available to the agent changes at every timestep.
Different from text-based games (Narasimhan et al., 2015; Yao et al., 2020) that involve text as both the input and the output, VirtualHome is a 3D realistic environment, involving structured, graph- based observations and discrete, factorial actions. Moreover, our Experiment 2B (Section 7), shows that it is possible to learn how to interface between these observations / actions and an LM without any explicit string- based representation, a result that has no analog in any past work on text- bext-based games. The tasks in VirtualHome are also challenging as they have lots of physical constraints and strict preconditions between actions. For example, the agent must grab the apple first before putting it on the kitchen table and must close to the kitchen table. Text adventure environments this sense are much easier as they have less physical constraints and less strict preconditions among ac- tions compared with 3D environments. The objects' 3D information is also important for decision making VirtualHome while text adventure envirouments do not have such 3D information. To train the models, we collect a set of expert trajectories in VirtualHome using regression planning (Korf, 1987). See Appendix Section C for more details.
EXPERIMENT SETUP
5.1 EVALUATION METRICS
We generate test sets that evaluate policies' ability generalize in four ways: (1) generalization to familiar goals in novel (but in-distribution) environments; (2) generalization to novel goals; (3) generalization to abnormal initial environment states (defined by abnormal initial object locations); and (4) simultancous generalization to both novel goals and abnormal initial states. We thus build four test subsets to evaluate model from the four aspects. (See Appendix D for more details.)
Normal. In the "Normal" testing subset, the types of predicates and their counts are randomly sampled based on the same distribution as the training data. There are 2 ~ 10 predicates in each task. The objects are initially placed in the environment according to common-sense layouts; (e.g. plates appear inside the kitchen cabinets rather than the bathtub). Note that even though goal predicates drawn from the same distribution the training data, the initial environments in the test set are different from the training set.
Abnormal Initialization. The objects are placed in random positions in the initial environment without common-sense constraints (e.g. apples may appear inside the dishwasher). There are 2 ~ 10 goal predicates in each task.
Zero-shot Combination.
The components of all goal predicates are never seen together during training (e.g. both plates and fridge appear in training goals, inplaae,, fridge) appears only in the test set. There are 2 ~ 8 goal predicates in each task.
Abnormal Initialization Zero-shot Combination. Objects are initialized in Abnormal Initial- ization above, and goals drawn from the same distribution as Zero-shot Combination above. There 2 ~ 8 goal predicates in each task.
5

---

Under review as a conference paper at ICLR 2022
Verb decoder: grab
Object decoder: apple
Context festure fo
Pooling
f2'
f%
Pre-trained Langnage Model
A/
Embhedding layer
Finherkdng layer fo
Embedding layer fa
Eridge
have
fridgs
apple
kilchen
cabinet
fridge
Serialization
Serialization
Serialization
ONiapple. table):1
INSIDEipanaake, fridge):: ...
<put> [pancake] (innsdee<iisiid//triieee
[kitcheen], jpuncake],
Fridge
Inside
kiichen
cabinet
O
inskle
apple
Goul predicules
History actions
Purtial observation
Figure 3: Pre-trained language models for interactive imitation learning. The objects in the current ob- servation, the goal predicates, and history actions first serialized templated English phrases. We extract the tokens and their features from the phrases and send them to pre-trained language model. The output of pre-trained language model are summarized into context feature vector by average pooling which is then used for verb and object prediction.
For each test subset, evaluate the success rate of different methods. A given episode is scored as successful the policy completes its entire goal within 70 actions, where 70 is the max steps of the collected trajectories. For each model variant described below, we report the results of 10 training using 10 different random seeds. On each test subset, we use 5 different random seeds and test 100 tasks under each seed. Thus there are 5000 examples used for evaluating each model.
5.2 MODELS
To study the benefits of pre-training on the production of embodied interactive behaviors, we com- pare LM-based fine-tuning to variety of baselines (see Appendix Section E for more details):
LM (ft) the main model: a pre-trained GPT-2 transformer language model fine-tuned a policy learning task (conditioned goals, observations, and histories) described in Section 3.3.
LM (scratch) uses the same model architecture and inputs, but randomly initialized.
LM (ft) and LM (scratch) w/o Hist are ablations that do not condition history information. MLP-N and MLP-1 take the goal, history actions, and the current observation as input and send them the multilayer perceptron neural network (MLP) predict actions. "MLP-1" has three more average- peorage-pooligg layers than "MLP-N" that average the features tokens in the goal, history actions, and the current observation, respectively, before sending them to the MLP layer.
LSTM uses a long short-term memory network (Hochreiter & Schmidhuher, 1997) to encode the history information. The hidden representation from the last time step together with the goal, and the current observation are used to predict the next action.
EXPERIMENT 1: LANGUAGE-BASER STATE AND ACTION ENCODINGS
Our first experiment aims to answer whether LMs can be used to initialize policies if state and action information presented to them format that looks, to the greatest extent possible, like a standard language modeling problem. To do so, encode the inputs the policy-includiig observations, goals, and action histories as templated English phrases (described in more detail below). The model architecture is shown in Figure 3. These phrases are passed directly the LM (using its pre-trained word embeddings) and used to obtain contextualized token representations. These token representations are averaged, and used as input to action and object classifiers.
6.1 INPUT ENCODING
Inspired by the performance of recent large-language models, we choose to convert the observa- tions, goal predicates, and history information, into text take advantage of the pre-trained lan- guage model. To do this, we serialize the goal predicates, history actions, and observations as text and send them to the language model leam the policy.
Goal. Each goal, consistsing of a sequence predicates and multiplicities (e.g. "INSID fridge):22 is translated into command in the format put two apples inside the fridge.

---

Under review as a conference paper at ICLR 2022
MLP-N
LSTM
LM wio
LM (Ouns)
MLP-1
(scratich) weo
LM (scratich)
50
40
Sheoss Roite
30
20-
10
0-
Normal
Abnormal
Zero-she shot-sho
Abnormal +Zero-sho
Initialization
Combination
Figure 4: Comparisons of the proposed method and baselines on different testing subsets. "MLP-N". "MLP-1", and "LSTTM" are baselines without using transformer (Vaswani et al., 2017). "LM (scratch) w/o Hist" and "LM (ft) w/o Hist" are based on the transformer architecture but do not use history in the input for decision making. "LM (scratch)" and "LM (ft) (Ours)" are based transformer and uses history in the input. The "scratch" means the transformer is trained from scratch our data while "ft" means the transformer pre-trained on language tasks and then fine-tuned on our data. Each experiment is performed times with different random seeds. The averaged results are reported.
Observation. To encode the agent's partial observation, all the baselines and the proposed method use the symbolic graph representation instead of RGB visual representation as our focus is not to tackle the embodied perception problem. We aim at investigating the effectiveness of LM pretrain- ing as a scaffold for learning control policies, and we thus utilize a common graph representation from previous works, e.g. (Puig et al., 2020) and (Liao et al., 2019). The graphs observations in VirtualHome consists of objects nodes in the current partial observation and the spatial relations among objects. Each object node has name, e.g. "oven", state description, e.g. "open, clean", and 3D world coordinates. Serialization of these scene graphs is the only part of the model that does not directly pass a string to the LM encoder. Instead, structured representations of each object are built up by combining name, attribute, and position embeddings; see Appendix A.1 for details. Solving the long-horizon tasks in VirtualHome is challenging because of the partial observability and strict preconditions among actions.
History information. Action histories are also converted into templated English sentences (e.g. "I have put the plate on the kitchen table and the apple inside the fridge") and passed directly to the language model.
6.2 OUTPUT DECODING
VirtualHome features a combinatorial action space involving verbs and objects. To accommodate this large action space, we factorize the action prediction, selecting verbs and objects separately.
Verb prediction. Given a vocabulary of verbs V. we need to select one verb from them based on the current observation 0, goal 9. and history h. We take the output of the pre-trained language model and summarize them into context feature fo by average pooling as shown in Figure 3. The context feature is then passed through fully connected layer and a softmax layer to predict the probability p(u.lg, h. o) of selecting verb i at single time step, where U, E V.
Object prediction. Because of partial observability, the observed objects change over time. To enable the agent to only interact with objects in the current observation, we use attention module to assign an attention score to each object and select the object with the highest attention score to interact with. We denote the feature of each object node after the pre-trained language model as fr. The attention score of each object node is represented the inner product between its feature and the context feature: s; = f fo. The attention scores (s , are then pass through a softmax layer to generate the probabilities p(o./g, gg, h, o) (where i E {1,-.., ,N) of the N objects in the current observation o.
To maximize the likelihood of expert actions, train the policy by jointly optimizing the verb and object predictions using the cross cross-entrops entropy loss:
Vv, log.oop;;: h, o))
Σ yy,, log(p h,
(3)
where yo, and Vo: are the binary labels of verb v; and object o,, respectively.
7

---

Under review as a conference paper at ICLR 2022
1M (sroarh
1M rit (Chnn) TM (f1) Random
LM (fO (Ouns)
cap Bist
rep Goal His Ots
60
Goal
Swccest rezee
10
Namual
Abromnal Inir Instulizztion Combiintioo +Zeno shoe
Abusenal
Zem=het
Nonual
Absummal
Zeroolia
Figure 5: Comparisons of pre-trained language models using natural and unnatural strings. "LM (scratch)" and "LM (ft)" are the pre-trained language model using natural strings as input while "LM (ft) (Random)" uses random strings input. In "LM (scratch)". the language model is trained from scratch on collected data while "LM (ft)" and "LM (ft) (random)" fine-tune the pre-trained language model on the collected data.
Figure 6: Comparisons pre-trained language codings and learned encodings. "LM (ft) (Ours)" uses the pre-trained language encodings. In "rop Goal", we use the learned encoding for goal and the pre-trained language encodings for history and observation. Simi- larly, "rep Hist" and "rep Obs" use the the learned en- coding for history and observation, respectively, "rep Goal-Hise Hasl-Hist-Obs" uses the learned encoding for goal, his- tory, and observation.
Intiplization Nerttin
6.3 RESULTS
The results of different models on four test subsets are shown in Figure 4. In the Normal setting, where the test tasks are drawn from the same distribution as training tasks, policies initialized with a pre-trained language model, "LM (ft)", match the success rate of policies trained from scratch, "L.M (scratch)"" A similar trend is observed the Abnormal Initialization setting. However, in the Zero-shot Combination and Abnormal Init+Zero-shot settings, where the test tasks requires generalization to novel goals or environment states that are never seen during training, we find that pre-trained policies, "LM (ft)", dramatically improve upon random initialization, "LM (scratch)"" and all other baselines. The fact that "LM (ft)" performs slightly better on "Zero-shot Combina- tion" and "Abnormal Init+Zero-shh attributable to the fact that these sets involve fewer subgoals on average per task. Baselines without pre-training do not generalize these subgoal and perform worse on the zero-shot setting.
(2 8 predicates) than on "Normal" (2 10 predicates) is
7 EXPERIMENT 2: NON-LINGUISTIC STATE AND ACTION ENCODINGS
We interested in different ways of building interfaces between inputs and LMs, including string- based serialization and learned embedding. Experiment 1 demonstrated that language- aaguage-bbseed based envi- ronment encodings contributed to effective generalization in LM-pretrained policies. Our two final experiments explore the sensitivity of this approach to the details of the input encoding.
7.1 EXPERIMENT 2A: NATURAL STRINGS V.S. UNNATURAL STRINGS
First, when using the pre-trained models' own string encoding mechanism, how important is it that strings passed as input resemble the training data? To evaluate this question, we replace the "natural language" tokens (e.g. serializing the goal "ON(fork, table) :1" put one fork the table) with random ones (e.g. serializing ON(frr,, table) as brought wise character trees fine yet). This is done by randomly permuting the entire vocabulary, mapping each token to a new index. The model architecture is the same as "LM (ft)" and the whole model is fine-tuned on our expert data, The results of permuting the vocabulary used to represent observation, goals, and history is in Figure 5.
In the "Normal" test condition, vocabulary scrambling has little effect: both natural and unnatural strings after fine-tuning able to fit the testing tasks that drawn from the same distribution as the training tasks. However, using unnatural strings significantly hurts the performance of the zero-shot setting where the testing tasks are unseen during training: even when not essential for in- distribution performance, natural string encodings are necessary to harness pretraining for stronger forms of generalization.
7.2 EXPERIMENT 2B: PRE-TRAINED LANGUAGE ENCODINGS V.S. LEARNED ENCODINGS
Given a non-linguistic task, if an effective string-based encoding cannot be generated arbitrarily, can such an encoding at least be learned? To answer this question, we retain the discrete, serial format of the goal, history, and observation representation, but replace the embedding layer from the pre-trained language model with a new embedding layer trained from scratch. We design four variants, some of which retain parameters from the original pre-trained LM: rep Goal uses new
8

---

Under review as a conference paper at ICLR 2022
Inneracted objects
predicutes: ONemik kitchembllee: INSIDEicopeclee stovey: I
Navigation
ONichicken, kiichenablee::
Navigazion and carrying abject
CLOSEEstee)) TimOON(oove)
Actions:
(walk) <kiichen cahinet 1>
[pu)] achickero dkachenshiie
[opea] <kincken cabunet 1>
(walk] <hedroanz
(walk) <kincten cabunet 2> 0
[walk) coalirats
(open) -dincten cabinet 2>
[open] <cabinet
[walk] <dicchen cabinet E>
Janb] saupeske>
<kinchen cabinet to
[valk] <kiechers
(grab) emille
<stove>
[walk] <kicctntahlees
[por] sespeakes> <atoneo
[pui] caillo 4acchemahie
[close] <alovc
0.0
(walls ethickers
[Iumon] ceont
[walk] ckiicentaaiics
Figure 7: Predicted trajectory and actions for a given household task. The policy learned by fine-tume teeettning the pre-trained language model successfully finishes the task described in the goal predicates. We highlight the key actions in the map, where the agent is finding, grabbing, or placing objects in the target positions.
embedding layer encode goals while using the pre-trained embedding layer encode observations and action histories. rep Hist and rep Obs use new learned embedding layer only for action histories and observations respectively. Finally rep Goal-Hist-Obs replaces the entire pre-trained language embedding layer, using a representation of the input to the policy learned entirely from scratch. Both our proposed model and the variants use serialized inputs. However, the variant models do not assume string-based inputs. The variant models use the embedding layer trained from scratch to represent each element in the serialized inputs, while our proposed model uses the language tokens generated by the tokenizer from the pre- treitrained language model and embeds them using the embed- ding layer from the pre-traine's language model. See Section A.2 for more details. The results of these four variants and "ML (ft)" are shown in Figure 6. We find that replacing the embedding layer from the pre-trained language model has only negligible impact the performance compared with "ML (ft)". We thus conclude that the effectiveness of pre-training is not limited to string-based representations, but can be accessed in input representations learned entirely from scratch. To the best of our knowledge, neither template-string sting---trng-b representations nor learned embeddings, com- bined with LM pretraining, has been applied to embodied decision-making problems in this way before: our main claim is that both approaches work better than learning from scratch in Virtual- Home given non-visual inputs.
QUALITATIVE RESULTS
In Figure 7, we show one example of "LM (ft)" completing the household orga- nization task in VirtualHome. Given a goal described by a set of goal predi- cates, (Otes,"N((mil)) 1k, kitchentable):l; ON(chicken, stove): : 1; CLOSE(stOVe))
kitchentable): 11; INSIDE(cupcall (stovee"" The agent first explores the environment until finds the milk. Then the agent grabs the milk, walks to the kitchen table, and puts the milk on the kitchen table. The agent moves to the next subtask until it finishes all the subtasks. Completing all the subtasks is challenging because of the partial observability, large action spaces, and long time horizons. Furthermore, there are strict preconditions between actions, e.g. to put the milk on the kitchen table. the agent must grab the milk first. The proposed method that serializes observations, goals, and history information as English strings and takes advantage of the pre-trained language model is able to finish the task efficiently.
CONCLUSION
We have presented set of experiments studying the effect of pre-training on embodied interactive behaviors in VirtualHome. To leverage LM pre-training, we described policy representation in which observations, goals, and history information were serialized as templated English strings, and found that language models fine-tuned as policies with these inputs enjoyed substantial benefits in zero-shot task and environment generalization over randomly initialized baselines. Additional ex- periments exploring the sensitivity pre-training to the encoding of the policy-learning task showed that poorly designed string encodings removed these generalization benefits, but that effective en- coding layers could be learned from scratch in the absence of a string- based goal and observation representation. These results suggest that language model pre-training produces representations not just of language, but of the abstract structure of goals and plans, and that these representations might be useful in a variety of tasks beyond language processing.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint arXiv:2109006122 2021.
Prithviraj Ammanabrolu and Mark O Riedl. Playing text-adventure games with graph-based deep reinforcement learning. arXiv preprint arXiv:/8/2.01633 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. 200.::2005.11165 2020. 3
Lili Chen, Kevin La, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345, 2021.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Textworld: learning environment for text-based games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-44) 407, 1990. 3
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18Xvv18810.04805, 2018. 1, 3
Susan T Dumais. Latent semantic analysis. Annual review of information science and technology, 38( 38(1):188-230 2004.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency. Taylor Berg- Kirg-Kiiikkkkkk,, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision- rision-and-langeeee lad-lang navigation. arXiv preprint /0Xx::8806.02244 2018.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instraction-followingg with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09338, 2020.
1,2
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997. 6, 23, 25
Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing text models for common ground with visual representations. arXiv e-prints, pp. arXiv-2005, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence mod- eling problem. arXiv preprint Stti::20060000339 2021. 4
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 2019.
Nikita Kitaev, Steven Cao, Dan Klein. Multilingual constituency parsing with self-attention and pre-training. arXiv preprint arXiv:1812.11766 2018. 3
Richard Korf. Planning as search: A quantitative approach. Artificial intelligence, 33(1):65-88, 1987. 5, 21
Belinda Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021. 3
10

---

Under review as a conference paper at ICLR 2022
Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler. Synthesizing environment-awwrt activities via activity sketches. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6291-6299, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for visionaand-langaag tasks. arXiv preprint 1:Xiv::908.002655 2019.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprini arXiv:2103.05247 2021. 3
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra.
Improving usion-and-language navigation with image-text pairs from the web. In European Conference on Computer Vision, pp. 274.. Springer, 2020.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 31 5111-3119, 2013.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv: 1506.0500.800 2015. 5
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000. 4
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv:1810.00%0
2018. 4
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv: 1802.05663. 2018. 2
Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, et al. Value-agnostic conversational semantic parsing. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Volume 1: Long Papers). pp.
2021. 2
Dean A Pomerleau. Alvinm: An autonomous land vehicle in neural network. Technical report, CARNEGIE-MELLOX UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSY- CHOLOGY .... 1989. 4
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1 ((1):88-97) 97, 1991. 4
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating houschold activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-He and-p A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890. 2020. 1, 4, 7, 21, 22
Alec Radford, Karthik Narasimhan, Tim Salimans. and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018. 2,3, 18
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019. 4, 18
11

---

Under review as a conference paper at ICLR 2022
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. 3
Anirban Santara, Abhishek Naik, Balaraman Ravindran, Dipankar Das, Dheevatsa Mudigere, Sasikanth Avancha, and Bharat Kaul. Rail: Risk-averse imitation learning. arXiv preprint arXiv:1707.06658, 2017. 4
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Shyamal Buch, Claudia D' Arpino, Sanjana Srivastava, Lyne Tchapmi, al. igibson, simulation envi- ronment for interactive tasks in large realisticscenes. arXiv preprint arXiv:2012.02924, 2020. 4, 21
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020. 4, 21
Alessandro Suglia, Qiaozi Gao, Jesse Thooaaon, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, Lanuuage-guided visual task completion. arXiv preprint arXiv:2108. 04927, 2021. 1
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint
vrXiv::805.0195. 2018. 4
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. arXiv preprint arXiv:2106. 13884, 2021. 1,3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint (tld::1706037666 2017. 3, 21
Bailin Wang. Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat- sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv: 191.0494, 2019. 3
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint 1910.379,, 2019. 3, 14,
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. Bert the IEEE/CVF Winter Conference
representations for video question answering. In Proceedings the on Applications of Computer Vision, pp. 1556-1565, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understand- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https::/prooceedings nwwrips..pp.. /ttpppps..cpppppe///////L dc6a7e655d7e5840066 www.eeeeeeeeeeeeece--eeeeee
pdf.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv:2010.02903, 2020. 3,5
Shunyu Yao, Karthik Narasimhan, and Matthew Hausknecht. Reading and acting while blindfolded: The need for semantics in text game agents. arXiv preprint arXiv:2 13552, 2021. 3
Xingdi Yuan, Alex-dle Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize in text- based games. arXiv preprint / /525, 2018.
Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monito arXiv preprint 10..122/06.03427, 2021.
12

---

Under review as a conference paper at ICLR 2022
Jinhua Zhu, Yingee Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Incorporating bert into neural machine translation, arXiv preprint arXiv:2002.06823, 2020. 2, 3
13

---

Under review as a conference paper at ICLR 2022
Appendix
In this appendix, we first provide more implementation details of the proposed model in Section A. We then show the additional results in Section B, including the analysis of using history information as input and the analysis of attention weights from language models. In Section C, we give more details of collecting the expert data. Section D lists the goal predicates in different test subsets. The implementation details of baselines are shown in Section E.
A MORE IMPLEMENTATION DETAILS OF THE PROPOSED MODEL
In Section 1, provide more details of the model architecture used in the main paper Section 6. We then introduce the training detail in Section A.3.
A.1 MODEL ARCHITECTURE DETAILS
In the main paper Section 6, we introduced the model architecture used for training an interactive policy. Our model consists of three parts, i.e, inputs, the pre-trained language model, and outputs. We take the goal g. history fıg and the current partial observation on as inputs and send them to the policy network initialized with the pre-trained language model. The output action a consists of verb and an object. For brevity, we will omit the time subscript t from now on.
In VirtualHome, the partial observation of the environment state can be represented as scene graph, with nodes representing objects and edges describing their spatial relationships. Each object node has name, e.g. "oven", a state description, e.g. "open, clean", and world coordinates. In the main paper Section 6, we briefly describe how to encode the observation. In this part, we provide more details of encoding the name, state, and position of each object in the current observation. Figure Al shows the model architecture we used to encode the observation.
Instead of flattening the graph observation as plain text, we keep the graph entities in observa- tions. This is because graphs representing agent observations are very large, involve a large number of distinct relations and LMs are not effective at numerical reasoning, such as 3D position infor- mation. For example, we have use many sentences describe the relations between objects, e.g. "the apple on the kitchen table, the apple is close the banana, the kitchen table is close to the fridge The pre-trained GPT2 model from Hugging Face (Wolf al., 2019) can feed in long
maximum of 1024 inputs. Describing every object and the relations between objects to be encoded with the LMs. In addition, the numerical features, such as 3D position information, maybe could be translated into naturalistic sentence- like format by saying "the apple is in position 0.01) and is 0.5 meters away from the banana ... ". but cannot be en- coded efficiently using language models as LMs are not effective at numerical reasoning. We thus utilize a graph encoder to remain as close as possible to natural language input without producing string-based graph representations that are too long to encode with the LM.
(xr 0.02, 0.07,
Name encoding. For each object node, we serialize its object name as an English phrase s For each word wp in the English phrase s", we extract its token t and features using the tokenizer and the embedding layer of the pre-trained language model, respectively. Since one object name might generate several English tokens using the tokenizer from the pre-trained language model, e,g. the tokens of "kitchencabinet" is [15813, 6607, 66, 316), we take the averaged features of all the tokens in the object name and obtain "name" feature J name for each object node as shown in Figure A1.
State encoding. Given the current observation o, objects have state description, e.g. "oven: open, clean", There are six types of object states in the whole training dataset, including. "clean", 6-dim vector to
"closed"' , "off". "on", "open", and "none". Thus for each object node, represent its state. Taking the "oven" as an example, if the oven is open and clean, its state vector would be [1, 0, 0, O, 1, 0). This state vector is then passed through fully-connected layer generate state feature fr of object o,.
14

---

Under review as a conference paper at ICLR 2022
Object feature fi
Concatenation + FC
Object name feature ft faname
Ohject state feature f poposition
Average pooling
Object state feature fi postate
FC + ReLU + FC
Embedding layer fe
Embedding layer fe
FC
Position vector:
1o Oly 012 0Ix ax
[15813, 6607, 66, 6014, 316]
State vector: [0,0,0.0,100]
Oly 0y. azl
Tokenizer
Object og: kitchencabinet
Object o kitchencabinet (open)
Object og: kitchencabinet
Object name encoding
Object state encoding
Object position encoding
Figure Al: Object encoding, In VirtualHome, partial observation o environment state can repre- sented as seene graph, with nodes representing objects and edges describing their spatial relationships. Each object node in the observation has a name, a state description, and world coordinates. Object name encoding: for each object node, serialize its object name extract its tokens and features using the tokenizer and the embedding layer of the pre-trained language model, respectively. We take the averaged features all the English tokens feature J pomsmme for each object node. Object state encoding: there are six types of object states in the whole training dataset, including, "clean", "closed", "off", "on", "open", and "none". Thus for each object node, we a 6-dim vector represent its state. This state vector then passed through
English phrase. For each word in the English phrase,
object name and obtain a "name"
cully-connected layer generate state feature f. object o.. Object position encoding: we concatenate the world coordinates (Or,x; Di,y, 0i.z) } each object and their spatial distance the agent {ag: ay a, to generate a position vector (oi,z, Ol,y. oi,a 0s,x layers with ReLU layer in the middle to generate a position feature f p-ppadiion of object og. The final feature fe of each object node is obtained by passing the concatenation of its name feature f. f. state feature f. and position feature 1 p.-poiition through fully-connected layer.
0s,y Ag.
az). This position vector is then passed through two fully- fulle-com
Position encoding, To encode the position information of each object o,, we take their world co- ordinates (oi,z; Oi.gr 0z,z and their spatial distance to the agent {azz ay. a,) to generate position vector [ot,=, Oi,y Oi,2, Oi,æ Oz 0i.y Og Oi,z az). This position vector is then passed through po.position
two fully-connected layers with a ReLU layer in the middle to generate position feature fo of object o,
The final feature f of each object node is obtained by passing the concatenation of its name feature f po state feature f yro.statee and position feature fo po.position through a fully-connect layer. The obser- vation at single step can be written as set of features {fp, , where N is the number of objects the current observation o.
A.2 MODEL DETAILS IN EXPERIMENT 2B
In the main paper Section 7.2, we introduced four model variants that replace the embedding layer from the pre-trained language model with a new embedding layer trained from seratch. this sec- tion, we provide more details of the proposed model and the four variants. We show the model architectures of the proposed model, "rep Goal", "rep Hist", "rep Obs", and "rep Goal-Hist- Hist-Obs" in Figure A2, Figure A3, Figure A4, Figure A5, and Figure A6, respectively.
To train the variant models in Experiment 2B, we assign an index ID to each "element" in the dataset. The "element" could be: 1) the object names the observation, e.g. "fridge", "kitchentable" and other objects if the agent is in the kitchen; 2) the object names in the goal predicates, e.g. "plate",
"kitchentable" and "fridge" in the goal predicate "ON(plate, kitchentable); INSIDE(plate, " or the actions, e.g. "walk" and "bathroom" in the action "[walk) bathroom >" We thus ob-
15

---

Under review as a conference paper at ICLR 2022
Verb decoder: grab
Object decoder: apple
Context feature
Pooling
Pre-trained Language Model
f
ff"
Pre-trained embodding layer
Pre-trained embedding layer
Pre-trained embedding layer
Tokens from the Pre-trained Tokenizer
Tokens from the Pre-truined Tokenizer
Tokens from the Pre-trained Tokenizer
put
two
apple
fridge
have
put
fridge
apple
kitchen
cabinet
fridge
Scrialization
Scrialization
Scrialization
INSIDErpancake, fridge)2; ...
<walk> [kitchen], <walk> [pancake).....
Inside
character
cabinet
apple
ON(apple, table):1
<put> [pancake] <inside=[[tidgee)
fridge
kitchen
inside
Goal predicates
History actions
Partial observation
Figure A2: Model architecture of the propose model the main paper Section 6.
tain a set "elements" " from the whole dataset, such as ("fridge". "kitchentable", "plate", "walk", "bathroom", "none"]. We then assign an index (starting from 0) to each of them, such as "fridge": 0, "kitchentable": 1, "plate": 2, "walk": 3, "bathroom" 4, ...,, "none": MAX ID J. Then we use one-hot vector to represent each "element" based on their index ID.. For example, "fridge" could be represented as a vector of [1, 0, ...].
To train the variant models in Experiment 2B, taking the "rep Goal" in Figure A. example, first this one-hot vector represent all the "elements" in the goal predicates. The one-hot fea-
vector then sent to learned embedding layer that trained from seratch to generate po(leamed) ture representation for each "element" in the goal predicates [fp(leamood) _ gg(leamed) ..... Jk To encode the partial observation and history actions, we keep the same encoding as the proposed model in Figure A2, where we first use the tokenizer from the pre-trained language model to gener- ate the language token for each "word" in the partial observation and history actions and then send the tokens to the embedding layers from the pre-trained language model to generate feature repre- IN (ppretrained) for the partial observation
sentation for each "word". Lee. [ (peetrani)) :/2
pofpetrained))
and [ hipretrained) p pl(peerriined)
S h for the history actions. The output features of goal predicates, partial observation, and history actions are then concatenated and sent the pre-trained language model for training.
Similarly, in the variant model "rep Hist", the history actions use the learned embeddings while the goal predicates and partial observation use the pre-trained embeddings. In "rep Obs" and "rep we use learned embeddings as described above to encode the "name" information of each object in the current observation. We use the same object state encoding and object position encoding described in Section A. to encode the "state" and "position" information of each object.
Note that the representations of "elements" and "words" are different. For example, "kitchentable" a single "clement" and has an index ID "I" the models with learned embeddings, but its English token extracted from the pre-trained GPT2 is "[15813, 2395, 429, 540]". Another example "plate" having an index ID of "2" in the models with learned embeddings but its English token extracted from the pre-trained GPT2 is "[6816]"
A.3 TRAINING DETAILS
Our proposed approach and baselines are trained on Tesla 32GB GPUs. We train every single model on Tesla 32GB GPU. Each experiment was trained on 10 different seeds and tested on 5 different seeds. All experiments used the Adam optimizer with the learning rate 1 1, We utilize standard
16

---

Under review as a conference paper at ICLR 2022
Verb decoder: grab
Object decoder: apple
Context feature fc
Pooling
fi
Pre-trained Language Model
1/20
1/
JP
Learned embedding layer
Pre-trained embedding layer
Pre- Pre-trained embedding layer
Index IDs - One-hot vector
Tokens from the Pre- Pre-truined Tokenizer
Tokens from the Pre- Pre-trained Tokenizer
put
two
apple
fridge
have
fridge
apple
kitchen
cabinet
Iridge
Scrialization
Scrialization
Scrialization
INSiDE(pancake, fridge):2;
<walk>> [kitchen], <walk> [pancake).....
inside
character
ON(pppee, table):
<put> |pancake [<idgeee[[iidge]
fridge
kitchen
cabinet - inside
apple
Goal predicates
History actions
Partial observation
Figure A3: Model architecture of "rep Goal" used in the main paper Section 7.2.
Verb decoder: grab
Object decoder: apple
Context feature
Pooling
A
fl
Pre-trained Language Model
fr
At
ff"
fi
Pre-trained embedking layer
Learned embedding layer
Pre-trained embedding layer
Tokens from the Pre-trained Tokenizer
Index IDs - One-hot vector
Tokens from Pre-trained Tokenizer
put
two
apple
fridge
have
put
fridgo
apple
kitchen
cabinet
fridge
Serialization
Scrialization
Scrialization
INSIDE(paneake, fridge);; ...:
<walk> [kitchen], <walk> [pancake)....
Inside
character
ON(apple, table):1
<put> [pancake] (inside=(tridge)
fridge
kitchen
cabinet
inside
apple
Goal predicates
History actions
Partial observation
Figure A4: Model architecture of "rep Hist" used in the main paper Section 7.2.
Verb decoder: grab
Object decoder: apple
Context Teature fa
Pooling
f9
Pre-trained Language Model
12
18
Pre-trained embeddinn layer
Pre-trained embedding layer
Learned embedding layer
Tokens from the Pre-trained Tokenizer
Tokens from the Pre-trained Tokenizer
IDs One-hot vector
put
fridge
have
put
fridge
apple
kitchen
cabinet
fridge
Serialization
Serialization
Serialization
INSIDEipancake, Iridge):;; ON(pple labley:1
<put>> [pancake] [ridge
[kitchen), <walk> Ipancakel....
fridge
inside
kirchen
cubinet
character
incide
apple
Goal predicates
History actions
Partial observation
Figure AS: Model architecture of "rep Obs" used in the main paper Section 7.2.
17

---

Under review as a conference paper at ICLR 2022
Verb decoder: grab
Object decoder: apple
Context feature
Pooling
Pre-trained Language Model
11"
K
f
ff'
fi
Leamed embedding layer
Learned embedding layer
Learned embedding layer
Index IDs > One-hot vector
Index IDs - One-hot vector
Index IDs One-hot vector
put
two
apple
fridge
have
put
fridge
apple
kitchen
cabinet
fridge
Scrialization
Scrialization
Scrialization
INSIDErpancake, fridge)2; ...:
<walk> [kitchen], <walk> [pancake).....
Inside
character
cabinet
apple
ON(apple, table):1
<put> [pancake] cinside=[[tridee
fridge
kitchen
inside
Goal predicates
History actions
Partial observation
Figure A6: Model architecture of "rep Goal-Hist-Obs' used in the main paper Section 7.2.
pre-trained language model, GPT-2 that is trained on Webtext dataset (Radford et al., 2018), in our experiments by using Huggingface library (Wolf et al., 2019).
B ADDITIONAL RESULTS
In the main paper Section 6, we show that the policy with pre-trained language model outperforms other baselines by large margin on the zero-shot setting. In this appendix Section B.I, we further discuss the effectiveness of using history information. In Section B.2, we show the attention weights from the attention layers in the language model after fine-tuned on our data.
B. SERIALIZE HISTORY INFORMATION AS SENTENCES
Given the partial observability of the tasks, the models require history information know what subtasks have been finished. To take advantage of the pre-trained language model, we serialize history actions as sentences and send them into the language model for action prediction. As shown in the main paper Figure 4, such an operation brings an improvement on all four test subsets, i.e. "LM (ft)" outperforms "L.M (ft) w/o Hist" for models with pre-training and "LM (scratch)" outperforms "L.M (scratch) w/o Hist" for models without pre-training.
In the Zero-shot Combination and Abnormal Init+Zero-shot settings, we find that the improve- ment brought by the history sentences is more obvious for models with pre-training, i.e. the differ- ence between "LM (ft)" and "LM (ft) w/o Hist" is much larger than the difference between "LM (scratch)" and "LM (scratch) w/o Hist".
In addition, we find that "LM (ft)" is slightly better than "LSTM" on Normal setting, but "LM (ft)" outperforms "LSTM" by more than 40% on the Zero-shot Combination setting. Even "LSTM" takes extra history information from the hidden representation of previous steps, its per- formance drops dramatically the zero-shot setting with unseen tasks.
Together, serializing history information as sentences is helpful for embodied interactive behaviors. Such operation enables utilize the pre-trained language model and outperforms the LSTM model significantly. The advantage of using pre-training is more obvious on the Zero-shot Combi- nation setting.
B.2 ATTENTION WEIGHTS
In the proposed model "LM (ft)", utilize the pre-trained GPT-2 (Radford al., 2019) to model the input sentences. To better understand how does the policy with a pre-trained language model make decisions, extract the attention weight from self-attention layers in GPT-2 (GPT-2 is
18

---

Under review as a conference paper at ICLR 2022
Goal:
Goal:
INSIDE (cutleryfork, dishwasher): (wineglass, sink): 2 INSIDE (waterglass, dishwasher): (dishwasher)
INSIDE (pancake, microwave) ON (chicken, kitchentable): ON (milk, kitchentable) CLOSE (mierowave)
TURNON (dishwasher)
(microwave)
Odirchencchinet
Odkttchencahinet
Oxplate
Orplate
Odkiccitccabinet
Okkttchnncabinet
Orchicken
Oxchicken
Occutleryknife
Occutlerykmife
encountendrower
Odkishemooutertrrwee
Orchicken
Oschicken
Ocknifeblock
Ocknifeblock
Ookilcbawwwumleerzwwwr
D'kitchcncouuter
O.kitshoncounter
Ossink
Ossink
encountendrawer
Oxkishencnumerrrrweer
wineglass
Orwineglass
Ormigrowave
mmicrrwwaee
aucounerdrwwer
O
idhwashingliquid
Oddishwshhnggliuud
encouncrdrawwer
O.kitshonouunerdrswer
Okflcchenouumeedrrwwwr
Orri
Odlishwasher
Ordishwasher
Orplate
Orplate
Ostovee
O:stove
Ockitchentable
O-kitchentable
Ostovelfu
Ostoveeiaa
Oscreamyhuns
Ouuneamybuns
Ortvstand
Ortvstand
Occurleryfiek
Oveutleryfork
Dchooolstesyrrpp
Occhocolatasyrap
Occullryffock
Oveutleryfakk
Orchips
Oschips
Oscurleryfork
Ozeutleryfark
Orchips
Occhips
wincgllss
Orwineglass
O pudding
Ospudding
Giclose
Gcclose
Orpancaee
Orpancake
Octomato
Ostorato
Gowasher
Grwasher
kitchen
O-kitchen
Grum
Gitum
G: on
G:
Gielose
Gielose
G: dish
G: dish
microwave
G: microwave
Gowasher
Gowasher
Geturn
Geturn
G-put
Giput
G: on
G: on
G:
G:3
G: microwave
G: microwave
G:
G:
Geput
Geput
G:ler
Goler
G:
Gry
Gty
punc
G:
Grfork
Giake
Grake
inside
G: inside
inside
inside
G: the
G: the
G: dish
G: dish
microwave
G: microwave
Gowacher
Gowasher
Grput
Geput
Geput
Gxput
G:
G:
G:
G:2
G: chickkm
G: chicksm
wine
G: wine
G: an
G:
Goglass
Giglass
G: the
G: the
kat
G:
Giche
Goahe
G: sink
G: sink
Gant
Gant
Grput
Giput
Gaable
Gzable
G:
G:
Grput
Grput
G: waller
G: waler
G:
G:
Goglass
Giglass
G: milk
G: milk
G inside
G: inside
on
G:
Figure A7: Attention weights layer named "Head 3 Layer 2", We show attention weights differ- ent tasks. We find that "Head Layer is able capture objects goal predicates. such "wineglass" and "cutleryfork" the left tigure, and "pancake" and "chicken" in the right figure (the tigures are cropped for visualization).
19

---

Under review as a conference paper at ICLR 2022
Goal:
Goal:
INSIDE (pancake, microwave): 1 ON (chicken, kitchentable): 2 ON (milk, kitchentable): CLOSE (microwave) TURNON (microwave)
INSIDE (pancake, microwave): ON (chicken, kitchentable): ON (milk, kitchentable): CLOSE (microwave) TURNON (microwave)
Action:
Action:
[grab] <milk>
[grab] J <milk
Osdoor
O:doar
O.dnor
Ordoor
Oschicken
Orchicken
Orchicken
Oschicken
Ossink
Ossink
Ossink
Ossink
Dikitchencounter
Orkitchencouuter
Odkichhncabinet
Oskitchencabinet
Odkitchencabinet
Orkitchennaabint
encounterdrawer
Onkicchencouttedrrwwwe
encounterdrawe
Onktcchencoummeddraeer
Osnicrowave
Ormicrowave
Ozmicrowave
Ormicrowave
Otkicchnnabinet
Odkinchensabbiet
Ockicchencabinet
O:kirchencabinet
encomnterdrower
O Oknichensoounteedrrwwer
encounterdrawer
O: Oikchhencoumeerdrwwe
encomaterdrawer
encounterdrawee
O likichencouuneeddree
cncomuterdrower
O rereeteessweerrrrweeee
encounterdrawee
okkchheecoumeerrrrww
Orkitchentable
Ockiccheutbbe
O.kiteheutable
Orkirchentable
O:book
O:book
O.book
O:book
Okkirebencsbinet
Odkitehescabinet
Kiichhencabie
O Oiichhenncb
Ormilk
Ormilk
Ormilk
Ormilk
Ocbreadslice
Orbeeadslice
breadslice
Orbocadsslice
O Ospancake
Orpancake
O pancake
Orpancake
O:bathroom
Orbathroom
O.bethroom
Orbathbroou
Ockitchen
Oskiteben
O'kitchenn
Orkitchen
G:close
Giolose
G:close
Grelose
G: microwave
G:: microwave
G: microwave
G: microwave
Grtarn
Gitun
Cittun
Grtumn
G: microwave
microwave
G: microwave
microwave
Gxpul
Grpul
Grput
Grput
G:
G:
G:
pane
puzio
pane
pane
Grake
Grake
Grake
G:ake
G: insode
G: inside
G: inside
G: inside
G: the
the
G: the
G: the
microwave
G:put
G:pul
G:put
Grput
G:
G:
G: I
G:
G: chicken
chicken
G: chicken
G: chicken
G:
om
on
G: the
G: the
G: the
G: the
G: kit
G: kit
G: kit
G: kit
Grche
Giche
Giche
Giche
Grable
Giable
Grable
Gsable
Grput
Giput
Giput
Grput
G:
G:
milk
milk
G: on
G: on
G: on
G: on
G: the
G: the
G: the
G: the
Figure A8: Attention weights of layers named "Head Layer (left) and "Head Layer 11" (right), Given the goal predicates. history, and the current observation. "LM (ft)" predicts the next action, ie. "grab milk". We find that "Head Layer 2" is able to capture objects in the goal predicates, such as "milk", "pancake", and "chicken" while "Head 4 Layer 11" focuses on the interacted object in the predicted action, such as "milk".
20

---

Under review as a conference paper at ICLR 2022
Transformer model (Vaswani et al., 2017)). In Figure A7 and Figure A8, show the attention weights from the input (left) to the output (right) of GPT-2. The order of tokens in the input and ouput is observation, goal, and history.
Figure A7 illustrates the attention weights layer named "Head 3 Layer 2". We show attention weights on two different tasks. We find that "Head 3 Layer 2" is able to capture objects in the goal predicates, such as "wineglass" and "cutleryfork" in the left figure, and "pancake" and "chicken" in the right figure (the figures are cropped for visualization).
Figure A8 illustrates the attention weights of layers named "Head Layer 2" (left) and "Head Layer 11" (right). Given the goal predicates, history, and the current observation, "L.M (ft)" predicts the next action, i.e. "grab milk". We find that "Head Layer 2" is able to capture objects in the goal predicates, such "milk", "pancake", and "chicken" while "Head Layer 11" focuses on the interacted object in the predicted action, such as "milk".
We find that the attention weights from different self-attention layers are significantly different, some self-attention layers assign high attention weight to objects in the goal predicates while some layers focus on the interacted object. There are also some layers that do not have interpretable meanings. The attention weights just provide us an intuition of how does the internal language model works, more quantified results are reported in the main paper Figure 4, Figure 5, and Figure 6.
C EXPERT DATA COLLECTION
In this section, we provide more details of expert data collection described in the main paper Sec- tion 4. To train the models, we first collect a set of expert trajectories using regression planning (RP) (Korf, 1987). We follow the implementation of the regression planner in (Puig et al., 2020), Given task described by goal predicates, the planner generates an action sequence to accomplish this task. As shown in Figure A9, the agent has belief about the environment, i.e. imagined distribution of object locations. As the agent explores the environment, its belief of the world becomes closer to the real world. At every step, the agent updates its belief based the latest observation (see (Puig et al., 2020)), finds a new plan using the regression planner, and executes the first action of the plan. If the subtask (described by the goal predicate) has been finished, the agent will select new unfinished subtask, otherwise, the agent will keep doing this subtask until finish it.
As in much of past work (Shridhar et al., 2020; Shen al., 2020; Puig et al., 2020), the planner used to generate training data has access to privileged information, such as full observation of the environment and information about the pre-conditions and effects of each action, while the actual deployed policy do not have such information. The planner allows an agent to robustly perform tasks in partially observable environments and generate expert trajectories for training and evaluation.
Eventually, we generate 80, 416 trajectories for training and 3, 758 trajectories for validation. Each trajectory has a goal, an action sequence, and the corresponding observations after executing each action.
D TEST SUBSETS
We introduced four subsets for interactive evaluation in the main paper Section 5, In Table A1, we provide a detailed description of each subset, including the count of goal predicate types and the number of goal predicates in each task. The Normal setting has 37 goal predicates in total and each task has 2 ~ 10 goal predicates. The tasks are drawn from the same distribution as the training tasks. The Abnormal Initialization setting also has 37 goal predicates and each task has 2 10 goal predicates. The objects randomly placed in the initial environment. The Zero- shot Combination setting has 22 goal predicates in total and each task has 2 ~ 8 goal predicates. The tasks are never seen during training. The Abnormal Init+Zero-shot setting also has 22 goal predicates and each task has 2 ~ goal predicates. The objects are randomly placed in the initial environment and the tasks are never seen during training.
In Figure A2 and Figure A3, we list the full goal predicates used in each subset. The goal predicates of task are randomly sampled from the corresponding predicates pool.
21

---

Under review as a conference paper at ICLR 2022
Goal: INSIDE(plate, dishwasher): ON(plate, table): 1
Selected an unfinished goal predicate: INSIDE(plate, dishwasher):
Partial observation
Belief graph
Regression planner
Pat. glass
Execute first action
1. walk. walk.plate platte
walk, dishwasber
Figure A9: Regression planner. Given task described by goal predicates, the planner generates an action sequence to accomplish this task. The agent has belief about the environment, Le. imagined distribution of object locations. As the agent explores the environment,
world becomes closer the real world. At every step, the agent updates its belicf based on the latest observation (see (Puig et al., 2020)), finds a new plan using the regression planner, and executes the first action of the plan. If the subtask (described by the goal predicate) has been finished, the agent will select new unfinished subtask, otherwise, the agent will keep doing this subtask until finish
belief
Table AI: Summary of four test subsets. For each test subset, we show count goal predicate types and the number of goal predicates in each task.
Pred. Types #Pred. Per Task
Compared training
Normal test
Tasks from same distribation as training tasks.
Abnormal Initialization
37
The objects randomly placed in the initial environment.
Zero-shot Combination
22
2
Tasks during training.
Abnormal Init+Zero-shot
22
2 2
The combination of Abnormal initialization and Zero-shot combination.
E IMPLEMENTATION DETAILS OF BASELINES
We introduced six baselines in the main paper Section 5.2, including "MLP-N", "MLP-1", "LSTM", "LM (scratch) w/o Hist", "LM (ft) w/o Hist", and "LM (scratch)". "LM (scratch)" uses the same architecture as the proposed method "LM (ft)" in Figure 3 in the main paper. "LM (scratch) w/o Hist" and "LM (ft) w/o Hist" also have a similar architecture as "LM (ft)" except that they do not take the history information input. Thus in this section, provide more details of "MLP-N", "MLP-1", and "LSTM".
MLP-N. The model architecture of "MLP-N" is shown Figure A10. The input and output are the same as "LM (ft)" as we introduced in the main paper Section 6 and this appendix Section A.I. The difference that instead of sending the concatenated features of the goal, history, and observation to the pre-trained language model, "MLP-N" sends them to an MLP followed by an average pooling layer to generate the context feature fc. The MLP consists of two fully-connected layers with ReLU layer in the middle. The verb decoder and object decode are the same as "LM (ft)".
MLP-1. The model architecture of "MLP-1" shown Figure A11. The input and output are the same as "LM (ft)" as we introduced in the main paper Section 6 and this appendix Section 1. "MLP-1" obtains an averaged feature f, of the task goal by averaging the features of all the tokens in J Similarly, "MLP-1" obtains an averaged feature of the history fr. and an averaged feature of the observation fo by averaging the features of all the tokens in the history sentences {ff ... f and all the observed objects (ff.--. f& }. respectively. The averaged goal feature fg. the averaged history feature fn. and the averaged observation feature fo are concatenated and sent MI.P to generate the context feature fc- The MI. consists of two fully-connected
the goal sentences {
22

---

Under review as a conference paper at ICLR 2022
Table A2:
Goal predicates in the Normal and Ab- normal Initialization settings.
ONCcutleryfork, k
ON(plate, kitchentable) :0
ON(waterglass, (atchentaale)):001
ON(wineglass, kitchentable) ~
(cutlerroo,,, dishwasher):
Table A3: Goal predicates in the Zero-shot Com-
INSIDE (plate, dishwasher):0
bination and Abnormal Inttialization-Zerrsstoo Combination settings.
INSIDE INSIDE (waterglass, dishwasher)):0 :0 ~
INSIDE (wineglass, : ~
INSIDE (cutlerroor, sink):0
INSIDE(milK dishwasher):0l ~0
INSIDE ( sink):0~ ~
INSID((chicken dishwasher):
INSIDE (waterglass, sink):0 ~
INSIDE (cupcakk,, dishwasher) :00 ~
INSIDE (wineglass, sink):0~3
INSIDE (pancake, dishwasher) :0 ~
INSIDE (milk, fridge):0 -0
ON(mik,, sink):0~2 ~.
INSIDE (chicken, fridge):0+
ON(chicken, sink):0~
INSIDE (cupcake, fridge) ~3
(cuupcak,, sink):0m-
INSIDE INSIDE(pancake, fridge):0 -~3
ON (N(pacake, sink):0
INSIDE INSIDE (poundcake, fridge):0~
INSIDE (cutleryfork, fridge):0~>3
ON(milk, kitchentable)::0-33
INSIDE(plate fridge):0~ ~0
ON(chicken kitchentable):00
INSIDE INSIDE (waterglass, fridge):0
ON(cupcake, kitchentable):0->
INSIDE (wineglass, fridge):0~
(N(paccake, kittchetable)):0->
INSIDE INSIDE (cutleryfork, microwave):
(ppundcae,,,, kitchentable))0003
INSIDE (plate, microwave):0
INSIDE(chicken) microwave): ~
INSIDE (waterglass, microwave):
INSIDE (cupcake, microwave):0
INSIDE (wineglass, microwave):
INSIDE (pancake, microwave):0 ~
INSIDE (milk, microwave):0
INSIDE (poundcake, microwave): :0 ~
INSIDE INSIDE (cutleryfork, stove):0
(kei stove):0 ~
INSIDE (plate, stove) :0
INSIDE (cupcake, stove): INSIDE (pancake, stove):
INSIDE (waterglass, stove) :0
INSIDE (wineglass, stove):0
INSIDE (poundcake, stove):0
INSIDE (milk, stove)
CLOSE (stove) :0 ~
CLOSE (wwahherrr) :0
CLOSE (microwave)::
CLOSE (friiggr)
TurnON(stove) :0
TurnON swwasher):atteoe
shhwasher)) ::
TurnoN(mi
layers with a ReLU layer in the middle. The verb decoder takes the context feature fe as input and outputs verb prediction. Different from "LM (ft)". the object decoder takes the object features right after the "Embedding layer fo" box as input to predicate the interacted object.
LSTM. The model architecture of "LSTM" shown in Figure A12. The input and output the same "LM (ft)" as we introduced in the main paper Section 6 and this appendix Section A. "MLP-N" in Figure A10. The difference in the ways to compute the context feature fc. The feature fc after the pooling layer is first concatenated with the hidden feature f from the last step and then sent to LSTM (Hochreiter & Schmidhuber, 1997). LSTM outputs the context feature fc and hidden feature h2 that will be used for the next step. The verb decoder and object decode are the same "MLP-N".
"LSTM" has a similar architecture
23

---

Under review as a conference paper at ICLR 2022
Verb decoder: grab
Object decoder: apple
Context feature
Pooling
MLP
f!'
fi/
Embedding layer fo
Embedding layer fo
Embedding layer fo
put
two
apple
fridge
put
fridge
apple
kitchen
cabinet
fridge
Serialization
Serialization
Serialization
INSIDE(pancake, fridge): ....
<walks [kitchen], swalk> [pancuke)..... ....
Inside
ON(apple, table):1
<pot>> cinenkeeecisiieefiiiiii
fridge
kitchen
cabinet
inside
apple
Goal predicates
History actions
Partial observation
Figure A10: Baseline MLP-N. The input and output are the same as "LM (fi"
introduced in the main
paper Section and this appendix Section A.I. The difference that instead of sending the concatenated features of the goal, history, and observation the pre-trained language model, "MLP-N" sends them an MLP followed by an average pooling layer to generate the context feature fe. The MLP consists of two fully- connected layers with ReLU layer in the middle. The verb decoder and object decode are the same as "LM (ft)".
Verb decoder: grab
Object decoder: apple
Context feature fe
Concatenation MLP
Pooling
Pooling
Pooling
1
1xt
12
ff
Embeedding layer
Embeddiing layer fo
Embedding layer fo
Iwo
apple
Fridge
have
fridge
apple
kitchen cubinet
fridge
Serialization
Serialization
Serialization
INSIDEqpancake, fridgg
<walk>> [kiichen], <wall- [pancake),
charetter
apple
ONiapple, table):1
<put> [pancake
fridge
inside
Goal predicates
History actions
Partial observation
Figure A11: Baseline MIP-1. The input and output are the same as "LM (ft)" as we introduced in the main paper Section 6 and this appendix Section A.I. "MLP-1" obtains an averaged feature fn of the task goal by averaging the features of all the tokens the goal sentences ........J f ). Similarly, "MLP-1" obtains averaged feature of history fn. and an averaged feature of the observation fo by averaging all the tokens in the history sentences {ST, ..... .L} and all the observed objects {ff ... . J respectively. The averaged goal feature fy. the averaged history feature fú. and the averaged observation feature fo are concatenated and sent MLP to generate the context feature fe. The MLP consists of two fully- colyecconeettd layers with ReLU layer the middle. The verb decoder takes the context feature fo as input outputs a verb prediction. Different from "LM (ft)", the object decoder takes the object features right after the "Emhedding layer fo" hox as input to predicate the interacted object.
features of
24

---

Under review as a conference paper at ICLR 2022
Hidden feature h,
Verb decoder: grab
Object decoder: apple
teature fc
Concatenation LSTM
Pooling
ML.P
1P
19
Embedding layer
Emberdding layer
Embeedding layer
TWO
apple
frigge
have
put
tragge
apple
kinchen
cabinet
fiidgo
Serialization
Serialization
Serializaiinn
Ilidden feature he-1
INSIDEipancake, fridge):: :..: ONopple, lablej:
ewallo [kiichen], swalks (puncakel..... <put> canncckooiiniideesffiii
insie
cabinet
chameter
apple
step
frige
kitchen
Goal predicates
History actions
Partial observation
Figure A12: Baseline LSTM. The input and output are the same "LM (ft)" we introduced in the main paper Section and this appendix Section A. "L.STM" has similar architecture "MI.P-N" in Figure A10.
The difference is in the ways compute the context feature fe. The feature fo after the pooling layer is first concatenated with the hidden feature h- from the last step and then ber, 1997), LSTM outputs the context feature fo and hidden feature lt will be used next step. The verb decoder and object decode are the same as "MLP-N".
LSTM (Hochreiter & Schmidhu-
25


Reviews:
[{"review_id": "ReGUNkVijb2", "reviewer": "Reviewer_NUGX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors empirically demonstrated the effectiveness of initialization with the pre-trained language model, but this result cannot give new insight that is different from existing knowledge about the effect of the pre-trained language model. Moreover, some results seem to leave concerns and questions. The detailed concerns and questions are as follows:\n\n1. The authors assume string-based inputs (goal predicates, history actions, observation) through serialization, but I cannot agree with why it is necessary to consider this setting. Moreover, the comparison with simple baselines in Figure 4 (experiment 1) seems natural result based on existing knowledge that is not different from existing knowledge about the effectiveness of pre-trained language model. \n\n2. For this work to be meaningful, it is thought that it should show better performance than the result performed without a pre-trained language model with original inputs that have not been serialized. However, Figure 6 (experiment 2.B) shows a negligible impact on the performance compared with learned encoding with original inputs. Based on these results, it does not sufficiently support the claim that using a pre-trained language model is helpful for generalization.", "summary_of_the_paper": "This paper investigates the effectiveness of the language model for training the policy in embodied environments. The authors use a pre-trained GPT-2 to initialize the policy, then show the generalization effect in policy learning. In the experiments, the authors demonstrated the language model shows a better generalization effect with simple baseline and ablation studies.", "summary_of_the_review": "This paper provides various experiments to show the generalization effect in policy learning with the pre-trained language model. However, most results cannot clearly support the main claim of the paper and it remains some concerns and questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K_u5CFiVB6d", "reviewer": "Reviewer_9PU7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the empirical results in Fig 4 do show benefits of initializing the model components from a pre-trained language model, the following are quite confusing/unclear from reading the paper:\n\n(1) The main motivation of the paper is to explore the effect of LM pretraining on a non-linguistic task. However, the task chosen in the work is inherently linguistic -- so much so that the visual aspects of the task (i.e., observations from the environment) are represented as English tokens. Whether the tokens are structured in a formulaic representation (e.g., inside(fridge, apple)) or in traditional English sentences seems unrelated to classifying a task as linguistic v/s non-linguistic.\n\n(2) It is unclear why the goal and history are encoded as English sentences before feeding as input to the embedding layer while the observations are input as graph entities (similar to original formulaic representation)? Is there any difference between encoding observations as graph entities v/s as English sentences? Indeed from Experiment 7.2, it is clear that there is almost negligible difference in representing goal/history/observations as English sentences vs original formulaic entities. \n\n(3) From Fig 5, it can be observed that `LM(scratch)` is worse than `LM(ft) Random`. Given that `LM(ft) Random` is trying to learn the task using random strings, it is concerning that `LM(scratch)` is unable to outperform even that. Does this mean `LM(scratch)` is not properly tuned and/or trained till convergence?\n\nOverall, this work presents limited empirical studies and has the above mentioned weaknesses.", "summary_of_the_paper": "The paper studies the effect of using pre-trained components in a neural policy network for an embodied agent in a 3D environment called VirtualHome (VH). The empirical results present 3 main observations -- (1) a model with components initialized from a pre-trained language model generalized better in zero-shot settings than a model with all components randomly initialized, (2) a poorly designed (i.e., random) string encoding removed these generalization benefits, and (3) an effective encoding layer could be learned from scratch in the absence of a string-based goal and observation representation.", "summary_of_the_review": "This work presents limited empirical studies and has several critical weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ELZweulkI1V", "reviewer": "Reviewer_g58W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find the paper very well-written. It is largely empirical and the experiments are carefully setup and deliver insightful results. The claims in the introduction are supported by the experiments (but needs be further restricted to reflect the limited scope of the paper). I would like to ask whether the visual observations are used for training the models (it seems like they are not), and if not, why? In general, it'd be interesting to see whether LM-learned representation can complement other rich representations (e.g., visual representation). The symbolic, graph-based representation may be too hard for the model to learn from, and thus it is easier to observe improvement upon this representation. I suggest the authors revise the claims, emphasizing that they apply to a specific task/environment and a non-visual input representation. ", "summary_of_the_paper": "**After rebuttal**: I am keeping my score. But I will not fight against rejecting the paper. I think the results are promising but the scope of the experiments are limited and the claims need to be more precise. As pointed out in my discussion with the authors, there are also several important missing details that makes it hard to understand and appreciate the experimental settings. I like the idea of experiment 2B but the change from the string-based representation to the one-hot representation does not seem to be a significant change, as they both give the language model a sequence of word vectors. This may just show that the order of the words in the goal and history does not matter for the navigation decisions (which is kind of expected given the simple templates used to generate the string-based representation). The paper's claims would also be strengthened with reasonable explanations for the observed phenomena. The paper currently treats the experiment observations as \"conclusions\", which I think may be over-generalization given the limited scope of the experiments (one simulator, one type of current-state input representation).\n\n**Before rebuttal**:\nThe paper proposes and studies the effectiveness of using pre-trained LMs to solve a sequential decision-making problem where the observations, goals, and actions are not originally represented in language. The authors design three experiments:\n1. Convert the problem to language modeling and measure the effectiveness of pre-training the LM. \n2. Compare performance of using the language input representation versus a random-string input representation. \n3. Determine whether converting the inputs to texts is necessary. \n\nThe paper concludes that (1) language modeling improves generalization in policy learning, (2) language-based environment encodings are not needed to benefit from LM-pretraining, and (3) the results point the possible effectiveness of language modeling as a general-purpose pre-training scheme.\n\n", "summary_of_the_review": "The paper offers novel, interesting results that contribute new knowledge about pretrained LMs. I recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9271IOxbmTu", "reviewer": "Reviewer_KsBt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\nThe results of language pretraining is quite large and well supported by the experimental evidence\nThe randomized word experiment is quite interesting and I think quite clearly demonstrates that the language pre-training is injecting information about the world and when the words are replaced by nonsense words, that knowledge can no longer be effectively transfered\n\nNegatives:\nI think the specific claim of novelty in this paper is quite weak and compared to other relevant work (which should be cited but is not) this paper is not a good contribution to the field.\n\nSpecifically, the claim is that this paper is the \"First to demonstrate improved generalization in a non-linguistic problem over a standard neural-network baseline using a pre-trained language model\"\nEither the claim is mistaken or it is being drawn deliberately narrowly to exclude work which I think is clearly related \nSpecifically, I think excluding methods which have been done on text adventure games is either an accidental oversight, or this claim is just not an interesting claim of novelty. There has been quite a lot of work in this area, but I think the paper to look at that most clearly makes the claims of novelty less convincing is [1]. This paper uses a pre-trained GPT-2 model on text adventure games (specifically the  text games from the Jericho framework), also demonstrates how the pre-training improves performance, including on held-out unseen games. With [1] as context, I do not think this specific novelty claim is correct, or at best it is not an impressive claim of novelty.\n\nThis begs the question of what are the differences between this environment and text adventure environments? I would argue that in pretty much every way, this environment is less interesting. \nOne thing that might have been interesting is if this paper had used the 3D environment version of VirtualHome. Then I think there would be a good argument that this environment presents new challenges over text adventure games because there is an added perception problem on top of the planning / text understanding one. But since this paper exclusively used the graph-based observations and discrete symbolic actions, this is not true. I think actually that with this observation and action space it is less interesting than text world.\nFirst, in this version, everything is templated symbols. There is a very fixed set of actions, predicates and objects that always appear written in exactly the same templated way. Contrast with text world, the space of actions is huge (essentially only limted by the original game designers imagination), there are any number of objects or other entities to interact with. The games were designed to be played by humans and are thus in much more natural language and encorporate a potentially broader set of scenarios. In VirtualHome it's just a fit set of objects and actions and the goals are just to satisfy a fixed set of predicates. And many of them are just involving the placement of various objects into other objects in a home. The fact that a symbolic planner can be used to generate ground truth (whereas you cannot do this straightforwardly in text adventure games without providing a lot of hidden information) to me indicates a less natural, less interesting setting.\n\nIn general, and specifically compared to the generalization in [1], I think the generalization in this paper is pretty limited. In this paper the test-time generalization includes novel combination of known objects in known predicates, but for instance does not include novel objects and novel predicates. There is also no novelty in actions, or in the general environment (the most is putting objects in unusual places). Contrast with [1], where entire games are held out meaning that actions, goals and objects can all be different in testing environments, making this form of generalization much more interesting.\n\nPutting aside this specific claim of novelty, I think there are actually not a lot of new idea in this paper. The architecture is not particularly novel (just pooling the outputs of a transformer model), there isn't much novelty in training (it's just behavioral cloning) or methods, the environment is not new and as discussed is not as interesting as text games. The only real claim to novelty is that they used pretrained LMs on this specific kind of environment.\n\nThis is either a minor clarification or kind of a big problem. Is there an experiment where the raw observation/goal/actions from the environment are fed into the network COMPARED TO the transformer trained from scratch. I know there is the experiment for the first one, but not seing a comparison to a from-scratch. But in general, the confusing placement of experiments (why is the major experiment introduced and shown in the introduction?) I might have just missed this. If it's just placement, I would recommend cleaning up the experiments section. Otherwise, I think it's kind of important that the comparison is missing, because this means that the pre-training only works because the authors have essentially hand-crafted these templates to get GPT-2 to have useful pre-training.\n\nI also found the architecture used pretty strange. Usually when people use GPT models, they will use the string token output head and just finetune it to produce the correct output. So in Figure 3, you would have all of the same inputs, but then would not generate outputs at every timestep, you would just generated a token sequence out after the last input. This gives a couple advantages:\nThe output heads actually contain useful training since they're trained to generate text\nIt lets you do zero-shot and few-shot experiments because the model can generate arbitrary string outputs. There are quite impressive results in these settings using large language models, so it is odd that this was not done.\nWas there a reason why this was not done in this case? Or was this tried and found not to be as effective?\n\nMinor: \nThe choice of reaching the goal in 70 steps seems somewhat arbitrary to me. Is this the default in the original environment?\nALFRED should always be in caps\n\n[1] Keep CALM and Explore: Language Models for Action Generation in Text-based Games. Shunyu Yao, Rohan Rao, Matthew Hausknecht, Karthik Narasimhan. EMNLP 2020.", "summary_of_the_paper": "This paper takes a transformer-based language model, pre-trained on a large text corpus (In this case GPT-2) and uses it for the symbolic version of the VirtualHome environment. The observation, goals and action history of the agent are encoded as text strings in a few various ways and fed as input to the transformer and the output of the model is pooled to predict the agent action. The paper demonstrates that in cases where the test distribution differs in some way from the training distribution, that using pre-trained transformers greatly improves performance on the task", "summary_of_the_review": "The paper has some interesting results, but the paper lacks novelty. Pre-trained LMs have already been shown to be effective in similar text/symbolic RL environments (text adventure games) and the paper is not as impressive even as some earlier work (see [1]). It does not provide new insights/methods/architectures compared to prior literature and so I think would not be accepted.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the use of pretrained language models (LM) for training the policy in embodied environments. Specifically, a pretrained GPT-2 LM is used to initialize the policy. Environment observations, goals, and actions are encoded appropriately (e.g., converted into text strings) to apply the LM-based policy. The experiments study the generalization effect of initializing with pretrained LMs. Reviewers have found the paper made limited contributions. In particular, prior works on text adventure games have explored the use of pretrained LMs for playing games and studied the generalization effect, such as [1]. It's also suggested that the paper should revise the claims made in the experiments, given the limited experimental scope and results. 

[1] Keep CALM and Explore: Language Models for Action Generation in Text-based Games. Shunyu Yao, Rohan Rao, Matthew Hausknecht, Karthik Narasimhan. EMNLP 2020.
Decision: Reject