SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark
Authors: Victor Zhong
Austin W. Hanjie
Sida Wang
Karthik R Narasimhan
Luke Zettlemoyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: language grounding
reinforcement learning

Abstract: Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that re- quire interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan com- plexity. In addition, we propose the first shared model architecture for RL on these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-environment benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SILG enables the community to quickly identify new methodolo- gies for language grounding that generalize to a diverse set of environments and their associated challenges.

SILG: The Multi-enviromment Symbolic Interactive Language Grounding Benchmark
Victor Zhong! Austin W. Hanjie,, Sida I. Wang?, Karthik Narasimhan' and Luke Zettlemoyer
Department Computer Science, University of Washington Department of Computer Science, Princeton University 'Facebook Al Research
Abstract
Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection diverse grounded language learning environments under common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), well symbolic counterparts of visual worlds that quire interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan com- plexity. In addition, propose the first shared model architecture for RL these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-ent envirnmennt benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SII.G enables the community to quickly identify new methodolo- gies for language grounding that generalize to a diverse set of environments and their associated challenges.
Introduction
An ideal language-conditioned agent should interpret language in diverse environments with vary- ing observation space, action space. language, and plan complexity. However, existing language- grounding literature typically focuses on single environments, and proposes methodological con- tributions specific to those environments [35, 51]. In order to determine which contributions are crrromment-specifie and which apply across multiple environments, it critical develop universal models that can be easily evaluated in many different settings.
To facilitate this research, we present the multi-environment Symbolic Interactive Language Ground- ing Benchmark (SILG). We focus on symbolic environments with semantic symbols instead of raw visual observations for efficiency, interpretabilityy and emphasis on abstractions over perception. SILG consists of diverse environments including grid-worlds RTFM [58], Messenger [22], and NetHack [34], which require generalization new dynamics (i.e. how entities behave), entity refer- ences, and partially observed worlds. SILG also contains symbolic counterparts of visual grounding
Corresponding author Victor Zhong vzhonggcs. washington. edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Agent
World obs, text obs, relative pos, reward
Action
SILG conversion
RTFM
Messenger
NetHack
ALFWorld
Touchdown
Dynamics, goal, inventory
Enemy, goal, messenge
In-game message, player stats
Goal, in in-game message
Navigation Instruction
Figure I: Environments included SILG. The world observations and text fields are shown for each environment. Detailed examples are in Appendix F.
environments ALFRED [48] and Touchdown [9], which require interpreting rich natural language in complex scenes. For the former, we use its textual variant ALFWorld [49]. For the latter, we create SymTD by applying object segmentation to Touchdown panoramas. Despite significant implementa- tion differences, we unify these environments under a common interface in SILG, so that one can easily develop and evaluate language grounded RL methods across all of these challenges.
SILG environments present variety unique grounding challenges the richness of the observation space, action space, language specification, and plan complexity. We quantify these challenges and additionally analyze success rate and lengths expert playthroughs For visual grounding environments, we show symbolic variants (ALFWorld and SymTD) facilitate faster learning and result policies that transfer to their visual counterparts. While a unified model may not outperform specialized models engineered for specific environments, it can be helpful to understand whether particular modelling innovations are environment specific or more general techniques. Furthermore, while the challenges in each environment very different, want to encourage the development of unified architectures and approaches that can scale across many language grounding tasks.
In addition to SILG, we propose the Symbolic Interactive Reader (SIR), the first shared model archi- tecture for these environments. We combine SIR with several recent advances in language-conditiong RL, including FiLM² [58], egocentric local convolution
[27]. recurrent state-tracking [34], entity- centric attention [22]. and large pretrained LMs [28]. On most environments, SIR achieves compara- ble performance methods designed specifically for single environments. In addition, we find that many recent advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for enviromennt benchmark. Finally, the best models significantly underperform humans SILG (10-85% depending on environment), which suggests ample room for modelling improvements that generalize across environments.
In summary, we (1) combine five language-groundie environments under the same interface to evaluate language grounded RL methods across diverse grounding challenges. (2) present the first shared model architecture for these environments, and (3) analyze recent modelling contributions across these environments. We hope SILG enables the community quickly identify new models and learning algorithms that generalize diverse set of environments and their associated challenges. The code for SILG available at https: github....... wom/vzhong/sii
2
SILG Environments
SILG contains five language-grounding environments including both grid-worlds (RTFM, Messenger, SILGNNeHHack) and symbolic counterparts of 3D-visual worlds (ALFWorld, SymTD). While all in- volve agents situated in interactive worlds, each presents unique challenges in richness of observation space, action space, language specification, and plan complexity. Table quantifies their theoretical complexity along these dimensions as well as empirical complexity using expert playthroughs.
For each environment, expert plays as many episodes necessary to learn about the game. We then record the playthroughs compute the empirical win and trajectory length. More details in Appendix

---

Table 1: SILG statistics. "dynamics" are high level rules dictating behaviour of entities. "Ref hops" are number of intra-text references the agent must resolve to determine correct course of action. Messenger and SymTD text are human-writte instead of procedurally generated. Distinctive properties are bold.
RTFM
Messenger
SIL.GNeHacck
ALFWorld
SymTD
Action space
5 fixed
5 fixed
23 fixed
50+ choices
1-5 choices
6 x 6 grid
10 x 10 grid
21 x 79
102 nodes
29.6k complex panoramas
State space
5 entities
14 entities
partial obs
191 entities
Mean text len
31 words
30 words
9 words
100 words
90 words
Vocab size
262 words
595 words
~ 100 words
1237 words
4999 words
new instr
Generalization
new dynamics
new dynamics
new layouts
new instr
new layouts
Ref hops
6 hops
3 hops
1 hop
4 hops
~ hops
100% new instr
Human win %
100%
100%
78.1%
61.5%
100% +layouts
7.8 steps instr
Human # steps
6.0 steps
2.2 steps
34.4 steps
9.6 steps +layouts 33.6 steps
Env FPS
240
1627
439
7
779
multi-step reasoning
adversarial generalization
large
complex language
Key challenge
partial obs
action space
The goal of SILG to provide a simple-to-use benchmark that allows researchers to quickly evaluate methods across all of these environments as well as their respective challenges. We thus combine these environments under a unified interface built on top of OpenAl Gym [7]. In each environment instance, the agent observes text inputs well as world observations. For grid worlds such as RTFM, Messenger, and SlekGk,tHack, the agent receives a 2-D bird's-eyes --eee-view symbolic grid as observations. For visually inspired environments such ALFWorld and SymTD, the agent receives symbolic egocentric view of the present scene. Figure shows how SILG environments are rendered to players via the play utility. In the rest of this section, we describe each SILG environment in detail. Appendix shows how to use SILG in Python. Appendix G shows licensing for SILG environments.
Selection criteria We select interactive environments that span the challenges presented in Table 1, are easily converted to symbolic representations, and avoid the use of additional simulators (e.g. Mat- terport3D [1]). While visual perception clearly important for language grounding [19], we focus on the unique challenges of symbolic environments such as multi-hop reasoming and generalization rich sets of procedurally generated dynamics. We leave the challenge of developing a visually rich multi-environment grounding benchmark to future work. Due to the lack of gold trajectories in many of the selected environments, we do not support imitation learning (IL. in this version of SILG.
RTFM RTFM [58] is a grid-world environment where an agent interprets text to acquire the correct items to fight the correct monsters. A key challenge in RTFM is multi-modal multi-step reasoning (at least 6 steps) combining world observations with texts associated with multiple entities. Given a team to beat, the agent must identify which monster is on the team, then identify the item descriptor that would beat the monster descriptor. Finally, the agent must acquire the item with the correct descriptor and engage the correct monster to win. RTFM evaluation is on games with unseen rules, foreing agents to make novel reasoning steps to generalize successfully. At each step, the agent receives a symbolic grid containing names of entities present, as well as texts indicating the high level rules, the agent inventory, and the goal of the particular game instance. We include all 4 RTFM curriculum stages, but only show results for the first stage in this preliminary study.
Messenger Messenger [22], is grid environment where the agent must acquire a message and deliver it to the goal while avoiding an enemy after extracting entity-role assignments from a text manual. key challenge Messenger is the adversarial vain-evaluation split without prior entity-text grounding. There is no overlap in entity-role assignments between training and evaluation, forcing agents to make compositional entity-role generalizations. each step, the agent receives a symbolic grid containing symbol IDs of entities present, well as texts indicating roles of each entity. The
3

---

factng dtrection marked Cassse corresponding direction vent
Colour key:
masy sideuali, be sall,, fenre, sale, traffic Light, traftir sig,, segetation, terrate, thy, parson, riders bonoreselly Moyelle
20th7 TEAT
Turn and rase the sonn strection as traffic. When you get to intersection, turn left, After you turn the corner building you
Rewart: com revent:
Stepez:
Dave: Fatze
Your historicst scorest
Figure 2: The SILG play utility (shown here for SymTD) enables human playthrough as well as visualizing what input the model observes. Because all environments are symbolic, the playutility works in console (e.g. via ssh, tmux) without need for X-forwarding.
entities are referred to in text by many names, which have lexical overlap with their symbol ID. That is, the text "dog" in the text for example is the non-textual symbol 2 in the observation and the association between entities and references must be learned via interaction. We include all Messenger curriculum stages, but only show results for the first stage in this preliminary study.
SlL.GNeHHack NetHack is complex rogue-like game from the NetHack learning environ- ment [34]. In SILGGNethack, we combine 3 tasks (Score, Gold, and Scout) and specify the task to complete for each episode via text prompt. SII SIL.GNeHHack is challenging due to its large state space and partial observability. The agent may descend multiple floors and sections of each floor may be obscured until exploration by the agent. Because of the different score distributions of each task, we mark trajectory as successful if it exceeds a task-specific score threshold determined from human playthroughs. We evaluate agents on previously unseen map layouts that are procedurally generated with new seeds disjoint from the ones used during training. More information about the SILG multi- task SILGNetHack in Appendix D. At each step, the agent receives a symbolic grid containing symbol IDs of entities present, as well texts denoting the goal, agent stats, and feedback from the environment after the agent's last action. SII.GNeHHack vocabulary is technically infinite because players can arbitrarily name things, however in our expert playthroughs of SIIG SII Hee,Hk,tacck, we observe just over 100 unique words. Human experts win just under 80% of games with an average of 34 steps, which demonstrates the challenge of SILGNetHack. All failures can be attributed to hitting the step limit before acquiring the necessary win conditions.
ALFWORLD (text ALFRED) In ALFWorld, an agent navigates and manipulates objects inside 3D kitchen [49]. Its large text action space, with more than 50 valid actions (given by the game engine) for most scenes is a key challenge. Unlike its visual counterpart ALFRED [48] where the agent observes 3-D images of the kitchen, in ALFWorld the agent must rely on language descriptions kitchen. Goals are provided in human written language (e.g. put clean sponge on the metal rack). The language in ALFWORLD is not complex, but are 100 words on average due to a large number of items in single scene. Following recent work [49], we evaluate on both unseen instructions (new instr) and unseen room layouts (new layouts). At each step, the agent receives the goal text and a list of items present in the room (c.g. "cup 1", "bottle 2").. We concatenate the names of these items into a symbolic world observation grid each entry containing the name one item. The agent then selects from plausible commands given what is present in the scene.
the
TouchTown (SymTD, VisTD) In Touchdown, the agent navigates through Google Street View panoramas according to long compositional instructions that tests spatial reasoning [9, 37, 38). key challenge is the rich human-written navigation instructions that describe photorealistic images. Touchdown's 's long human- written instructions contain many intra-text reference hops, which we

---

RL Policy
RL baseline estimate
H
H
FILM' network
Concat
A
Jaint text summary conditioned field
Summary
D
Concat
Encode text
Attention and weighted averaging
N
Encode text
Encode text
Z
U
T
R2
Position emb
World obs emb
Joint text
Text field i-1
Text field
Figure 3: The Symbolic Interactive Reader (SIR) baseline. Inputs green, intermediate results white, outputs red, and model components yellow. Details about the FiLM² layer is in Appendix C.
approximate as the number of sentences plus the number of sequential connectors such as "then". We convert Touchdown a symbolic environment by segmentating its panoramas into semantic grids. In each step, the agent observes the instruction text and grid of discretized segmentation class IDs corresponding the current panorama. It then chooses among a list of radial directions to proceed to the next panorama. The agent wins if it passes the goal location. We use the same train-test split as the original Touchdown environment, which features unseen navigation texts.
We show that our symbolic Touchdown (SymTD) facilitates faster learning compared to learning in its visual equivalent (VisTD). Human performance demonstrates some limitations of SymTD, with an expert win rate just over 60%. This may be due to the symbolic representations removing information referenced by the instructions such as color, because the segmented features are visually disparate from real-world views [17]. We also include manual stop variants of SymTD and VisTD, which are functionally equivalent the original Touchdown. Appendix details these variants, SymTD/VisTD creation as well as discussions on human performance. Compared to prior work on Touchdown and ALFWorld, we train using RL. without supervised trajectories opposed imitation learning.
The Symbolic Interactive Reader Baseline Model
Figure 3 shows the SIR baseline for the SILG benchmark. To the best of our knowledge, this is the first shared model architecture for RTFM, Messenger, NetHack, ALFWorld, and Touchdown. Consider an agent situated in an arbitrary SILG environment. At each time step I, the model receives from the environment the following inputs (precise inputs for each environments are shown in Appendix F).
World observations X
hxx where h and w are the height and width of the observa- tion and each element corresponds to the k-word symbol ID(s) of its content.
Joint text T E IR of I tokens of the text to attend over.
Text fields A E gnxm where the ith row contains the ith of n environment text field such agent inventory or environment feedback. is the max token count of these texts.
Relative position Z E hxxx2 cell-wise feature that denotes the position of each cell relative to the player agent in the x and y directions.
As policy learner, the model must output distribution Y over the action space. We additionally output baseline estimate of the value function stabilize policy learning [18]. Let and denote embedding and bidirectional LSTM sizes. We first sum embeddings for each cell in the world observation obtain world representation U sum (emb (X)) Rxxwxd Next, encode the ith text field R, and the joint text T using an bidirectional LSTMs [30].
N; = BiLSTMN (emb (R))) E Rx
(1)
= BILSTMp (emb (T)) E g/xr
(2)
We then compute weighted average over text fields C and attention A, over the joint text.
5

---

C2 weightave, (N₂)
Σ
softmax (linear, (Ni)), Nos R
(3)
A = attend (D.C:)=
Σ softmax (Do:), Dj E ER"
(4)
We compress Ö E ga and A gaxe again to support any number of text fields.
C = weightave (+)E
EER' R' (5)
A = weightave, A A R"
(6)
We now have representations for world observations U, text fields C. and joint text conditioned on text fields A. We apply successive FiLM layers to build multiple levels of codependent representations between texts and world observations to model multiple cross-man moddll reasoning steps [58]. To support arbitrary number of text fields, we modify the text input of the ith FiLM layer to be the concatenation of the text fields C, attention over joint text conditioned on text fields A, and attention over joint text conditioned on the visual summary of the last FiLM² layer s(i-1).
FiLM ([v(e-); z], [C. A, attend (D,s(i-1)])
(7)
We use the definition of FiLM (visuals, texts) from Zhong et al. [58] and summarize its intuition and computation in Appendix C. We define (1) and 5(4) to be the initial world observation U and its spacial max- max-pooling. Finally, we use multi-layer perceptron to build a fixed-size codependent repre- sentation of the inputs based on the last FiLM² layer's output which is used compute the baseline estimate of the value function B ML.P z (H) and the policy (H) expressed as a distribution over actions. While the core architecture of SIR is identical for all environments, different policy module Y is necessary for different types of action spaces.
tanh (lincar, (flatten( (V(ss))))
Fixed sized action space (RTFM, Messenger, SIL.NetHacck) We simply apply multilayer perceptron to the final representation Y MLPY (H).
Multiple-choice text action space (ALFWorld) Let Q denote tokens for the jth choice (e.g. pick
up the mug), which encode bidrectional LSTM G, BiLSTMc (emb (Q,)). We then attend over this text using the final representation H to score for jth choice Y; linear4 (attend (Gj, H)).
Multiple-choice navigation action space (SILGTouchdown) Let j denote the index of the world representation corresponding to a movement direction. For example, for a world observation width of 100, the index corresponding to advancing in the 30 degrees direction is 30~100 x 8. We encode the navigation choice by selecting its corresponding world observation representation, then scoring it via dot product with the final output representation Y linears (U,) H.
360
4 Experiments
Setup How well does shared architecture do across all five SILG environments? To answer this, train and evaluate SIR using Torchbeast [33], distributed RL framework with importance weighted actor-learners based on IMPALA [18]. For each environment (separately), we train on training, do early stop validation, and evaluate on test. NetHack does not distinguish between train and evaluation, hence we create our own splits by dividing the seed range (first million seeds for training, second for validation, and third for test). We rum 5 random seeds for each environment. The hyperparameter and compute resources are respectively shown in Appendix H and I. SILG.
Results Figures 4 through 8 show learning curves for each environment. Table 2 shows the test performance for the baseline model and the best model variant. Despite sharing the same core model architecture, SIR achieves reasonable performance across all environments except Messenger, where overfits due lack of pretrained LM and entity-centric attention. Nevertheless, the best performing model significantly trails human performance, indicating room for further improvement.
4.1 Analyses of recent grounded language RL modelling contributions
Next, we SILG to evaluate recent modelling advances for language grounding across environments by adding them to the SIR baseline. These modelling enhancements were proposed for (and resulted in key gains on) the environments included in SILG. Namely, analyze the effectiveness of recurrent state-tracking, entity-centric local convolution, entity-centric attention, and pretrained LMs.

---

Table 2: Success rate on test environments for SIR and its best variant. Standard deviation are in brackets. We early stop on validation and evaluate best checkpoint on test. For RTFM, Messenger, and SILGNetHack, we evaluate 100 episodes. For ALFWorld and Touchdown, we evaluate on initial states from each test episode. The variant with best performance across envs +state. The SOTA for RTFM, Messenger, and ALFWorld are respectively from Zhong et al. [58], Hanjie et al. [22], and Shridhar et al. [49] (std was not reported in ALFWorld). SOTT for ALFWorld relies on supervised trajectories and beam search, which SIR does not use. There are previous results for multitask STI.GNetHack and SymTD as they are introduced here. Though not comparable, the manual stop VisTD SOTA trained using imitation learning on supervised trajectories is 16.7% [56].
Model
RTFM
Messenger
SII.GNetHack
ALFWorld
SymTD
inst
new inst+layouts
Base
88.8 (22.4)
(0)
23.8 (0.8)
21.0 (1.5)
16.0 (2.1)
9.7 (1.3)
Best
+state
+all
+local conv
+state
+state
+state
99.2 (0.7)
31 (2.6)
25.4 (3.3)
23.6 (2.8)
16.6 (2.9)
14.9 (1.8)
SOTA
83 (21)
85 (1.4)
N/A
40
374
N/A
Human
100
100
78.1
100
100
61.5
base
base
+all
+all
win rate Smoods
0.8
0.6
#concat
cconcctt
riocal conv
tlocal Con
+iiiity attn
+entlly attn
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
frames
10B
frames
14B
Figure 4: RTFM performance. Left: train envs, right: validation envs.
Recurrent state tracking (state) As Küttler et al. [34], we augment the SIR baseline with a state-tracking LSTM by replacing the final H with H' H + LSTM (H, S,-1), where S,- is the previous LSTM state (summing LSTM output and H outperforms replacing H with LSTM output). State-tracking consistently improves convergence and generalization, is fully determined by current world observations (c.g. RTFM). This may be because it helps prevent local minima that cause repetitive actions. The exception to this is Messenger, where state-tracking does not help generalize to the evaluation distribution.
when the correct next step
Entity-centric local convolution (local conv) Hill et al. [27] proposed local convolution around the agent obtain an egocentric view of world observations. While this helps generalize in SIL- GNetHack, it does not help significantly in other environments. One reason is that this provides redundant information as positional embeddings, which already included in the base model and is a cheaper alternative to adding an additional egocentric convnet.
Entity -cettrc attention (entity attn) Hanjie et al. [22] propose replacing entity representa- tions with attention over text specification, such that the world observations are forcibly composed using text representations. We add this by replacing world representation U with entity attention over text fields R as described in Hanjie et al. [22]. This constraint causes underfitting of SIR on most environments. Since the entity representation is built entirely using the text, when there is incomplete entity information or is difficult to extract the relevant information from the manual text this can be a handicap. However, for Messenger, entity-centric attention prevents overfitting.
Pretrained language model (bert) natural question in language-groundng! is how leverage large, pretrained LMs [28]. We use simple method to incorporate BERT [16] by replacing all text encoding with the summation of the original bidirectional L.ST encoding and BERT encoding. Due to the memory requirement of large pretrained LMs, we cannot fine-tune the LM during training, and thus keep the LM parameters fixed. Pretrained LMs (bert and all) help generalization in Messenger but does not improve performance on other enviromment in experiments. For tasks such as RTFM
7

---

mean win rate smochh
2.8
3.6
+bcal conv
Tcnnicl
+local conv
conccal
+wrlity wttn
tently alto
0.4
167
Figure 5: Messenger performance. Left: train envs, right: validation envs.
0.40
Base
tal
Fall
Wit Fete Smodls
0.30
+hert
+bet
Istate
I-taate
0.25
0.20
tontyy sttn
0.15
0.05
5.00
frames
frames
Figure 6: SILGNetHack performance. Left: train envs, right: validation envs.
and SIL,GNeHacck, use of general- purepos- LM may not be beneficial for the highly specific language used in those tasks (i.e. fantasy world with word like shaman, goblin, mage etc). We stress that this a preliminary investigation into the use of LMs these environments, and encourage future research on how to effectively use pretrained LMs across environments using SILG.
4.2 Analyses of SILG environments
Finally, examine performance of SIR and variants to analyze challenges presented by SILG.
Generalization requirement of environments SILG's evaluation environments require different types of generalization. RTFM requires generalizing to new environment dynamics by referring between world observations and multiple texts; because SIR adopts FILM² from Zhong et al. [58], it able to achieve such generalization. Messenger requires compositional entity-role generalizations. That is, if an entity (e.g. dog) has certain role (e.g. message holder) in training, such an entity-role assignment never appears validation or test. SIR quickly overfits to to entity-role assumptions (e.g. dog as the message) in training suggesting the need for additional work on achieving this type of generalization using a joint model architecture. Combining pretrained LM with other enhancements (+all) results in generalization improvement, however the convergence remains very slow. This suggests that generalizing to new dynamics across environments without obvious lexical cues from the text remains difficult challenge. SIL.NetHack and ALFWorld require generalizing to new procedurally generated scenes, which SIR achieves. the additional out-of-domain ALrld evaluation where the model must generalize to new layouts, trate-trcckin allows the model to generalize faster. Touchdown requires generalizing to new natural language instructions. Here, the baseline suffers from large generalization gap. We hypothesize that more effective means of incorporating pretrained LMs necessary to achieve this type of generalization.
Necessity of separate text fields In concat, we concatenate text fields into single string, which we encode using a bidirectional LSTM. In this case, both joint text and text field representations N set to this encoding. This degrades performance especially in RTFM, which shows that multi-hop references is more easily learned when the text fields are separated and modeled via structured attention. Note that this model variant is not shown for Touchdown because it only has one text field.
Learning from symbolic vs visual world observations Table shows that policies learned the symbolic environment transfer to the 3-D environment. Using oracle and Masked- RCNN [24], the ALFWorld policy can transferred by filling observation text templates using detected objects. Our result with oracle detector is in line with Shridhar et al. [49], though our performance is weaker

---

0:40
base
base -slate
base
0.35
Iscale
+ilale
meen Wi rate smrods
0.30
0.20
0.15
0.10
0.00
0
4
trames
167
frames
1e7
trames
1e7
Figure 7: ALFWorld perfomance. Left: train envs, middle: new instruction validation envs, right: new instruction+new layouts validation envs. For efficiency only evaluate subset (50 out of 1 140) of the validation environments for early stopping. We do train BERT variants here due to computational constraints. ALFWorld does not have entity IDs and agent location, hence do not show local convolution nor entity attention experiments.
+all
mean Wh. Late Smooth
-bet
+bert
sttato
state
+local
0.1
0.8
fromes
frames
Figure 8: SDmTD performance. Left: train envs, right: validation envs. Touchdown does not have entities, hence we do not show experiments for entity attention.
because we do not use annotated data nor DAgger [44]. As with prior results, transfer to visual worlds with layouts remains very challenging [49]. Transfer using Masked-RCNN RCNN results in large drop in performance, nevertheless SILG allows perception, albeit an important challenge, to be factored out that
focus and quickly iterate abstraction challenges. Table 3 also shows that models trained on SymTD outperform those trained on VisTD (where
10-dim PCA features from ResNet [23] panorama encoding) despite being faster (383 for SymTD 344 frames per second for VisTD). That is, by applying segmentation to obtain SymTD, we are able to obtain a better policy than training directly with visual features using VisTD. The results from both ALFWorld and SymTD show that learning in faster symbolic environments such as SILG can transfer to their visual counterparts, and allows certain perception challenges to be factored out.
Future work We find that some of the most challenging aspects of situated interactive language grounding include (1) grounding text references entities without lexical overlap, (2) choosing from large textual action spaces, and (3) interpreting complex natural language deseriptions. On the methodology front, further work needed to investigate how to effectively language grounding. Moreover, apart from recurrent state tracking, the other model enhancements do yield significant gains environments other than the ones they were proposed for. These results highlight the need for modelling techniques that generalize across environments.
pretrained LMs for
SIR suggests that with additional improvements, it may be possible to have performant model with the same architccture (but trained independently) across environments. Future work may explore whether (1) single model with the same parameters accomplish all tasks, (2) single model with pretraining can be quickly finetuned on each task, and (3) learning in one environment transferable to another. We believe SILG is well-suited help answer these questions. Furthermore, SILG is designed to easily extensible, with opportunities to add additional environments in the future.
Related Work
Benchmarks for NLP and RL. NLP benchmarks helped the development of models that generalize
across different tasks [53, 54]. Similar benchmarks have furthered research in [10 12, 50). SII.G the first benchmark for symbolic interactive language grounding with diverse set of language and

---

Table 3: Transfer task success rate from symbolic to visual envs for baseline and its best variant. Standard deviation shown in brackets. For ALFRED, we give ALFWorld-rainned models language templates filled with detected objects from vision using oracle and Masked-RCNN object detectors.
Model
ALFWord//ALFRED
SymTD to
VisTD
new inst
new inst + layouts
text
oracle
m-renn
text
oracle
m-renn
VisTD
Base Best
21.0(1.5)
11.2(3.3)
3.0(1.7)
16.0(2.1)
0.7(0.4)
0.3(0.2)
9.7(1.3)
4.3(4.0)
+state
+state
+state 14.9(1.8)
base
23.6(2.8)
11.3(1.9)
7.1(1.1)
16.6(2.9)
1.3(1.1)
0.7(0.6)
4.3(4.0)
RL challenges. SILG evaluates generalization to new dynamics with (RTFM) and without lexical cues (Messenger) between text and entities, large partially observed worlds (SII GNeeHeakkkk, large actions spaces (ALFWorld), and complex natural language instructions in rich visual scenes (SymTD)) Finally, SILG provides standard interface for symbolic interactive grounding environments via Gym, and considers the transfer to their visual counterparts (ALFWorld, SymTD). For reference, there is a host of perception-rich embodied environments not included in SILG due to the latter's emphasis on symbolic environments [1, 15, 32, 36, 39]. This emphasis allows SILG to provide an efficient benchmark for situated interactive language grounding. There are other complementary symbolic language grounding environments not included in this initial release of SILG due time consideration such as [10, 45, 46]. We look forward to incorporating these in future iterations.
Interactive language grounding Language grounded policy-learning has been explored in the context of instruction following in tasks like navigation [8, 14, 20, 26, 31, 39, 55], games [2, 4, 21, 34, 43], and robotic control [5, 25, 52]. Touchdown, NetHack, and ALFWorld are three examples of such work included in SILG. While the above environments typically assume a small fixed set of world dynamics, other work explores settings where an agent must read text manuals formulate appropriate policies for the game at hand. Branavan et al. [6] developed an agent to play Civilization more effectively by reading the game manual. Narasimhan et al. [40] and Zhong et al. [58] used text descriptions of game dynamics to learn policies that generalize to new environments and dynamics, without requiring feature engineering. Unlike these two works, Hanjie et al. [22] does not assume initial lexical overlap between entities in the world and entity references in the text manual. RTFM and Messenger are two examples of such work included in SILG.
Generalization to new environments in interactive language grounding In previous instruction following work, evaluation environments typically differ from training in their world observations. These difference range from differences in object placement in the same/new rooms (e.g. ALFWorld) to procedural generation of large game levels (e.g. NetHack). Moreover, some study generaliza- tion to new compositional instructions (e.g. Touchdown). Recent works explore generalization to new environment dynamics, which must be inferred by reading. These range from multi-step reasoning across texts (e.g. RTFM) to grounding entities new text references (e.g. Messenger). The environments in SILG explore a variety of these generalization challenges. Many modelling techniques have been proposed address these generalization challenges, including environmental variations [27], memory structures [29], pretrained language models [28], incremental guidance [11], subgoal-specification [3], and hierarchical RL [41]. Our baseline and analyses explores some of these techniques, including bidirectional feature-wise linear modulation [58], recurrent state-tracking [34], entity-centric convolution [34], entity-centric attention [22], and pretrained language modelling [28].
Conclusion
We introduced SILG, a new benchmark for evaluating language grounded agents across unique challenges posed by five symbolic interactive environments. Using SILG, we proposed the first shared architecture and analyzed recent methodological advancements in grounded language learning across on these environments. We showed that a shared architecture achieves comparable result to environment-speede speciiic methods, and that most advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for modelling techniques that generalize across environments. Finally, the most models significantly trail human performance on SILG, which suggests ample room for future work. We hope that SII. will provide unified platform for evaluating future methodological advances.
10

---

Acknowledgements
We are grateful to members of UW NLP, Princeton NLP, and Facebook AT Research for their feedback, as well as the anonymous reviewers for their helpful comments and suggestions. In particular, we thank Howard Chen for detailed discussion on Touchdown and Shunyu Yao on the manuscript. Moreover, we thank Yoav Artzi, Jesse Thomason, Edward Grefenstette and Tim Rocktäschel for their invaluable feedback during the initial stages of this project. Victor is supported in part by the ARO (AROW91 INF-ING 100-1-1--10220 (-02-1 and by the Apple AI/ML fellowship. Austin is supported by the Princeton University Graduate Fellowship.
References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-add-Languaal Navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
[2] Jacob Andreas and Dan Klein. Alignment-based compositional semantics for instruction following. In EMNLP. 2015.
[3] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. JCML, 2017.
[4] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning follow language instructions with adversarial reward induction. arXiv preprint arXiv: /80660194666 2018.
[5] Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2019.
[6] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, 2016.
[8] David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instructions from observations. San Francisco, CA, pages 859-865, 2011.
[9] Howard Chen, Alane Suhr, Dipendra Kumar Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2018.
[10] Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for OpenAl Gym. https: // thub. como ht////xximmmbbbcee /ttm//ggmmmgggeggee soyymmmnigrid,, 2018.
[11] John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-leaming. In ICLR, 2019.
[12] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In ICML, 2020.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[14] Andrea Daniele, Mohit Bansal, and Matthew Walter. Navigational instruction generation as inverse reinforcement learning with neural machine translation. In HRI. IEEE, 2017.
[15] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In CVPR, 2018.

---

[16] Jacob Devlin, Ming-
Ling-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[17] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L. Griffiths, and Alexei A. Efros. Investigating human priors for playing video games. In ICMI,, 2018.
[18) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RU RL with importance weighted actor-learner architectures. In ICML, 2018.
[19] Francis Ferraro, Nasrin Mostafazadch, Ting-Hat Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. A survey of current datasets for vision and language research. In EMNLP, 2015.
[20] Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In NAACL, 2018.
[21] Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial descriptions. In EMNLP, 2010.
[22] Austin W. Hanjie, Victor Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In ICML, 2021.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[25] Sachithra Hemachandra, Matthew R Walter, Stefanie Tellex, and Seth Teller. Learning spatial- semantic representations from natural language descriptions and scene classifications. In ICRA, 2014.
[26] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint arXiv: 1706.06561 2017.
[27] Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In /CLR, 2020.
[28] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-followwe with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382 2020.
[29] Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. In ICLR, 2021.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[31] Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-67 2018.
[32] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-Across Room: Multilingual vision-and- lan-uan navigation with dense spatiotemporal grounding. In Conference on Empirical Methods for Natural Language Processing (EMNLP), 2020.
[33] Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: PyTorch Platform for Distributed RI. arXiv preprint arXiv:1910.0333 2019.
12

---

[34] Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack learning environment. NeurIPS, 2020.
[35] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. IJCAI, 2019.
[36] Matt MacMahon, Brian J. Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006.
[37] Harsh Mchta, Yoav Artzi, Jason Baldridge, Eugene Ic, and Piotr Mirowski. Retouchdown: Adding touchdown to streetlearn as shareable resource for language grounding tasks in street view. arXiv preprint arXiv:2001.03677 2020.
[38] Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Denis Teplyashin, Karl Moritz Her- mann, Mateusz Malinowski, Matthew Koichi Grimes, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, al. The streetlearn environment and dataset. NeurIPS, 2018.
[39] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
[40] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
[41] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML, 2017.
[42] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[43] Hilke Reckman, Jeff Orkin, and Deb Roy. Learning meanings of words and constructions, grounded in virtual game. Semantic Approaches in Natural Language Processing, page 67, 2010.
[44] Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
[45] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake, A benchmark for systematic generalization in grounded language understanding. In NeurlPS, 2020.
[46] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In NeurlPS Datasets and Benchmarks Truck, 2021.
[47] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In NeurIPS, 2020.
[48] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED; A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
[49] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021.
[50] Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm con_cottoll: Software and tasks for continuous control. arXiv preprint arXiv:2006. /2983, 2020.
[51] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
13

---

[52] Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. Learning semantic maps from natural language descriptions. In RSS, 2013.
[53] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In BlackboxNLP workshop at EMNLP, 2018.
[54] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurlPS, 2019.
[55] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR, 2019.
[56] Jiannan Xiang, Xin Eric Wang, and William Yang Wang. Learning stop: A simple yet effective approach to urban vision-language navigation. In Findings of EMNLP, 2020.
[57] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[58] Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to Novel Environment Dynamics via Reading. In ICLR, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See H
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See G
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See F
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See F
15


Reviews:
[{"review_id": "hK-1n77-Jx", "reviewer": "Reviewer_UhjS", "summary": "This paper introduces a unified, OpenAI Gym-based interface to five language-and-action grounding benchmarks. This interface is designed to make it easy for researchers to evaluate whether language grounding approaches work well across a variety of different benchmarks and tasks which are challenging in different ways, as past work has largely been focused on individual benchmarks. \n\nThe benchmarks are RTFM, Messenger, NetHack, ALFWorld (all from previous work), and a new version, SymTD, of the existing Touchdown vision-and-language benchmark, which has replaced visual inputs with symbolic ones using semantic segmentations produced by a pre-trained vision model (a PSPNet trained on Cityscapes). \n\nThe paper evaluates a baseline language-conditioned policy based on a strong model developed for RTFM (FILM^2), and also experiments with various architectural additions introduced by past work on other tasks (entity attention, recurrent state tracking, egocentric convolution, and BERT). Finally, the paper presents results on symbolic-to-visual transfer for ALFWorld -> Alfred and SymTD -> Touchdown.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:*\n\n1. I found the work well-motivated: a standardized interface to these tasks should make it easier for future work to evaluate on multiple tasks, and may lower the barrier for researchers who have worked on one task to work on another. I also appreciated the symbolic-to-visual transfer motivation, which is an exciting area for future work and may lower the barrier to entry for work on embodied visual grounding.\n\n2. The baseline architecture seemed reasonable and strong, and the experiments with it are thorough. I also appreciated the paper's aim of exploring additional components introduced for these tasks, and it overall did an admirable job of this given the high computational cost of RL training on these environments (although see below for some concerns).\n\n3. While the set of tasks is diverse, the paper did a good job overall of identifying common input and output formats and common abstractions across the tasks (although see below for some questions & concerns I had about the RL formulation of the SymTD / VisTD tasks in comparison to past work).\n\n4. The paper collected expert human trajectories (from a single individual) to establish a rough performance upper bound.\n\n*Weaknesses:*\n\n1. For those benchmarks that have trajectory demonstrations available (e.g. Touchdown / VisTD), it wasn't clear to me whether the proposed framework will allow forms of imitation learning or whether it is limited to RL using e.g. graph distance reward (which may not directly match the gold trajectories -- as the appendix acknowledges -- and so might also not match the instructions exactly), or reward shaping based on trajectory supervision (which would need to be designed for each task). Imitation learning using trajectory supervision has been a standard source of supervision on tasks like Touchdown, and I suspect would be quite helpful to be able to explore a variety of model architectures within reasonable compute limitations, as IL requires less interaction with the environment than RL. \n\n2. Given the large set of embodied language grounding datasets (in addition to the ones this paper provides an interface to, there are also other influential benchmarks such as SAIL [MacMahon et al. 2006; Chen and Mooney 2011], CHAI & LANI [Misra et al. 2018], EQA [Das et al. 2018], BabyAI [Chevalier-Boisvert et al. 2018], Room-to-Room [Anderson et al. 2018], Room-Across-Room [Ku et al. 2020], and gSCAN [Ruis et al. 2020]), the paper should give a bit more justification about why these particular benchmarks and tasks were chosen, e.g. since they can be feasibly convertible into symbolic representations, or span the set of challenges in embodied grounding tasks in some way?\n\n3. The Touchdown task (which I'm most familiar with of these tasks) is set up in a way that seems to make comparison to past work challenging, as past work most closely corresponds to the Manual condition (requiring the agent to predict when to stop) detailed in the appendix, which the RL setup in this paper appears to make very difficult (from Fig 8 and 9 in the appendix).\n\n4. While I think that this framework will allow future work to address interesting questions about architecture generality across tasks like these, I feel that this paper itself made limited progress on answering these questions. Many of the components added to the baseline weren't fully explored (see below for some specific concerns), so that I don't feel the paper is justified in drawing conclusions like \"most of these components are beneficial only to the environments they were designed for\" (in the abstract, and several other places in the text).\n\n*Comments on experimental results and analysis:*\n1. It wasn't clear to me how to determine whether \"reasonable performance across all environments\" (L186) is being obtained. While the model outperforms SOTA on RTFM, it seems substantially worse than SOTA on Messenger and ALFWorld. It might help to say a bit more about what approaches these SOTA use and whether they're complementary to the approaches produced here (expanding on the Table 2 caption).\n2. If possible, given the difference between the RL setup for Touchdown and the standard supervised learning / IL setup (Chen et al. 2018) it would help to contextualize the results here in comparison to past work, ideally by training a past SOTA architecture using the RL setup here or training the baseline here via IL.\n3. The results on Messenger generally seem strange, as performance on train looks to be anti-correlated with performance on test. Since all methods (at least the ones explored here) seem to radically overfit, more should be said about what's needed to succeed in this environment.\n4. If I'm interpreting the results correctly, it seems that BERT substantially hurts performance in all conditions it was tried in except in Messenger validation (and it does hurt in Messenger train; I don't think this can really be claimed to \"help generalization in Messenger\" given my concern above). Since BERT isn't fine-tuned it should provide strictly more information about the language than the baseline so it seems odd that it usually hurts training performance so much; I feel this should be investigated more before including these results in the paper as a negative result. \n5. Entity attention seems to also produce no benefits except in Messenger validation (and see the comment above). Some more explanation of what was tried here or analysis of why it isn't working should be done. It also wasn't evaluated on ALFWorld or SymTD.\n6. 10-dimensional PCA features from the ResNet features on TouchDown seem to be quite small. Without results on VisTD that use the full ResNet features, I'm skeptical about the claim that \"models trained on SymTD outperform those trained on VisTD\" (L248).\n\n*Questions:*\n1. How conducive is the task and environment setup to training via some form of imitation learning with trajectory supervision, or how important do you think this is?\n2. How feasible would it be to add other embodied grounded environments to this benchmark (I wouldn't expect this paper to do so necessarily, but for the sake of future work that builds on it)?\n\n*Suggestions:*\n- It wasn't totally clear from Figure 1 which of the text observations are instance-specific versus non instance-specific. It would also help to clarify what an instance is (a single trajectory? a single map?).\n- It would be helpful to augment Table 1 and the text with indicators of which of the tasks involve language written by people versus language produced synthetically or by templates, as these lead to different challenges.\n- The set of tasks in this paper is diverse (as Table 1 nicely lays out) -- although I don't necessarily expect the authors to take a stand on this, if they have expectations about whether a single approach should or could do well on all of them it would be helpful to say.\n- It would help to include a few more details about the tasks to make the paper more self-contained:\n    - The adversarial and non lexical overlap nature of Messenger was a bit confusing to me. For completeness it could help to give a short example of the generalization required.\n    - How are the plausible commands in ALFWorld generated (line 113)?\n    - Add references in the SymTD section to the appendix, which answers questions I had about how the segmentations are produced and the \"passes the goal location\" evaluation.\n    - Give details on the oracle and masked-RCNN setup in ALFWorld.\n- For researchers who are less familiar with the RL training setup (e.g. from the NLP community which more frequently uses imitation learning with trajectory supervision) it might help to clarify the setup in training: is the agent simultaneously evaluated on the train and validation environments for the purposes of early stopping; and test evaluation: is the policy is evaluated on the test environments without learning (i.e. the model parameters are frozen)?\n- Given the paper's focus on Touchdown, it would make the related work more complete to also cite the original VLN task paper by Anderson et al, which introduced the Room-to-Room (R2R) dataset. Given the large body of work on R2R, I feel the paper should also give some justification for why R2R wasn't included in its suite of tasks (I'm not necessarily saying that the paper should include it, as Touchdown presents a separate set of challenges and recent work has moved past R2R to other datasets like Room-4-Room and Room-Across-Room, but it would help to further motivate this choice.)\n\n*Model suggestions:*\n- If I understand right, attending over N_i or C_i conditioned on the recurrent state could allow the agent to focus on portions of the text relevant to the history of actions it's completed.\n- It wasn't totally clear to me (either from the main text or D.2 in the appendix) whether the adjacent nodes are encoded relative to the agent's current heading, which past VLN work has found to be helpful (see the \"panoramic representation\" of Fried et al. 2018, \"Speaker-Follower Models for Vision-and-Language Navigation). \n- It might help to try other methods of combining the fixed BERT embeddings with the original LSTM encodings rather than adding, e.g. concatenation.\n\n*Minor comments:*\n- line 300: \"across on\" -> \"across\"\n- A.547: \"principle\" -> \"principal\"\n\n=== update after response ===\n\nThanks to the authors for the very thorough response! It largely addressed my concerns (more details below), and I also agree with the positive points the other reviewers pointed out. I've raised my score to a 7 (from a 6).\n\nIn my initial review I was concerned about:\n\n- *Doesn't allow imitation learning*: The authors plan to in a future release (points 1 and 10).\n- *Leaves out embodied / visual domains*: While I still do feel (in agreement with some of the other reviewers) that a symbolic benchmark leaves a lot out, I was mostly convinced by the author response about the range of challenges here even without vision and embodiment. It seems that there's a lot of progress remaining to be made that this benchmark could encourage (point 2).\n- *Comparison on touchdown is hard given the setup*: This isn't really resolved in the current work, but since the work doesn't make touchdown-specific claims, I think this is ok (point 3 and 5).\n- *Limited progress on comparing architectures*: Their explanations for the results (points 4, 6, 7, 8) were convincing to me. I do still feel that the claim about \"most of these components are beneficial only to the environments they were designed for\" would be too broad if taken as a claim about these methods (based on the results here), but if taken as motivation for the benchmark or tempered a bit (and the authors seem willing to), it's fine.\n\nRe: Messenger performance, I was convinced by the author response explaining the results on it. Messenger seems quite different as a task than the others since it requires such strong generalization, but I can see the benefit of including a compositional generalization test in this benchmark as it will likely encourage more work in this area.\n\nI also agreed that the paper was pretty clearly written as is, and the clarification points I had were all resolved in the author response (points 12-20 and 22).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the weaknesses above for suggestions on how to address what I see as the limitations in the framework (potential lack of ability to include imitation learning supervision, choice of benchmarks) and experiments (limited exploration of some of the additional components over the baseline).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Vl5Lffy6Oh3", "reviewer": "Reviewer_gUh8", "summary": "This paper proposes a multi-environment language and RL benchmark called Symbolic Interactive Language Grounding benchmark (SILG) which unifies a collection of diverse grounded language learning environments under a common interface. The environments included span language-guided grid-world environments (RTFM, Messenger, NetHack), and visual worlds (ALFWorld, Touchdown). The authors also propose a new architecture meant to comprehensively evaluate several innovations in this space on their new benchmark, and find that many innovations are environment-specific and do not transfer to new environments. These findings motivate a need for such a benchmark to standardize model evaluation and progress in this field.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This dataset integrates several important datasets in language and RL and I believe it will serve as a good basis for benchmarking future methods.\n2. The paper makes several interesting findings that innovations are environment-specific and do not transfer to new environments. This benchmark will be important to standardize model evaluation and progress in this field and I think it comes at a good time.\n3. The paper is well-written and well motivated. Experiments are clear and easy to read. As a researcher in this research area, I learned a lot from this paper.\n\nWeaknesses:\n1. I did not see a mention of any public release of this benchmark and standardized evaluation protocol. I am assuming that this will be publicly released for the community unless there is reason not to do so.\n2. Is there any reason to believe that a single model is suitable for all these environments? Given that the text, visual, and RL challenges are all quite different across these datasets, this might indeed cause certain design decisions to work better for one environment and not for another.\n3. There are many multimodal methods for language and visual grounding that I believe this paper should also cover, beyond the simple attention-based models considered in this paper. It would be better to compare more closely to works such as \"A survey of reinforcement learning informed by natural language\" and be more thorough in the comparisons of proposed approaches in that paper as well.\n4. Why were these 5 environments chosen in particular? And why were only 'egocentric local convolution, recurrent state-tracking, entity-centric attention' classified as recent advances? It would be good to describe a more systematic system for choosing and expanding these environments and models as this benchmark is used. It would be even better to organize an open call for environments and models so that the research community can expand on both of these.\n\nMinor:\n1. It would be good to add references to recent works in multimodal benchmarks in different domains such as:\n\nhttps://arxiv.org/abs/2107.07502\n\nhttps://aclanthology.org/D15-1021.pdf\n\nhttps://link.springer.com/chapter/10.1007/978-3-030-49724-8_2\n\nhttps://github.com/georgian-io/Multimodal-Toolkit\n\nnot a benchmark, but a good unifying algorithm of several approaches: https://openreview.net/pdf?id=rylnK6VtDH\n\n\n=== post rebuttal ===\nI have raised my score to a 7. I am glad that the authors will be releasing the benchmark publicly, and the authors have also alleviated some of my concerns regarding the choices of methods, techniques, and environments used in the benchmark, and I am glad they plan to expand it in the future.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have done a good job illustrating the broad impacts of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns as far as I can tell.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "HPrnpK28pf0", "reviewer": "Reviewer_MASS", "summary": "This paper proposes a benchmark for symbolic interactive language grounding that integrates 5 existing environments into a unified gym-like setting. The authors converted tasks in different environments to unified action/state space, task descriptions, within-trial information, etc., and designed a baseline model that performs over all the benchmarks. The authors also provide various adjustments for the baseline model considering the different generalization tasks within each environment. The overall evaluation shows hints on potential directions on symbolic interactive language grounding tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The SILG proposed in this paper integrates 5 existing symbolic generalization environments and serves as a unified benchmark for evaluating models' generalization capabilities over new dynamics, new layouts, new instr, etc. This benchmark has the potential to become the general testbed for symbolic interactive language grounding and could be beneficial for the field.\n\nThe paper is overall clearly written with detailed descriptions for model design. The provided possible adjustments to the backbone model summarize previous works' ideas on tackling each specific kind of generalization in different sub-environments. The analysis of the results makes sense and smoothly leads to the general challenges for modern interactive symbolic grounding tasks summarized in (L.255-259).\n\nOn the other hand, there are some concerns for the current benchmark which I hope the authors can address:\n\n(1) Model's are separately trained on different sub-environments and evaluated separately as well. As a general test-bed, it should not stop at integrating all environments into one unified framework and model design. The current evaluation protocol doesn't provide clues on what models need to possess in general in order to solve all tasks in SILG (e.g. for all the (+) modules, which one is generally the best across sub-environments). A general evaluation score (e.g. average success rate over all environments) will help illustrate things needed for models better.\n\n(2) As a constructive suggestion for building this general test-bed, it will also be good to see how we can design our models, or over what tasks can we pre-train on so that a model can be quickly adapted into all these sub-environments, for example in a more challenging few-shot setting. This is similar to the authors' intuition of leveraging (+bert), but since the current model is trained from scratch in each environment, the current evaluation is in fact showing models' ultimate potential for each task instead of a generalization capability across tasks. Compared to (1), this is an additional step to discover what ingredients are needed and how these ingredients could be obtained from other domains.\n\n(2) The authors might want to clarify some details in the experiment section. There exists early stopping of different model-design throughout the experiment section in training (Fig.1 - Fig.7), which is not discussed by the authors in the manuscript. This leads to questions on the validity of the reported results since in some experiments (e.g. Fig.6, Fig.7), the training clearly hasn't converged and steps are different from model to model (e.g. +all vs. base).  Further descriptions or illustrations will be helpful for justifying these results.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please check the previous sections for concerns and constructive feedback.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2G2JWbUcJVM", "reviewer": "Reviewer_h3yz", "summary": "The authors propose a new interactive language grounding benchmark by combining five different language grounding environments that pose different types of challenges; RTFM for multi-step multi-stage reasoning, Messenger for systematic generalisation in making entity-role assignments, Nethack for learning in partially observable environments with large state spaces, ALFWORLD for learning in environments with a large action space and Symbolic Touchdown tests for spatial reasoning. The authors also propose a new baseline model that combines recent advances in language grounding models and test it on their benchmark with different ablation tests. They use these results to analyse the challenges posed by the different environments to current methods while also assessing the benefits of the different methods used in the baseline model.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros:\n\n- The paper proposes a new benchmark by combining existing environments in a common framework (similar to benchmarks like GLUE and SUPER GLUE, which are used widely for evaluating NLI models).\n- I found the OpenAI Gym-like framework to be quite neat and convenient to use.\n- The analysis of the various features of the baseline model by using ablations,  by conducting comparisons with human participants and then discussing the results made the benefits provided by different techniques quite clear.\n- Similarly, the analysis of the challenges of solving the different environments gave a quantitative justification to the author's choice of environments.\n- Overall, the proposed SILG benchmark provides an easy way to test for and iterate over agents that interactively learn language.\n\n### Concerns /Suggestions:\n\n- The paper was quite clear and easy to understand. However, I would have appreciated more clarity at some places: The types of tasks in each environment would be more clear if Figure 1 could be changed/appended with more information about the examples shown.\n- I wasn't convinced about the justification given for using a purely symbolic setup in the benchmark. The authors provide a comparison to learning from visual observations and say that their benchmark allows \"perception challenges to be factored out\" and allows one to focus on quickly iterating on abstraction challenges. One of the most interesting challenges in grounded language learning is the interaction of language learning with learning to perceive. By factoring out perception, the benchmark does not allow perception to be influenced by the language. \nThe benchmark would have been more comprehensive if visual perception was an integral part of it.\n\n- The baseline models seem to struggle with systematic generalization in tasks like Messenger and Sym-touchdown. A related citation that is missing is GSCAN [1].\n\nReferences:\n[1] Ruis, L., Andreas, J., Baroni, M. Bouchacourt, D., and Lake, B. M. (2020). A Benchmark for Systematic Generalization in Grounded Language Understanding. Advances in Neural Information Processing Systems 33.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of the limitations and societal impact by the authors was adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Contributing a unified API and benchmark for language grounding in symbolically represented environments can potentially be valuable for the community. Such a benchmark can focus efforts, spark discourse about appropriate evaluation, and avoid bespoke, over-engineered models from dominating individual tasks and obscuring more general solutions. Reviewers all advocate for acceptance and the AC agrees. Authors are encouraged to revise the paper to address reviewer concerns -- especially to clarify how/why these environments were chosen (both these specific ones and why perceptual environments were excluded). Further, the details on Messenger's performance included in the response would be useful for future readers as well.
Decision: Accept (Poster)