SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark
Authors: Victor Zhong
Austin W. Hanjie
Sida Wang
Karthik R Narasimhan
Luke Zettlemoyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: language grounding
reinforcement learning

Abstract: Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that re- quire interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan com- plexity. In addition, we propose the first shared model architecture for RL on these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-environment benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SILG enables the community to quickly identify new methodolo- gies for language grounding that generalize to a diverse set of environments and their associated challenges.

SILG: The Multi-enviromment Symbolic Interactive Language Grounding Benchmark
Victor Zhong! Austin W. Hanjie,, Sida I. Wang?, Karthik Narasimhan' and Luke Zettlemoyer
Department Computer Science, University of Washington Department of Computer Science, Princeton University 'Facebook Al Research
Abstract
Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection diverse grounded language learning environments under common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), well symbolic counterparts of visual worlds that quire interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan com- plexity. In addition, propose the first shared model architecture for RL these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-ent envirnmennt benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SII.G enables the community to quickly identify new methodolo- gies for language grounding that generalize to a diverse set of environments and their associated challenges.
Introduction
An ideal language-conditioned agent should interpret language in diverse environments with vary- ing observation space, action space. language, and plan complexity. However, existing language- grounding literature typically focuses on single environments, and proposes methodological con- tributions specific to those environments [35, 51]. In order to determine which contributions are crrromment-specifie and which apply across multiple environments, it critical develop universal models that can be easily evaluated in many different settings.
To facilitate this research, we present the multi-environment Symbolic Interactive Language Ground- ing Benchmark (SILG). We focus on symbolic environments with semantic symbols instead of raw visual observations for efficiency, interpretabilityy and emphasis on abstractions over perception. SILG consists of diverse environments including grid-worlds RTFM [58], Messenger [22], and NetHack [34], which require generalization new dynamics (i.e. how entities behave), entity refer- ences, and partially observed worlds. SILG also contains symbolic counterparts of visual grounding
Corresponding author Victor Zhong vzhonggcs. washington. edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Agent
World obs, text obs, relative pos, reward
Action
SILG conversion
RTFM
Messenger
NetHack
ALFWorld
Touchdown
Dynamics, goal, inventory
Enemy, goal, messenge
In-game message, player stats
Goal, in in-game message
Navigation Instruction
Figure I: Environments included SILG. The world observations and text fields are shown for each environment. Detailed examples are in Appendix F.
environments ALFRED [48] and Touchdown [9], which require interpreting rich natural language in complex scenes. For the former, we use its textual variant ALFWorld [49]. For the latter, we create SymTD by applying object segmentation to Touchdown panoramas. Despite significant implementa- tion differences, we unify these environments under a common interface in SILG, so that one can easily develop and evaluate language grounded RL methods across all of these challenges.
SILG environments present variety unique grounding challenges the richness of the observation space, action space, language specification, and plan complexity. We quantify these challenges and additionally analyze success rate and lengths expert playthroughs For visual grounding environments, we show symbolic variants (ALFWorld and SymTD) facilitate faster learning and result policies that transfer to their visual counterparts. While a unified model may not outperform specialized models engineered for specific environments, it can be helpful to understand whether particular modelling innovations are environment specific or more general techniques. Furthermore, while the challenges in each environment very different, want to encourage the development of unified architectures and approaches that can scale across many language grounding tasks.
In addition to SILG, we propose the Symbolic Interactive Reader (SIR), the first shared model archi- tecture for these environments. We combine SIR with several recent advances in language-conditiong RL, including FiLM² [58], egocentric local convolution
[27]. recurrent state-tracking [34], entity- centric attention [22]. and large pretrained LMs [28]. On most environments, SIR achieves compara- ble performance methods designed specifically for single environments. In addition, we find that many recent advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for enviromennt benchmark. Finally, the best models significantly underperform humans SILG (10-85% depending on environment), which suggests ample room for modelling improvements that generalize across environments.
In summary, we (1) combine five language-groundie environments under the same interface to evaluate language grounded RL methods across diverse grounding challenges. (2) present the first shared model architecture for these environments, and (3) analyze recent modelling contributions across these environments. We hope SILG enables the community quickly identify new models and learning algorithms that generalize diverse set of environments and their associated challenges. The code for SILG available at https: github....... wom/vzhong/sii
2
SILG Environments
SILG contains five language-grounding environments including both grid-worlds (RTFM, Messenger, SILGNNeHHack) and symbolic counterparts of 3D-visual worlds (ALFWorld, SymTD). While all in- volve agents situated in interactive worlds, each presents unique challenges in richness of observation space, action space, language specification, and plan complexity. Table quantifies their theoretical complexity along these dimensions as well as empirical complexity using expert playthroughs.
For each environment, expert plays as many episodes necessary to learn about the game. We then record the playthroughs compute the empirical win and trajectory length. More details in Appendix

---

Table 1: SILG statistics. "dynamics" are high level rules dictating behaviour of entities. "Ref hops" are number of intra-text references the agent must resolve to determine correct course of action. Messenger and SymTD text are human-writte instead of procedurally generated. Distinctive properties are bold.
RTFM
Messenger
SIL.GNeHacck
ALFWorld
SymTD
Action space
5 fixed
5 fixed
23 fixed
50+ choices
1-5 choices
6 x 6 grid
10 x 10 grid
21 x 79
102 nodes
29.6k complex panoramas
State space
5 entities
14 entities
partial obs
191 entities
Mean text len
31 words
30 words
9 words
100 words
90 words
Vocab size
262 words
595 words
~ 100 words
1237 words
4999 words
new instr
Generalization
new dynamics
new dynamics
new layouts
new instr
new layouts
Ref hops
6 hops
3 hops
1 hop
4 hops
~ hops
100% new instr
Human win %
100%
100%
78.1%
61.5%
100% +layouts
7.8 steps instr
Human # steps
6.0 steps
2.2 steps
34.4 steps
9.6 steps +layouts 33.6 steps
Env FPS
240
1627
439
7
779
multi-step reasoning
adversarial generalization
large
complex language
Key challenge
partial obs
action space
The goal of SILG to provide a simple-to-use benchmark that allows researchers to quickly evaluate methods across all of these environments as well as their respective challenges. We thus combine these environments under a unified interface built on top of OpenAl Gym [7]. In each environment instance, the agent observes text inputs well as world observations. For grid worlds such as RTFM, Messenger, and SlekGk,tHack, the agent receives a 2-D bird's-eyes --eee-view symbolic grid as observations. For visually inspired environments such ALFWorld and SymTD, the agent receives symbolic egocentric view of the present scene. Figure shows how SILG environments are rendered to players via the play utility. In the rest of this section, we describe each SILG environment in detail. Appendix shows how to use SILG in Python. Appendix G shows licensing for SILG environments.
Selection criteria We select interactive environments that span the challenges presented in Table 1, are easily converted to symbolic representations, and avoid the use of additional simulators (e.g. Mat- terport3D [1]). While visual perception clearly important for language grounding [19], we focus on the unique challenges of symbolic environments such as multi-hop reasoming and generalization rich sets of procedurally generated dynamics. We leave the challenge of developing a visually rich multi-environment grounding benchmark to future work. Due to the lack of gold trajectories in many of the selected environments, we do not support imitation learning (IL. in this version of SILG.
RTFM RTFM [58] is a grid-world environment where an agent interprets text to acquire the correct items to fight the correct monsters. A key challenge in RTFM is multi-modal multi-step reasoning (at least 6 steps) combining world observations with texts associated with multiple entities. Given a team to beat, the agent must identify which monster is on the team, then identify the item descriptor that would beat the monster descriptor. Finally, the agent must acquire the item with the correct descriptor and engage the correct monster to win. RTFM evaluation is on games with unseen rules, foreing agents to make novel reasoning steps to generalize successfully. At each step, the agent receives a symbolic grid containing names of entities present, as well as texts indicating the high level rules, the agent inventory, and the goal of the particular game instance. We include all 4 RTFM curriculum stages, but only show results for the first stage in this preliminary study.
Messenger Messenger [22], is grid environment where the agent must acquire a message and deliver it to the goal while avoiding an enemy after extracting entity-role assignments from a text manual. key challenge Messenger is the adversarial vain-evaluation split without prior entity-text grounding. There is no overlap in entity-role assignments between training and evaluation, forcing agents to make compositional entity-role generalizations. each step, the agent receives a symbolic grid containing symbol IDs of entities present, well as texts indicating roles of each entity. The
3

---

factng dtrection marked Cassse corresponding direction vent
Colour key:
masy sideuali, be sall,, fenre, sale, traffic Light, traftir sig,, segetation, terrate, thy, parson, riders bonoreselly Moyelle
20th7 TEAT
Turn and rase the sonn strection as traffic. When you get to intersection, turn left, After you turn the corner building you
Rewart: com revent:
Stepez:
Dave: Fatze
Your historicst scorest
Figure 2: The SILG play utility (shown here for SymTD) enables human playthrough as well as visualizing what input the model observes. Because all environments are symbolic, the playutility works in console (e.g. via ssh, tmux) without need for X-forwarding.
entities are referred to in text by many names, which have lexical overlap with their symbol ID. That is, the text "dog" in the text for example is the non-textual symbol 2 in the observation and the association between entities and references must be learned via interaction. We include all Messenger curriculum stages, but only show results for the first stage in this preliminary study.
SlL.GNeHHack NetHack is complex rogue-like game from the NetHack learning environ- ment [34]. In SILGGNethack, we combine 3 tasks (Score, Gold, and Scout) and specify the task to complete for each episode via text prompt. SII SIL.GNetHack is challenging due to its large state space and partial observability. The agent may descend multiple floors and sections of each floor may be obscured until exploration by the agent. Because of the different score distributions of each task, we mark trajectory as successful if it exceeds a task-specific score threshold determined from human playthroughs. We evaluate agents on previously unseen map layouts that are procedurally generated with new seeds disjoint from the ones used during training. More information about the SILG multi- task SILGNetHack in Appendix D. At each step, the agent receives a symbolic grid containing symbol IDs of entities present, as well texts denoting the goal, agent stats, and feedback from the environment after the agent's last action. SII.GNeHHack vocabulary is technically infinite because players can arbitrarily name things, however in our expert playthroughs of SIIG SII Hee,Hk,tacck, we observe just over 100 unique words. Human experts win just under 80% of games with an average of 34 steps, which demonstrates the challenge of SILGNetHack. All failures can be attributed to hitting the step limit before acquiring the necessary win conditions.
ALFWORLD (text ALFRED) In ALFWorld, an agent navigates and manipulates objects inside 3D kitchen [49]. Its large text action space, with more than 50 valid actions (given by the game engine) for most scenes is a key challenge. Unlike its visual counterpart ALFRED [48] where the agent observes 3-D images of the kitchen, in ALFWorld the agent must rely on language descriptions kitchen. Goals are provided in human written language (e.g. put clean sponge on the metal rack). The language in ALFWORLD is not complex, but are 100 words on average due to a large number of items in single scene. Following recent work [49], we evaluate on both unseen instructions (new instr) and unseen room layouts (new layouts). At each step, the agent receives the goal text and a list of items present in the room (c.g. "cup 1", "bottle 2").. We concatenate the names of these items into a symbolic world observation grid each entry containing the name one item. The agent then selects from plausible commands given what is present in the scene.
the
TouchTown (SymTD, VisTD) In Touchdown, the agent navigates through Google Street View panoramas according to long compositional instructions that tests spatial reasoning [9, 37, 38). key challenge is the rich human-written navigation instructions that describe photorealistic images. Touchdown's 's long human- written instructions contain many intra-text reference hops, which we

---

RL Policy
RL baseline estimate
H
H
FILM' network
Concat
A
Jaint text summary conditioned field
Summary
D
Concat
Encode text
Attention and weighted averaging
N
Encode text
Encode text
Z
U
T
R2
Position emb
World obs emb
Joint text
Text field i-1
Text field
Figure 3: The Symbolic Interactive Reader (SIR) baseline. Inputs green, intermediate results white, outputs red, and model components yellow. Details about the FiLM² layer is in Appendix C.
approximate as the number of sentences plus the number of sequential connectors such as "then". We convert Touchdown a symbolic environment by segmentating its panoramas into semantic grids. In each step, the agent observes the instruction text and grid of discretized segmentation class IDs corresponding the current panorama. It then chooses among a list of radial directions to proceed to the next panorama. The agent wins if it passes the goal location. We use the same train-test split as the original Touchdown environment, which features unseen navigation texts.
We show that our symbolic Touchdown (SymTD) facilitates faster learning compared to learning in its visual equivalent (VisTD). Human performance demonstrates some limitations of SymTD, with an expert win rate just over 60%. This may be due to the symbolic representations removing information referenced by the instructions such as color, because the segmented features are visually disparate from real-world views [17]. We also include manual stop variants of SymTD and VisTD, which are functionally equivalent the original Touchdown. Appendix details these variants, SymTD/VisTD creation as well as discussions on human performance. Compared to prior work on Touchdown and ALFWorld, we train using RL. without supervised trajectories opposed imitation learning.
The Symbolic Interactive Reader Baseline Model
Figure 3 shows the SIR baseline for the SILG benchmark. To the best of our knowledge, this is the first shared model architecture for RTFM, Messenger, NetHack, ALFWorld, and Touchdown. Consider an agent situated in an arbitrary SILG environment. At each time step I, the model receives from the environment the following inputs (precise inputs for each environments are shown in Appendix F).
World observations X
hxx where h and w are the height and width of the observa- tion and each element corresponds to the k-word symbol ID(s) of its content.
Joint text T E IR of I tokens of the text to attend over.
Text fields A E gnxm where the ith row contains the ith of n environment text field such agent inventory or environment feedback. is the max token count of these texts.
Relative position Z E hxxx2 cell-wise feature that denotes the position of each cell relative to the player agent in the x and y directions.
As policy learner, the model must output distribution Y over the action space. We additionally output baseline estimate of the value function stabilize policy learning [18]. Let and denote embedding and bidirectional LSTM sizes. We first sum embeddings for each cell in the world observation obtain world representation U sum (emb (X)) Rxxwxd Next, encode the ith text field R, and the joint text T using an bidirectional LSTMs [30].
N; = BiLSTMN (emb (R))) E Rx
(1)
= BILSTMp (emb (T)) E g/xr
(2)
We then compute weighted average over text fields C and attention A, over the joint text.
5

---

C2 weightave, (N₂)
Σ
softmax (linear, (Ni)), Nos R
(3)
A = attend (D.C:)=
Σ softmax (Do:), Dj E ER"
(4)
We compress Ö E ga and A gaxe again to support any number of text fields.
C = weightave (+)E
EER' R' (5)
A = weightave, A A R"
(6)
We now have representations for world observations U, text fields C. and joint text conditioned on text fields A. We apply successive FiLM layers to build multiple levels of codependent representations between texts and world observations to model multiple cross-man moddll reasoning steps [58]. To support arbitrary number of text fields, we modify the text input of the ith FiLM layer to be the concatenation of the text fields C, attention over joint text conditioned on text fields A, and attention over joint text conditioned on the visual summary of the last FiLM² layer s(i-1).
FiLM ([v(e-); z], [C. A, attend (D,s(i-1)])
(7)
We use the definition of FiLM (visuals, texts) from Zhong et al. [58] and summarize its intuition and computation in Appendix C. We define (1) and 5(4) to be the initial world observation U and its spacial max- max-pooling. Finally, we use multi-layer perceptron to build a fixed-size codependent repre- sentation of the inputs based on the last FiLM² layer's output which is used compute the baseline estimate of the value function B ML.P z (H) and the policy (H) expressed as a distribution over actions. While the core architecture of SIR is identical for all environments, different policy module Y is necessary for different types of action spaces.
tanh (lincar, (flatten( (V(ss))))
Fixed sized action space (RTFM, Messenger, SIL.NetHHcck) We simply apply multilayer perceptron to the final representation Y MLPY (H).
Multiple-choice text action space (ALFWorld) Let Q denote tokens for the jth choice (e.g. pick
up the mug), which encode bidrectional LSTM G, BiLSTMc (emb (Q,)). We then attend over this text using the final representation H to score for jth choice Y; linear4 (attend (Gj, H)).
Multiple-choice navigation action space (SILGTouchdown) Let j denote the index of the world representation corresponding to a movement direction. For example, for a world observation width of 100, the index corresponding to advancing in the 30 degrees direction is 30~100 x 8. We encode the navigation choice by selecting its corresponding world observation representation, then scoring it via dot product with the final output representation Y linears (U,) H.
360
4 Experiments
Setup How well does shared architecture do across all five SILG environments? To answer this, train and evaluate SIR using Torchbeast [33], distributed RL framework with importance weighted actor-learners based on IMPALA [18]. For each environment (separately), we train on training, do early stop validation, and evaluate on test. NetHack does not distinguish between train and evaluation, hence we create our own splits by dividing the seed range (first million seeds for training, second for validation, and third for test). We rum 5 random seeds for each environment. The hyperparameter and compute resources are respectively shown in Appendix H and I. SILG.
Results Figures 4 through 8 show learning curves for each environment. Table 2 shows the test performance for the baseline model and the best model variant. Despite sharing the same core model architecture, SIR achieves reasonable performance across all environments except Messenger, where overfits due lack of pretrained LM and entity-centric attention. Nevertheless, the best performing model significantly trails human performance, indicating room for further improvement.
4.1 Analyses of recent grounded language RL modelling contributions
Next, we SILG to evaluate recent modelling advances for language grounding across environments by adding them to the SIR baseline. These modelling enhancements were proposed for (and resulted in key gains on) the environments included in SILG. Namely, analyze the effectiveness of recurrent state-tracking, entity-centric local convolution, entity-centric attention, and pretrained LMs.

---

Table 2: Success rate on test environments for SIR and its best variant. Standard deviation are in brackets. We early stop on validation and evaluate best checkpoint on test. For RTFM, Messenger, and SILGNetHack, we evaluate 100 episodes. For ALFWorld and Touchdown, we evaluate on initial states from each test episode. The variant with best performance across envs +state. The SOTA for RTFM, Messenger, and ALFWorld are respectively from Zhong et al. [58], Hanjie et al. [22], and Shridhar et al. [49] (std was not reported in ALFWorld). SOTT for ALFWorld relies on supervised trajectories and beam search, which SIR does not use. There are previous results for multitask STI.GNetHack and SymTD as they are introduced here. Though not comparable, the manual stop VisTD SOTA trained using imitation learning on supervised trajectories is 16.7% [56].
Model
RTFM
Messenger
SII.GNetHack
ALFWorld
SymTD
inst
new inst+layouts
Base
88.8 (22.4)
(0)
23.8 (0.8)
21.0 (1.5)
16.0 (2.1)
9.7 (1.3)
Best
+state
+all
+local conv
+state
+state
+state
99.2 (0.7)
31 (2.6)
25.4 (3.3)
23.6 (2.8)
16.6 (2.9)
14.9 (1.8)
SOTA
83 (21)
85 (1.4)
N/A
40
374
N/A
Human
100
100
78.1
100
100
61.5
base
base
+all
+all
win rate Smoods
0.8
0.6
#concat
cconcctt
riocal conv
tlocal Con
+iiiity attn
+entlly attn
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
frames
10B
frames
14B
Figure 4: RTFM performance. Left: train envs, right: validation envs.
Recurrent state tracking (state) As Küttler et al. [34], we augment the SIR baseline with a state-tracking LSTM by replacing the final H with H' H + LSTM (H, S,-1), where S,- is the previous LSTM state (summing LSTM output and H outperforms replacing H with LSTM output). State-tracking consistently improves convergence and generalization, is fully determined by current world observations (c.g. RTFM). This may be because it helps prevent local minima that cause repetitive actions. The exception to this is Messenger, where state-tracking does not help generalize to the evaluation distribution.
when the correct next step
Entity-centric local convolution (local conv) Hill et al. [27] proposed local convolution around the agent obtain an egocentric view of world observations. While this helps generalize in SIL- GNetHack, it does not help significantly in other environments. One reason is that this provides redundant information as positional embeddings, which already included in the base model and is a cheaper alternative to adding an additional egocentric convnet.
Entity -cettrc attention (entity attn) Hanjie et al. [22] propose replacing entity representa- tions with attention over text specification, such that the world observations are forcibly composed using text representations. We add this by replacing world representation U with entity attention over text fields R as described in Hanjie et al. [22]. This constraint causes underfitting of SIR on most environments. Since the entity representation is built entirely using the text, when there is incomplete entity information or is difficult to extract the relevant information from the manual text this can be a handicap. However, for Messenger, entity-centric attention prevents overfitting.
Pretrained language model (bert) natural question in language-groundng! is how leverage large, pretrained LMs [28]. We use simple method to incorporate BERT [16] by replacing all text encoding with the summation of the original bidirectional L.ST encoding and BERT encoding. Due to the memory requirement of large pretrained LMs, we cannot fine-tune the LM during training, and thus keep the LM parameters fixed. Pretrained LMs (bert and all) help generalization in Messenger but does not improve performance on other enviromment in experiments. For tasks such as RTFM
7

---

mean win rate smochh
2.8
3.6
+bcal conv
Tcnnicl
+local conv
conccal
+wrlity wttn
tently alto
0.4
167
Figure 5: Messenger performance. Left: train envs, right: validation envs.
0.40
Base
tal
Fall
Wit Fete Smodls
0.30
+hert
+bet
Istate
I-taate
0.25
0.20
tontyy sttn
0.15
0.05
5.00
frames
frames
Figure 6: SILGNetHack performance. Left: train envs, right: validation envs.
and SIL,GNeHacck, use of general- purepos- LM may not be beneficial for the highly specific language used in those tasks (i.e. fantasy world with word like shaman, goblin, mage etc). We stress that this a preliminary investigation into the use of LMs these environments, and encourage future research on how to effectively use pretrained LMs across environments using SILG.
4.2 Analyses of SILG environments
Finally, examine performance of SIR and variants to analyze challenges presented by SILG.
Generalization requirement of environments SILG's evaluation environments require different types of generalization. RTFM requires generalizing to new environment dynamics by referring between world observations and multiple texts; because SIR adopts FILM² from Zhong et al. [58], it able to achieve such generalization. Messenger requires compositional entity-role generalizations. That is, if an entity (e.g. dog) has certain role (e.g. message holder) in training, such an entity-role assignment never appears validation or test. SIR quickly overfits to to entity-role assumptions (e.g. dog as the message) in training suggesting the need for additional work on achieving this type of generalization using a joint model architecture. Combining pretrained LM with other enhancements (+all) results in generalization improvement, however the convergence remains very slow. This suggests that generalizing to new dynamics across environments without obvious lexical cues from the text remains difficult challenge. SIL.NetHack and ALFWorld require generalizing to new procedurally generated scenes, which SIR achieves. the additional out-of-domain ALrld evaluation where the model must generalize to new layouts, trate-trcckin allows the model to generalize faster. Touchdown requires generalizing to new natural language instructions. Here, the baseline suffers from large generalization gap. We hypothesize that more effective means of incorporating pretrained LMs necessary to achieve this type of generalization.
Necessity of separate text fields In concat, we concatenate text fields into single string, which we encode using a bidirectional LSTM. In this case, both joint text and text field representations N set to this encoding. This degrades performance especially in RTFM, which shows that multi-hop references is more easily learned when the text fields are separated and modeled via structured attention. Note that this model variant is not shown for Touchdown because it only has one text field.
Learning from symbolic vs visual world observations Table shows that policies learned the symbolic environment transfer to the 3-D environment. Using oracle and Masked-R RCNN [24], the ALFWorld policy can transferred by filling observation text templates using detected objects. Our result with oracle detector is in line with Shridhar et al. [49], though our performance is weaker

---

0:40
base
base -slate
base
0.35
Iscale
+ilale
meen Win rate smrods
0.30
0.20
0.15
0.10
0.00
0
4
trames
167
frames
1e7
trames
1e7
Figure 7: ALFWorld perfomance. Left: train envs, middle: new instruction validation envs, right: new instruction+new layouts validation envs. For efficiency only evaluate subset (50 out of 1 140) of the validation environments for early stopping. We do train BERT variants here due to computational constraints. ALFWorld does not have entity IDs and agent location, hence do not show local convolution nor entity attention experiments.
+all
mean Wh. Late Smooth
-bet
+bert
sttato
state
+local
0.1
0.8
fromes
frames
Figure 8: SDmTD performance. Left: train envs, right: validation envs. Touchdown does not have entities, hence we do not show experiments for entity attention.
because we do not use annotated data nor DAgger [44]. As with prior results, transfer to visual worlds with layouts remains very challenging [49]. Transfer using Masked-RCNN RCNN results in large drop in performance, nevertheless SILG allows perception, albeit an important challenge, to be factored out that
focus and quickly iterate abstraction challenges. Table 3 also shows that models trained on SymTD outperform those trained on VisTD (where
10-dim PCA features from ResNet [23] panorama encoding) despite being faster (383 for SymTD 344 frames per second for VisTD). That is, by applying segmentation to obtain SymTD, we are able to obtain a better policy than training directly with visual features using VisTD. The results from both ALFWorld and SymTD show that learning in faster symbolic environments such as SILG can transfer to their visual counterparts, and allows certain perception challenges to be factored out.
Future work We find that some of the most challenging aspects of situated interactive language grounding include (1) grounding text references entities without lexical overlap, (2) choosing from large textual action spaces, and (3) interpreting complex natural language deseriptions. On the methodology front, further work needed to investigate how to effectively use pretrained LMs for language grounding. Moreover, apart from recurrent state tracking, the other model enhancements do yield significant gains environments other than the ones they were proposed for. These results highlight the need for modelling techniques that generalize across environments.
SIR suggests that with additional improvements, it may be possible to have performant model with the same architccture (but trained independently) across environments. Future work may explore whether (1) single model with the same parameters accomplish all tasks, (2) single model with pretraining can be quickly finetuned on each task, and (3) learning in one environment transferable to another. We believe SILG is well-suited help answer these questions. Furthermore, SILG is designed to easily extensible, with opportunities to add additional environments in the future.
Related Work
Benchmarks for NLP and RL. NLP benchmarks helped the development of models that generalize
across different tasks [53, 54]. Similar benchmarks have furthered research in [10 12, 50). SII.G the first benchmark for symbolic interactive language grounding with diverse set of language and

---

Table 3: Transfer task success rate from symbolic to visual envs for baseline and its best variant. Standard deviation shown in brackets. For ALFRED, we give ALFWorld-rainned models language templates filled with detected objects from vision using oracle and Masked-RCNN object detectors.
Model
ALFWord//ALFRED
SymTD to
VisTD
new inst
new inst + layouts
text
oracle
m-renn
text
oracle
m-renn
VisTD
Base Best
21.0(1.5)
11.2(3.3)
3.0(1.7)
16.0(2.1)
0.7(0.4)
0.3(0.2)
9.7(1.3)
4.3(4.0)
+state
+state
+state 14.9(1.8)
base
23.6(2.8)
11.3(1.9)
7.1(1.1)
16.6(2.9)
1.3(1.1)
0.7(0.6)
4.3(4.0)
RL challenges. SILG evaluates generalization to new dynamics with (RTFM) and without lexical cues (Messenger) between text and entities, large partially observed worlds (SII GNeeeeakkkk, large actions spaces (ALFWorld), and complex natural language instructions in rich visual scenes (SymTD)) Finally, SILG provides standard interface for symbolic interactive grounding environments via Gym, and considers the transfer to their visual counterparts (ALFWorld, SymTD). For reference, there is a host of perception-rich embodied environments not included in SILG due to the latter's emphasis on symbolic environments [1, 15, 32, 36, 39]. This emphasis allows SILG to provide an efficient benchmark for situated interactive language grounding. There are other complementary symbolic language grounding environments not included in this initial release of SILG due time consideration such as [10, 45, 46]. We look forward to incorporating these in future iterations.
Interactive language grounding Language grounded policy-learning has been explored in the context of instruction following in tasks like navigation [8, 14, 20, 26, 31, 39, 55], games [2, 4, 21, 34, 43], and robotic control [5, 25, 52]. Touchdown, NetHack, and ALFWorld are three examples of such work included in SILG. While the above environments typically assume a small fixed set of world dynamics, other work explores settings where an agent must read text manuals formulate appropriate policies for the game at hand. Branavan et al. [6] developed an agent to play Civilization more effectively by reading the game manual. Narasimhan et al. [40] and Zhong et al. [58] used text descriptions of game dynamics to learn policies that generalize to new environments and dynamics, without requiring feature engineering. Unlike these two works, Hanjie et al. [22] does not assume initial lexical overlap between entities in the world and entity references in the text manual. RTFM and Messenger are two examples of such work included in SILG.
Generalization to new environments in interactive language grounding In previous instruction following work, evaluation environments typically differ from training in their world observations. These difference range from differences in object placement in the same/new rooms (e.g. ALFWorld) to procedural generation of large game levels (e.g. NetHack). Moreover, some study generaliza- tion to new compositional instructions (e.g. Touchdown). Recent works explore generalization to new environment dynamics, which must be inferred by reading. These range from multi-step reasoning across texts (e.g. RTFM) to grounding entities new text references (e.g. Messenger).
The environments in SILG explore a variety of these generalization challenges. Many modelling techniques have been proposed address these generalization challenges, including environmental variations [27], memory structures [29], pretrained language models [28], incremental guidance [11], subgoal-specification [3], and hierarchical RL [41]. Our baseline and analyses explores some of these techniques, including bidirectional feature-wise linear modulation [58], recurrent state-tracking [34], entity-centric convolution [34], entity-centric attention [22], and pretrained language modelling [28].
Conclusion
We introduced SILG, a new benchmark for evaluating language grounded agents across unique challenges posed by five symbolic interactive environments. Using SILG, we proposed the first shared architecture and analyzed recent methodological advancements in grounded language learning across on these environments. We showed that a shared architecture achieves comparable result to environment-speede speciiic methods, and that most advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for modelling techniques that generalize across environments. Finally, the most models significantly trail human performance on SILG, which suggests ample room for future work. We hope that SII. will provide unified platform for evaluating future methodological advances.
10

---

Acknowledgements
We are grateful to members of UW NLP, Princeton NLP, and Facebook AT Research for their feedback, as well as the anonymous reviewers for their helpful comments and suggestions. In particular, we thank Howard Chen for detailed discussion on Touchdown and Shunyu Yao on the manuscript. Moreover, we thank Yoav Artzi, Jesse Thomason, Edward Grefenstette and Tim Rocktäschel for their invaluable feedback during the initial stages of this project. Victor is supported in part by the ARO (AROW91 INF-ING 100-1-1--10220 (--2-1 and by the Apple AI/ML fellowship. Austin is supported by the Princeton University Graduate Fellowship.
References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Wision-add-Languaal Navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
[2] Jacob Andreas and Dan Klein. Alignment-based compositional semantics for instruction following. In EMNLP. 2015.
[3] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. JCML, 2017.
[4] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning follow language instructions with adversarial reward induction. arXiv preprint arXiv: /80660194666 2018.
[5] Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2019.
[6] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, 2016.
[8] David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instructions from observations. San Francisco, CA, pages 859-865, 2011.
[9] Howard Chen, Alane Suhr, Dipendra Kumar Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2018.
[10] Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for OpenAl Gym. https: //gi::///ttthue thub. como ht////xximmmbbbcee /ttm//ggmmmgggeggee soyymmmnigrid,, 2018.
[11] John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-leaming. In ICLR, 2019.
[12] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In ICML, 2020.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[14] Andrea Daniele, Mohit Bansal, and Matthew Walter. Navigational instruction generation as inverse reinforcement learning with neural machine translation. In HRI. IEEE, 2017.
[15] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In CVPR, 2018.

---

[16] Jacob Devlin, Ming
Ling-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[17] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L. Griffiths, and Alexei A. Efros. Investigating human priors for playing video games. In ICMI,, 2018.
[18) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RU RL with importance weighted actor-learner architectures. In ICML, 2018.
[19] Francis Ferraro, Nasrin Mostafazadch, Ting-Hat Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. A survey of current datasets for vision and language research. In EMNLP, 2015.
[20] Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In NAACL, 2018.
[21] Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial descriptions. In EMNLP, 2010.
[22] Austin W. Hanjie, Victor Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In ICML, 2021.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[25] Sachithra Hemachandra, Matthew R Walter, Stefanie Tellex, and Seth Teller. Learning spatial- semantic representations from natural language descriptions and scene classifications. In ICRA, 2014.
[26] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint arXiv: 1701177566655 2017.
[27] Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In /CLR, 2020.
[28] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-followwe with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382 2020.
[29] Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. In ICLR, 2021.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[31] Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-67 2018.
[32] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-Across Room: Multilingual vision-and- lan-uan navigation with dense spatiotemporal grounding. In Conference on Empirical Methods for Natural Language Processing (EMNLP), 2020.
[33] Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: PyTorch Platform for Distributed RI. arXiv preprint arXiv:1910.0333 2019.
12

---

[34] Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack learning environment. NeurIPS, 2020.
[35] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. IJCAI, 2019.
[36] Matt MacMahon, Brian J. Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006.
[37] Harsh Mchta, Yoav Artzi, Jason Baldridge, Eugene Ic, and Piotr Mirowski. Retouchdown: Adding touchdown to streetlearn as shareable resource for language grounding tasks in street view. arXiv preprint arXiv:2001.03677 2020.
[38] Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Denis Teplyashin, Karl Moritz Her- mann, Mateusz Malinowski, Matthew Koichi Grimes, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, al. The streetlearn environment and dataset. NeurIPS, 2018.
[39] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
[40] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
[41] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML, 2017.
[42] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[43] Hilke Reckman, Jeff Orkin, and Deb Roy. Learning meanings of words and constructions, grounded in virtual game. Semantic Approaches in Natural Language Processing, page 67, 2010.
[44] Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
[45] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake, A benchmark for systematic generalization in grounded language understanding. In NeurlPS, 2020.
[46] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In NeurlPS Datasets and Benchmarks Truck, 2021.
[47] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In NeurIPS, 2020.
[48] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED; A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
[49] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021.
[50] Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm mon_cottoll: Software and tasks for continuous control. arXiv preprint arXiv:2006. /2983, 2020.
[51] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
13

---

[52] Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. Learning semantic maps from natural language descriptions. In RSS, 2013.
[53] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In BlackboxNLP workshop at EMNLP, 2018.
[54] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurlPS, 2019.
[55] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR, 2019.
[56] Jiannan Xiang, Xin Eric Wang, and William Yang Wang. Learning stop: A simple yet effective approach to urban vision-language navigation. In Findings of EMNLP, 2020.
[57] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[58] Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to Novel Environment Dynamics via Reading. In ICLR, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See H
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See G
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See F
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See F
15


Reviews:
[{"review_id": "juEs8rSEUay", "reviewer": "Reviewer_hUgP", "summary": "This paper presents SILG, a broad environment for language-conditioned multitask RL that unites several recently proposed symbolic language+RL environments, including RTFM, NetHack, Messenger, as well as instruction-following datasets including ALFWorld and a symbolic version of Touchdown. To unify such disparate environments, authors introduce a general framework for language conditioned RL, where agents operate given a partial observation of an environment, as well as language divided between \"dynamics text\", which consists of static, high-level language descriptions of the environment, and \"non-dynamics text\" which contains language specific to the agents' current observation and state, e.g. descriptions of the current observation, environment feedback, etc.\n\nAuthors then present a baseline model, SIR, a fairly standard language+RL model that conditions on text of varying levels of abstraction, applying successive FiLM2 layers to simulate multimodal reasoning between textual information and current observation. This entire architecture can then be trained without changes to each environment, which allows us to evaluate the generalizability of various modeling decisions in the language conditioned RL space. The authors show that many innovations proposed for certain environments (e.g. egocentric local convolution, forcing the building of representations from text dynamics) do not generalize across environments. SIR lags behind human performance on most of the environments, demonstrating potential for future work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this paper: there is undoubtedly substantial engineering work that is put into making such disparate environments accessible under a common modeling/dataset/environment umbrella, which should certainly speed up prototyping and evaluation in language conditioned RL. I do see a few weaknesses of the paper as presented currently, however.\n\n## Strengths\n\n- This paper provides a broad framework for multi-task RL that conditions on language at varying levels of abstraction (dynamics/non-dynamics text). Providing a unified, simple environment for evaluation across all of these environments will undoubtedly be very useful for future work in this field.\n- The paper shows quite nicely that several advancements developed for specific environments (perhaps unsurprisingly) fail to generalize reliably to other environments. This further highlights the need to come up with more general purpose language-conditioned RL techniques, in the same way that e.g. GLUE served as a good goal for more general-purpose NLU systems). Due to the unified environment and architecture, SILG should serve a similar role.\n\n## Weaknesses\n\nMy main concerns have to do with the SymTD environment, the sparse description of the included environments, the lack of ablation studies on the base SIR model itself.\n\n- Perhaps the most questionable part of the work is the SymTD environment, which appears to be a rather ad-hoc modification of an existing environment that has not yet been subject to the level of scrutiny of the existing environments, all of which are parts of peer-reviewed publications. There are several worrying aspects of SymTD:\n    - An expert win rate of 60% is quite low and demonstrates significant noise in the environment. It could also be that many tasks have been converted incorrectly and are therefore impossible. What kinds of obstacles do human players face that result in such a low win rate?\n    - Images are segmented to panoramas, but no evaluation about how accurate these panoramas are is presented. There are reasons to be suspect about the segmentation quality (e.g. because cityscapes are standard images, while touchdown is highly distorted panoramas).\n- FiLM2 itself is a \"recent grounded language RL modelling contribution\" along the lines of those presented in 4.1, and several of the same questions asked in that section apply to the base SIR model: does FiLM2 generalize across environments? It would be good to have ablations of FiLM2, and more general ablations of the base SIR model (e.g. no language, of either kind).\n- The descriptions of the environments included in SILG are extremely sparse (admittedly due to space constraints). The paper would benefit from clear examples of tasks across each environment, and a better characterization of the kinds of language information given in the environment and how they map onto the dynamics/non-dynamics text components of SIR. For example, for RTFM, presumably the game manual is categorized as dynamics text, but it's unclear whether the goal is dynamics text or non-dynamics text. Similarly, environments like ALFWorld and SymTD do *not* have high level game dynamics descriptions (I believe?)—is there simply no dynamics text in this setting? Touchdown only has one text field (L239) - is this dynamics or non-dynamics? One presentational suggestion is to add additional information in Table 1 that divides mean text len/vocab size by whether or not it's treated as dynamics text or not, and additionally a high level description of what kinds of language belong in which category (e.g. RTFM has game manual, ALFWorld has text description of environment, etc). (Maybe in Figure 1 as well?)\n- Along these lines, the \"relative position\" feature for SIR is only briefly mentioned in L141, which left me confused. Why is this relative position to \"a particular cell\"? What particular cell is this, and why is it necessary?\n- Some environments may be too easy: the inclusion of only the first stage of RTFM is rather odd, since we are already at human performance on this baseline (Table 2). What was the motivation for including only the first stage? (Similarly, only an easier subset of tasks was selected for Messenger and NetHack). Are the more difficult stages in these papers still compatible with both the SILG environment and the SIR framework? I would hope that authors will be able to include such harder environments in subsequent releases.\n\n## Questions\n\n- This was actually not super clear from the paper: SIR is a unified architecture, but it is trained in isolation on each individual environment, correct? Rather than training a single model across all environments.\n    - (Assuming the above is true) SILG might enable some interesting questions regarding training across multiple environments and transfer learning. E.g. perhaps training on some environments actually enables faster learning/transfer to others, since there are general game-playing ideas common to each environment. Did authors try the (dramatic) attempt of training jointly across all environments at once?\n- Table 2 lists performance of the best variant for each environment—what is the variant that leads to the best performance averaged across all environments? Presumably it's \"+state\"?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, Appendix A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "iJSHHIg9amY", "reviewer": "Reviewer_bubM", "summary": "Because existing language grounding work operates in single environments, it is difficult to determine whether model contributions apply across diverse settings. To remedy this, the authors present the Symbolic Interactive Language Grounding Benchmark (SILG), an environment simplifying and unifying five existing language grounding environments: RTFM, Messenger, NetHack, ALFWorld, and SymTD. The authors focus on symbolic environments over visual environments to abstract out the component of visual perception.\nIn addition, the authors provide a baseline model for SILG, Symbolic Interactive Reader (SIR). SIR's core architecture is compatible with all environments in SILG. They test SIR and several variants thereof containing recent modelling contributions, finding that while SIR achieves decent performance on the benchmark, it falls short of human expert playthroughs, suggesting room for future work. In addition, they find that most contributions except for recurrent state tracking are beneficial in only the environment for which they were originally developed. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Contributions:\n1. Introducing a multi-environment symbolic language grounding benchmark, SILG that combines five language-grounding environments under the same interface.\n\nWhile the authors make a good argument for training/testing models on multiple, diverse environments instead of single environments, the five complex existing environments are significantly simplified. The value added for combining these existing benchmarks under one umbrella is unclear.\n\n\n2. Providing a baseline model for this benchmark (SIR).\n\nSIR consists of a FiLM^2 model which is provided with different types of inputs. The authors should simply say they use FiLM^2 as the baseline model. \n\n\n3. Results of variants of FiLM^2 (termed SIR) on these benchmarks.\n\nIt’s unclear what insights are gained from these results which would lead to better models. This is underscored by the fact that it’s hard to compare performance across such different benchmarks on different models.\n\n\n===============================================================================\n\nHigh-level review\n\ni. Testing on multiple, diverse environments to analyze generalization of RL algorithms is an existing paradigm. It is not clear why we want to combine simplified versions of five existing environments into one benchmark. It’s also detrimental to keep the original names like (NetHack) for the simplified version; this will only sow confusion.\n\nii. Generalization is the main focus, but different models are used for different experiments and no cross-environment generalization is presented. This weakens the results significantly.\n\niii. Fundamental details are missing, such as what the dynamic text is for different games, making the results impossible to reproduce.\n\niv. A core result of the paper suggests that \"algorithm X works on environment A but not environment B\". This is misleading, as models are not comparable across environments (model size, environment size, hyperparameters).\n\n\nMore in the detailed review.\n\n===============================================================================\n\nTesting the generalization of existing algorithms to other environments is an interesting problem. However, in general, the method, model selection, and evaluation need to be better described and justified. There are several major issues with the manuscript as it stands, summarized here from the detailed review below with references to the detailed review:\n\n- (1, 2, 12) The authors should discuss how to interpret the analysis of different model enhancements. It's not clear that different models, environments, and tasks are comparable, especially when the base models have different hyperparameters for different environments, and when models are trained on single environments. This reduces the impact of combining multiple environments into one benchmark. If the result is that models are trained/tested on single environments, it is not clear that the authors have demonstrated significant progress towards solving the motivating problem.\n\n- (3, 5, 6, 11) As it stands, the work is not reproducible from reading the paper. The authors should report methodology with more detail, including environment simplification and model selection.\n\n- (4, 8, 10) Reported results (Tables 2, 3) are difficult to interpret and lacking some information. \n\nI would vote for rejecting the publication based on the comments above. \n\n========================================================================================\n\nDetailed comments:\n\nMajor:\n\n1) Why can we compare the algorithmic contributions across different environments? There is no relationship between the environments, and the models are different for each environment. This makes results very difficult to interpret. \n\n2) The evaluation metric in comparing the performance of recent modelling contributions is not stated (it would help to add a couple words on the evaluation metric at the beginning of Section 4.1). I believe the evaluation metric is an algorithm's performance improvement over the base.\n\n3) (Appendix G) How are model hyperparameters selected? Are learning hyperparameters the same for each environment? Tuning hyperparameters for each environment separately results in a different model per environment. \n\n4) Standard deviations in Tables 2, 3 are quite large and overlapping, which makes it hard to differentiate Base and Best. \n\n5)  Pg. 3 (NetHack): There are 23 fixed actions (Table 1). However, in the NetHack paper, there are 93 possible actions that players can take. How is the subset of actions chosen? In the expert playthroughs, do they also only access the 23 same fixed actions?\n\n6) Pg. 3 (NetHack) \"…in our expert playthroughs of SILG NetHack, we observe just over 100 unique words.\" How is this number relevant to the agent? How does the agent name objects? \n\n7) Lines 30-32, Figure 1, lines 57-58 state that SILG contains five language-grounding environments (RFTM, Messenger, NetHack, ALFWorld, SymTD). However, SILG does not contain RFTM nor NetHack, but rather one level of RFTM and three NetHack tasks. Like ALFWorld and SymTD, it is important to avoid stating that SILG contains RFTM and NetHack. This could be remedied by giving the simplified RFTM and NetHack versions a different name and stating their relationship with the original RFTM, NetHack environments. \n\n8) Please include the standard deviation for SOTA in Table 2 so that Base, Best, and SOTA are easier to compare. \n\n9) \"Table 3 shows that policies learned in the symbolic environment transfer to the 3-D environment.\"-- For sake of comparison, the SOTAs on ALFRED and VisTD should be included. \n\n10) Though one of the main conclusions of the paper is that recurrent state-tracking is beneficial across environments (including SymTD), it does not result in the best transfer results to VisTD. If transfer to VisTD is the main practical usage of training with SymTD, then finding that recurrent state-tracking is best in SymTD may not be relevant.\n\n11) A key challenge in reproducibility is that we don't know what the dynamics text and non-dynamics text are for each environment. It would be helpful to state for each environment where these texts are generated or found in instruction manuals, how they're processed, etc. \n\n12) Models differ per environment (thus incomparable) and are trained/tested on single environments. Training/testing on single environments was a motivation for the work. Moreover, environments are simplified versions of the originals. Then, how does using SILG improve from training individually on the original environments?\n\nMinor:\n\n14) How was the experimental subject recruited? If possible, it would be better to have more participants.\n\n15) \"This may be because it helps prevent local minima that cause repetitive actions\". I would look into pg. 2 in the Resnet paper (https://arxiv.org/pdf/1512.03385.pdf) for a theoretical explanation for why H' may converge faster than $LSTM(H, S_{t-1})$. \n\n16) \"Dynamics\" and \"non-dynamics text\" is confusing terminology. It may help to italicize these to help the reader view them as being defined in lines 66-67. \n\n17) Line 160: how do we get 4 in the subscript?\n\n18) Equations 5, 6: Perhaps replace the C, A in the subscripts or remove the subscripts? As they were being defined on the left and reused in the subscript.\n\n19) Reference hops are mentioned several times throughout the paper, and are estimated using the number of sentences and \"then\"s in textual inputs. How do reference hops relate to how good a model is? \n\n\nTypos:\n\nAppendix D: principle -> principal\n\nAppendix C: dtextdimensional -> d-dimensional\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is well-addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "13"}, "extra_text": {}}, {"review_id": "_SQGvObFAgy", "reviewer": "Reviewer_tPsp", "summary": "\nThis paper is about a novel benchmark to evaluate interactive language-grounding models called Symbolic Interactive Language Grounding benchmark (SILG). It regroups five different grounded language learning environments, each with its own key challenges. To perform well on SILG, a model would need to generalize to new dynamics, entities, and partially observed worlds, understand rich natural language and perform multi-step reasoning. The authors designed a common interface for researchers to evaluate their models across all of these environments. In the paper, in addition to describing SILG, the authors propose the first architecture, Symbolic Interactive Reader (SIR), that can be used for all of these environments. Experiments were conducted on SILG to study how several recent advances in language-conditioned RL generalize over these environments (FiLM, recurrent state-tracking, large pretrained LMs, etc). Empirically, it was shown that most of the proposed components are only useful for the environment they were proposed for.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Overview\n\nThe proposed benchmark, SIGL, is a valuable contribution to the research community to track progress on interactive language-grounding. To the best of my knowledge, the proposed model, SIR, is novel and offers a good first baseline to compare against on this benchmark. The authors clearly described what are the key challenges and properties of each environment in SIGL. To me, this submission appears to be technically sound and each engineering decision is well motivated. The paper is easy to follow and well-written. I believe the paper provides enough information for me to build the model (Figure 2, Section 3, and Supplementary material) and try it on SIGL (once release). Overall, I agree there's a need for unifying environments used by the interactive language-grounding research community. Having standardized benchmarks help tracking research progress. I recommend this paper for acceptance.\n\n\n## What I like about this paper\n\n- The proposed benchmark unifies five language-grounding environments under a common interface. SIGL allows to quickly test new methods and see how much progress the community is making.\n- The ablation study on the different recent advances in language-conditioned RL and how useful they are for environments other than the one they were proposed for.\n- Table 1 about SIGL statistics showing the distinctive key properties for each environment.\n\n\n## Concerns\n\n- p.6 line 198: State-tracking does not help generalize to the evaluation distribution.\n\n- Fig.6 seems to be missing +bert and +all lines.\n\n- In ALFWORLD, \"the names of which we concatenate into a symbolic world observation grid\". I don't understand how the symbolic world observation grid looks like exactly.\n\n-----\n## Typos\n- p.4: line 109: \"is not complex, but [observation?] are 100 words on average\", missing word?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did discuss how they have adapted each environment to fit under a common API (e.g., convert text description to world representation). They also mentioned some issues human experts had with SymTD due to the symbolic representations discarding information such as color.\n\nThe authors have included a thorough impact statement in the appendix. They discussed the lack of interpretability that results from complex policies, especially those learned via RL (learning may result in unintuitive policies that achieve the object in surprising ways).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6D1-fse2QLn", "reviewer": "Reviewer_zwvr", "summary": "The paper provides a benchmark that wraps multiple interactive language grounding environments in a shared symbolic environment paving the way for agents that can benchmark on multiple different tasks at once. A further shared architecture baseline, the Symbolic Interactive Reader, is benchmarked on each of the individual environments and is further ablated with respect to its components (state tracking, entity centric convolutions+attention, and pretrained language models). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe SILG benchmark itself is easily usable in the form of Gym wrappers.\n\nSIR provides a unique baseline and an initial method for other researchers to build off of in creating such combined models.\n\nThe paper was clearly written and the choices for the architecture and all of the individual environments were justified. \n\n\nCons/Questions:\n\nIts not immediately clear to me why we want an architecture for NetHack to do well in ALFWorld. They are both trying to ground language in environment mechanics, yes, but the core issues faced are different. ALFWorld has much more of a focus on language (richer descriptions, text -> vision transfer, etc.) while NetHack is all about generalizing to unseen environment structures (a natural consequence of the procedural generation, without nearly as rich a language component). RTFM <-> NetHack or ALFWorld <-> Touchdown makes more immediate sense. A discussion regarding exactly what properties these individual environments are trying to foster in RL agents would appear (and are desirable) in a SILG environment agent would go a long way towards tying this paper together (expand on the Table 1 key challenges line perhaps).\n\nAs an RL benchmark, it would be really useful to have environment only benchmarks such as FPS and if any FPS is lost/gained over the underlying environments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no Broader Impacts section provided. A brief section on the biases of the language present in the individual environments could be added given that the environments are trying to learn grounded language from these environments with potential for other downstream tasks.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no immediately applicable ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Codifying benchmarks and bringing them under a single umbrella encourages better research. Particularly in an area like RL where results are often tenuous, depending on factors like random seeds. Larger and more comprehensive benchmarks ensure that methods really do improve on the state of the art.
 
While the idea is good, two major concerns came up during the discussion.
 
1. The work seems incomplete.
 
None of us could understand why the authors include only partial variants of existing benchmarks when creating SILG. This seems like a significant weakness that limits the SILG's contribution. The authors cite GLUE as their inspiration, but GLUE included other benchmarks wholesale; one didn't need to separately evaluate on them anymore because the final GLUE result already did so. By not including all of the games, as they were meant to exist, the authors sell their own approach short. We would love to see SILG published, with the full games included.
 
The authors put forward that they only use partial games because current methods can't deal with full games anyway and the resulting evaluation numbers would be too poor. Many methods exist for dealing with this, for example, in the retrieval community, methods usually don't retrieve the top result with perfect accuracy, and so we measure top-n (R@5, R@10, etc.) retrieval accuracy. The authors could do the same. Include the full games and then measure performance at 1% of the game, 10%, 20%, etc. This would make SILG a far more lasting contribution.
 
2. Essential details, that determine the technical validity of the benchmark and of the results, are completely missing.
 
Since the authors cite GLUE as their inspiration, it's worth having a second look at that paper. It very carefully defined what the inputs, outputs, and metrics are for every single benchmark rolled into GLUE. Unfortunately, almost all details to do with how models interact with the games are missing here.
 
Including such details is essential. They make the difference between a technically correct and a flawed manuscript. Imagine that the authors had crafted a mapping to one of the games in such a way that the game is now unplayable or far more difficult than it normally is. This is very easy to do.
 
As it stands, reviewers could not evaluate the technical correctness of the manuscript.
 
Even in terms of the novel conceptual contribution of the manuscript, it's really this mapping that we should be evaluating. It's the core of the method that brings the games together, and it’s missing.
 
One more minor point, the human experiment was generally seen as inadequate having only one subject.
 
I encourage the authors to continue with SILG despite this unfortunate news. The resulting benchmark and manuscript will be far more useful and impactful when fleshed out fully and when presented with enough detail to both reproduce the work and judge its technical correctness.
Decision: Accept (Poster)