SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark

Abstract: Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that re- quire interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan com- plexity. In addition, we propose the first shared model architecture for RL on these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-environment benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SILG enables the community to quickly identify new methodolo- gies for language grounding that generalize to a diverse set of environments and their associated challenges.


Reviews and author responses:

This paper introduces a unified, OpenAI Gym-based interface to five language-and-action grounding benchmarks. This interface is designed to make it easy for researchers to evaluate whether language grounding approaches work well across a variety of different benchmarks and tasks which are challenging in different ways, as past work has largely been focused on individual benchmarks. 

The benchmarks are RTFM, Messenger, NetHack, ALFWorld (all from previous work), and a new version, SymTD, of the existing Touchdown vision-and-language benchmark, which has replaced visual inputs with symbolic ones using semantic segmentations produced by a pre-trained vision model (a PSPNet trained on Cityscapes). 

The paper evaluates a baseline language-conditioned policy based on a strong model developed for RTFM (FILM^2), and also experiments with various architectural additions introduced by past work on other tasks (entity attention, recurrent state tracking, egocentric convolution, and BERT). Finally, the paper presents results on symbolic-to-visual transfer for ALFWorld -> Alfred and SymTD -> Touchdown.  *Strengths:*

1. I found the work well-motivated: a standardized interface to these tasks should make it easier for future work to evaluate on multiple tasks, and may lower the barrier for researchers who have worked on one task to work on another. I also appreciated the symbolic-to-visual transfer motivation, which is an exciting area for future work and may lower the barrier to entry for work on embodied visual grounding.

2. The baseline architecture seemed reasonable and strong, and the experiments with it are thorough. I also appreciated the paper's aim of exploring additional components introduced for these tasks, and it overall did an admirable job of this given the high computational cost of RL training on these environments (although see below for some concerns).

3. While the set of tasks is diverse, the paper did a good job overall of identifying common input and output formats and common abstractions across the tasks (although see below for some questions & concerns I had about the RL formulation of the SymTD / VisTD tasks in comparison to past work).

4. The paper collected expert human trajectories (from a single individual) to establish a rough performance upper bound.

*Weaknesses:*

1. For those benchmarks that have trajectory demonstrations available (e.g. Touchdown / VisTD), it wasn't clear to me whether the proposed framework will allow forms of imitation learning or whether it is limited to RL using e.g. graph distance reward (which may not directly match the gold trajectories -- as the appendix acknowledges -- and so might also not match the instructions exactly), or reward shaping based on trajectory supervision (which would need to be designed for each task). Imitation learning using trajectory supervision has been a standard source of supervision on tasks like Touchdown, and I suspect would be quite helpful to be able to explore a variety of model architectures within reasonable compute limitations, as IL requires less interaction with the environment than RL. 

2. Given the large set of embodied language grounding datasets (in addition to the ones this paper provides an interface to, there are also other influential benchmarks such as SAIL [MacMahon et al. 2006; Chen and Mooney 2011], CHAI & LANI [Misra et al. 2018], EQA [Das et al. 2018], BabyAI [Chevalier-Boisvert et al. 2018], Room-to-Room [Anderson et al. 2018], Room-Across-Room [Ku et al. 2020], and gSCAN [Ruis et al. 2020]), the paper should give a bit more justification about why these particular benchmarks and tasks were chosen, e.g. since they can be feasibly convertible into symbolic representations, or span the set of challenges in embodied grounding tasks in some way?

3. The Touchdown task (which I'm most familiar with of these tasks) is set up in a way that seems to make comparison to past work challenging, as past work most closely corresponds to the Manual condition (requiring the agent to predict when to stop) detailed in the appendix, which the RL setup in this paper appears to make very difficult (from Fig 8 and 9 in the appendix).

4. While I think that this framework will allow future work to address interesting questions about architecture generality across tasks like these, I feel that this paper itself made limited progress on answering these questions. Many of the components added to the baseline weren't fully explored (see below for some specific concerns), so that I don't feel the paper is justified in drawing conclusions like "most of these components are beneficial only to the environments they were designed for" (in the abstract, and several other places in the text).

*Comments on experimental results and analysis:*
1. It wasn't clear to me how to determine whether "reasonable performance across all environments" (L186) is being obtained. While the model outperforms SOTA on RTFM, it seems substantially worse than SOTA on Messenger and ALFWorld. It might help to say a bit more about what approaches these SOTA use and whether they're complementary to the approaches produced here (expanding on the Table 2 caption).
2. If possible, given the difference between the RL setup for Touchdown and the standard supervised learning / IL setup (Chen et al. 2018) it would help to contextualize the results here in comparison to past work, ideally by training a past SOTA architecture using the RL setup here or training the baseline here via IL.
3. The results on Messenger generally seem strange, as performance on train looks to be anti-correlated with performance on test. Since all methods (at least the ones explored here) seem to radically overfit, more should be said about what's needed to succeed in this environment.
4. If I'm interpreting the results correctly, it seems that BERT substantially hurts performance in all conditions it was tried in except in Messenger validation (and it does hurt in Messenger train; I don't think this can really be claimed to "help generalization in Messenger" given my concern above). Since BERT isn't fine-tuned it should provide strictly more information about the language than the baseline so it seems odd that it usually hurts training performance so much; I feel this should be investigated more before including these results in the paper as a negative result. 
5. Entity attention seems to also produce no benefits except in Messenger validation (and see the comment above). Some more explanation of what was tried here or analysis of why it isn't working should be done. It also wasn't evaluated on ALFWorld or SymTD.
6. 10-dimensional PCA features from the ResNet features on TouchDown seem to be quite small. Without results on VisTD that use the full ResNet features, I'm skeptical about the claim that "models trained on SymTD outperform those trained on VisTD" (L248).

*Questions:*
1. How conducive is the task and environment setup to training via some form of imitation learning with trajectory supervision, or how important do you think this is?
2. How feasible would it be to add other embodied grounded environments to this benchmark (I wouldn't expect this paper to do so necessarily, but for the sake of future work that builds on it)?

*Suggestions:*
- It wasn't totally clear from Figure 1 which of the text observations are instance-specific versus non instance-specific. It would also help to clarify what an instance is (a single trajectory? a single map?).
- It would be helpful to augment Table 1 and the text with indicators of which of the tasks involve language written by people versus language produced synthetically or by templates, as these lead to different challenges.
- The set of tasks in this paper is diverse (as Table 1 nicely lays out) -- although I don't necessarily expect the authors to take a stand on this, if they have expectations about whether a single approach should or could do well on all of them it would be helpful to say.
- It would help to include a few more details about the tasks to make the paper more self-contained:
    - The adversarial and non lexical overlap nature of Messenger was a bit confusing to me. For completeness it could help to give a short example of the generalization required.
    - How are the plausible commands in ALFWorld generated (line 113)?
    - Add references in the SymTD section to the appendix, which answers questions I had about how the segmentations are produced and the "passes the goal location" evaluation.
    - Give details on the oracle and masked-RCNN setup in ALFWorld.
- For researchers who are less familiar with the RL training setup (e.g. from the NLP community which more frequently uses imitation learning with trajectory supervision) it might help to clarify the setup in training: is the agent simultaneously evaluated on the train and validation environments for the purposes of early stopping; and test evaluation: is the policy is evaluated on the test environments without learning (i.e. the model parameters are frozen)?
- Given the paper's focus on Touchdown, it would make the related work more complete to also cite the original VLN task paper by Anderson et al, which introduced the Room-to-Room (R2R) dataset. Given the large body of work on R2R, I feel the paper should also give some justification for why R2R wasn't included in its suite of tasks (I'm not necessarily saying that the paper should include it, as Touchdown presents a separate set of challenges and recent work has moved past R2R to other datasets like Room-4-Room and Room-Across-Room, but it would help to further motivate this choice.)

*Model suggestions:*
- If I understand right, attending over N_i or C_i conditioned on the recurrent state could allow the agent to focus on portions of the text relevant to the history of actions it's completed.
- It wasn't totally clear to me (either from the main text or D.2 in the appendix) whether the adjacent nodes are encoded relative to the agent's current heading, which past VLN work has found to be helpful (see the "panoramic representation" of Fried et al. 2018, "Speaker-Follower Models for Vision-and-Language Navigation). 
- It might help to try other methods of combining the fixed BERT embeddings with the original LSTM encodings rather than adding, e.g. concatenation.

*Minor comments:*
- line 300: "across on" -> "across"
- A.547: "principle" -> "principal"

=== update after response ===

Thanks to the authors for the very thorough response! It largely addressed my concerns (more details below), and I also agree with the positive points the other reviewers pointed out. I've raised my score to a 7 (from a 6).

In my initial review I was concerned about:

- *Doesn't allow imitation learning*: The authors plan to in a future release (points 1 and 10).
- *Leaves out embodied / visual domains*: While I still do feel (in agreement with some of the other reviewers) that a symbolic benchmark leaves a lot out, I was mostly convinced by the author response about the range of challenges here even without vision and embodiment. It seems that there's a lot of progress remaining to be made that this benchmark could encourage (point 2).
- *Comparison on touchdown is hard given the setup*: This isn't really resolved in the current work, but since the work doesn't make touchdown-specific claims, I think this is ok (point 3 and 5).
- *Limited progress on comparing architectures*: Their explanations for the results (points 4, 6, 7, 8) were convincing to me. I do still feel that the claim about "most of these components are beneficial only to the environments they were designed for" would be too broad if taken as a claim about these methods (based on the results here), but if taken as motivation for the benchmark or tempered a bit (and the authors seem willing to), it's fine.

Re: Messenger performance, I was convinced by the author response explaining the results on it. Messenger seems quite different as a task than the others since it requires such strong generalization, but I can see the benefit of including a compositional generalization test in this benchmark as it will likely encourage more work in this area.

I also agreed that the paper was pretty clearly written as is, and the clarification points I had were all resolved in the author response (points 12-20 and 22). Please see the weaknesses above for suggestions on how to address what I see as the limitations in the framework (potential lack of ability to include imitation learning supervision, choice of benchmarks) and experiments (limited exploration of some of the additional components over the baseline).

This paper proposes a multi-environment language and RL benchmark called Symbolic Interactive Language Grounding benchmark (SILG) which unifies a collection of diverse grounded language learning environments under a common interface. The environments included span language-guided grid-world environments (RTFM, Messenger, NetHack), and visual worlds (ALFWorld, Touchdown). The authors also propose a new architecture meant to comprehensively evaluate several innovations in this space on their new benchmark, and find that many innovations are environment-specific and do not transfer to new environments. These findings motivate a need for such a benchmark to standardize model evaluation and progress in this field.  Strengths:
1. This dataset integrates several important datasets in language and RL and I believe it will serve as a good basis for benchmarking future methods.
2. The paper makes several interesting findings that innovations are environment-specific and do not transfer to new environments. This benchmark will be important to standardize model evaluation and progress in this field and I think it comes at a good time.
3. The paper is well-written and well motivated. Experiments are clear and easy to read. As a researcher in this research area, I learned a lot from this paper.

Weaknesses:
1. I did not see a mention of any public release of this benchmark and standardized evaluation protocol. I am assuming that this will be publicly released for the community unless there is reason not to do so.
2. Is there any reason to believe that a single model is suitable for all these environments? Given that the text, visual, and RL challenges are all quite different across these datasets, this might indeed cause certain design decisions to work better for one environment and not for another.
3. There are many multimodal methods for language and visual grounding that I believe this paper should also cover, beyond the simple attention-based models considered in this paper. It would be better to compare more closely to works such as "A survey of reinforcement learning informed by natural language" and be more thorough in the comparisons of proposed approaches in that paper as well.
4. Why were these 5 environments chosen in particular? And why were only 'egocentric local convolution, recurrent state-tracking, entity-centric attention' classified as recent advances? It would be good to describe a more systematic system for choosing and expanding these environments and models as this benchmark is used. It would be even better to organize an open call for environments and models so that the research community can expand on both of these.

Minor:
1. It would be good to add references to recent works in multimodal benchmarks in different domains such as:

https://arxiv.org/abs/2107.07502

https://aclanthology.org/D15-1021.pdf

https://link.springer.com/chapter/10.1007/978-3-030-49724-8_2

https://github.com/georgian-io/Multimodal-Toolkit

not a benchmark, but a good unifying algorithm of several approaches: https://openreview.net/pdf?id=rylnK6VtDH


=== post rebuttal ===
I have raised my score to a 7. I am glad that the authors will be releasing the benchmark publicly, and the authors have also alleviated some of my concerns regarding the choices of methods, techniques, and environments used in the benchmark, and I am glad they plan to expand it in the future. The authors have done a good job illustrating the broad impacts of their work.

 *10. How conducive is the task and environment setup to training via some form of imitation learning with trajectory supervision, or how important do you think this is?*

For all tasks except Touchdown and ALFWorld, there are no gold trajectories available which would make an IL setup difficult to implement. For these tasks, effective models trained using RL exist. For ALFWorld and SymTD with auto-stopping, the RL approach learns a non-trivial policy. We are considering supporting imitation learning in SILG, however this makes the interface much more complex (we'll likely skip TorchBeast and rely on a different setup). Consequently we did not include IL in this initial version. In practice, we think it is definitely helpful to do IL if the supervised trajectories are available (i.e. it should be a supported option).

*11. How feasible would it be to add other embodied grounded environments to this benchmark (I wouldn't expect this paper to do so necessarily, but for the sake of future work that builds on it)?*

It is easy to extend SILG; since we are using gym, the benchmark supports the registration of additional environments.

*12. It wasn't totally clear from Figure 1 which of the text observations are instance-specific versus non instance-specific. It would also help to clarify what an instance is (a single trajectory? a single map?).*

We’ll clarify this in the manuscript. An instance is a single trajectory. 

*13. It would be helpful to augment Table 1 and the text with indicators of which of the tasks involve language written by people versus language produced synthetically or by templates, as these lead to different challenges.*

We will include these clarifications in our revisions.

*14. The set of tasks in this paper is diverse (as Table 1 nicely lays out) -- although I don't necessarily expect the authors to take a stand on this, if they have expectations about whether a single approach should or could do well on all of them it would be helpful to say.*

SIR suggests that with additional improvements, it may be possible to have a performant model with the same architecture (but trained independently) across environments. It is unclear whether a single model with the same parameters can accomplish all tasks, or whether a single model with pretraining can be quickly finetuned on each task. However, we believe SILG is well-suited to help answer these questions in the future.

*15. It would help to include a few more details about the tasks to make the paper more self-contained:
The adversarial and non lexical overlap nature of Messenger was a bit confusing to me. For completeness it could help to give a short example of the generalization required.*

See above comment on adversarial generalization. In RTFM the entities and their text references are identical, (e.g. ‘dog’ in manual is the same string ‘dog’ in the observation). In Messenger, symbols in the manual and observation do not overlap. We will provide some concrete examples in our revisions. 

*16. How are the plausible commands in ALFWorld generated (line 113)?*

They are generated from a grammar that accounts for the objects in the scene by the ALFWorld engine.

*17. Add references in the SymTD section to the appendix, which answers questions I had about how the segmentations are produced and the "passes the goal location" evaluation.*

We will add the appropriate appendix references to SymTD as you have suggested. 

*18. Give details on the oracle and masked-RCNN setup in ALFWorld.*

The masked RCNN and oracle are used by the BUTLER agent to transfer from ALFWorld to ALFRED, where the masked RCNN detects objects in the scene. These objects populate a template to form the observation text given to the agent. An oracle A* planner executes the text action produced by BUTLER in ALFRED. We will add these clarifications in our revisions.

*19. For researchers who are less familiar with the RL training setup (e.g. from the NLP community which more frequently uses imitation learning with trajectory supervision) it might help to clarify the setup in training: is the agent simultaneously evaluated on the train and validation environments for the purposes of early stopping; and test evaluation: is the policy is evaluated on the test environments without learning (i.e. the model parameters are frozen)?*

We do not use the validation for early stopping, and the policy is evaluated on test evaluation with all parameters frozen. We will provide these clarifications in our revisions.

*20. Given the paper's focus on Touchdown, it would make the related work more complete to also cite the original VLN task paper by Anderson et al, which introduced the Room-to-Room (R2R) dataset. Given the large body of work on R2R, I feel the paper should also give some justification for why R2R wasn't included in its suite of tasks (I'm not necessarily saying that the paper should include it, as Touchdown presents a separate set of challenges and recent work has moved past R2R to other datasets like Room-4-Room and Room-Across-Room, but it would help to further motivate this choice.)*

We will include the suggested reference in our revision. To make the benchmark easier to use and more self-contained, we wanted to avoid using Matterport3D which all the RXR tasks depend on.

*21. If I understand right, attending over N_i or C_i conditioned on the recurrent state could allow the agent to focus on portions of the text relevant to the history of actions it's completed. It might help to try other methods of combining the fixed BERT embeddings with the original LSTM encodings rather than adding, e.g. concatenation.*

Thank you for these ideas, we’ll take these into consideration in further experiments.

*22. It wasn't totally clear to me (either from the main text or D.2 in the appendix) whether the adjacent nodes are encoded relative to the agent's current heading, which past VLN work has found to be helpful (see the "panoramic representation" of Fried et al. 2018, "Speaker-Follower Models for Vision-and-Language Navigation).*

We use relative position encodings as a feature, which is then concatenated with the visual representations to give the agent an idea of where visual features are relative to the agent’s position.


 We thank the reviewer for their detailed comments - these are extremely helpful and thoughtful and will undoubted improve our paper. We are glad that you found our benchmark and ablations useful, and that our experiments were well-motivated and informative. We are also excited about the potential for SILG as a language-conditioned RL testbed for unified architectures. Please find our detailed response below.

*1. It wasn't clear to me whether the proposed framework will allow forms of imitation learning or whether it is limited to RL using e.g. graph distance reward (which may not directly match the gold trajectories*.

We plan on supporting IL in a future release. Please see 10 for more discussion.

*2. The paper should give a bit more justification about why these particular benchmarks and tasks were chosen, e.g. since they can be feasibly convertible into symbolic representations, or span the set of challenges in embodied grounding tasks in some way?*

We wanted to select environments that spanned the challenges presented in table 1, could be easily converted to symbolic representations, and avoided the use of additional simulators (e.g. Matterport 3D). Embodied language grounding is clearly a difficult and worthwhile area that we need to make progress in. It poses important though different challenges than SILG (e.g. more complex controls, richer perception, more limited domains+language). Hence, our work is complementary to embodied language grounding. We will add a section on embodied language grounding datasets in the related works in the manuscript.

*3. The Touchdown task (which I'm most familiar with of these tasks) is set up in a way that seems to make comparison to past work challenging, as past work most closely corresponds to the Manual condition (requiring the agent to predict when to stop) detailed in the appendix, which the RL setup in this paper appears to make very difficult (from Fig 8 and 9 in the appendix).*

The Touchdown task with the Manual stop condition is included in SILG, and although win rates are currently low, future work may improve these results. In the meantime, the auto-stop variant provides an intermediate evaluation. We considered potentially including imitation learning TD, however it would greatly complicate SILG (e.g. TD and ALFWorld would have distinct training flows). As a result, we chose to limit SILG to RL such that the interface is more intuitive and simpler to use.

*4. It wasn't clear to me how to determine whether "reasonable performance across all environments" (L186) is being obtained. While the model outperforms SOTA on RTFM, it seems substantially worse than SOTA on Messenger and ALFWorld. It might help to say a bit more about what approaches these SOTA use and whether they're complementary to the approaches produced here (expanding on the Table 2 caption).*

While we borrow the key idea of entity-attention from the SOTA EMMA model on Messenger, the shared downstream components of SIR are different from EMMA. One possibility is that the complexity in SIR allows learning on other environments but leads to overfitting in Messenger, which has a small set of training dynamics that is prone to overfitting. For ALFWorld, the SOTA result comes from imitation learning on human trajectories, which we do not use. The authors of ALFWorld shared preliminary RL results with us, which SIR actually outperforms, however this result is not public and hence we do not have reference for it (yet). In summary, the only significantly worse than SOTA performance is on Messenger, which we suspect comes from overfitting the smaller-than-others training distribution.

*5. If possible, given the difference between the RL setup for Touchdown and the standard supervised learning / IL setup (Chen et al. 2018) it would help to contextualize the results here in comparison to past work, ideally by training a past SOTA architecture using the RL setup here or training the baseline here via IL.*

We will add a section discussion the distinction between IL and RL for Touchdown and ALFWorld.

*6. The results on Messenger generally seem strange, as performance on train looks to be anti-correlated with performance on test. Since all methods (at least the ones explored here) seem to radically overfit, more should be said about what's needed to succeed in this environment.*

In Messenger, if an entity (e.g. dog) has a certain role (e.g. message) in training, you will never see the dog as a message in validation or test. Thus, it is easy for models to overfit to entity-role combinations (e.g. dog as the message). To succeed, Messenger requires compositional entity-role generalizations. 

*7. If I'm interpreting the results correctly, it seems that BERT substantially hurts performance in all conditions it was tried in except in Messenger validation (and it does hurt in Messenger train; I don't think this can really be claimed to "help generalization in Messenger" given my concern above). Since BERT isn't fine-tuned it should provide strictly more information about the language than the baseline so it seems odd that it usually hurts training performance so much; I feel this should be investigated more before including these results in the paper as a negative result.*

For many of these tasks (such as NetHack and RTFM), the language is very specific to a setting (e.g. fantasy with words like imp, goblin, shaman). It is possible that by passing these entity words through BERT, they become more similar than randomly initialized vectors, making the important step of distinguishing them more difficult. The entities in Messenger are more general (dog, plane, robot), and also feature numerous synonyms for each entity, which may be more conducive to the use of BERT. We agree that the BERT generalization results for Messenger is not conclusive. We will modify our paper so that we clarify that the effectiveness of using pretrained language models warrants further investigation.

*8. Entity attention seems to also produce no benefits except in Messenger validation (and see the comment above). Some more explanation of what was tried here or analysis of why it isn't working should be done. It also wasn't evaluated on ALFWorld or SymTD.*

In entity-attention, the entity representation is built entirely using the text. When there is incomplete entity information (e.g. NetHack) or it is difficult to extract the relevant information from the manual (e.g. RTFM) this can be a handicap. Since the entity-attention is an attention over text-fields and SymTD has only one, we cannot use entity-attention in SymTD. In ALFWorld, the observation is a single string and so the concept of entities is not well-defined.

*9. 10-dimensional PCA features from the ResNet features on TouchDown seem to be quite small. Without results on VisTD that use the full ResNet features, I'm skeptical about the claim that "models trained on SymTD outperform those trained on VisTD" (L248).*

The baseline used in the original Touchdown paper uses 1-dimensional features (an average of the ResNet features). Because of the large number of panoramas, adding significantly more features would make it difficult to fit into memory with multiple actors. In the limit of compute budget, we expect VisTD performance to exceed SymTD due to the former having more complete observations.

 We thank the reviewer for their detailed comments. We are glad you found our benchmark and ablations useful, and that our experiments were well-motivated and informative. We are also excited about the potential for SILG as a language-conditioned RL testbed for unified architectures. Please find our detailed response below.

*1. The current evaluation protocol doesn't provide clues on what models need to possess in general in order to solve all tasks in SILG (e.g. for all the (+) modules, which one is generally the best across sub-environments).*

We will include a discussion on the model feature (+state) that provides the most benefit across environments.

*2. A general evaluation score (e.g. average success rate over all environments) will help illustrate things needed for models better.*

Thank you for the suggestion, we will incorporate a unified score in our final benchmark

*3. As a constructive suggestion for building this general test-bed, it will also be good to see how we can design our models, or over what tasks can we pre-train on so that a model can be quickly adapted into all these sub-environments, for example in a more challenging few-shot setting. This is similar to the authors' intuition of leveraging (+bert), but since the current model is trained from scratch in each environment, the current evaluation is in fact showing models' ultimate potential for each task instead of a generalization capability across tasks. Compared to (1), this is an additional step to discover what ingredients are needed and how these ingredients could be obtained from other domains.*

Thank you for the suggestion, we agree that developing models that can leverage some form of pretraining effectively will be important and we believe SILG will help facilitate this kind of research.

*4. The authors might want to clarify some details in the experiment section. There exists early stopping of different model-design throughout the experiment section in training (Fig.1 - Fig.7), which is not discussed by the authors in the manuscript. This leads to questions on the validity of the reported results since in some experiments (e.g. Fig.6, Fig.7), the training clearly hasn't converged and steps are different from model to model (e.g. +all vs. base). Further descriptions or illustrations will be helpful for justifying these results.*

We will clarify this in the manuscript, including a more extended plot of Fig 6 and 7.


 We thank the reviewer for their detailed comments. We are glad that you found our benchmark neat and convenient, our ablations useful, and our choice of environments justified. We are also excited about the potential for SILG as a language-conditioned RL testbed for unified architectures. Please find our detailed response below.

*1. Would have appreciated more clarity at some places: The types of tasks in each environment would be more clear if Figure 1 could be changed/appended with more information about the examples shown.*

Thank you for the suggestion, we will make these modifications in our revisions.

*2. I wasn't convinced about the justification given for using a purely symbolic setup in the benchmark. The authors provide a comparison to learning from visual observations and say that their benchmark allows "perception challenges to be factored out" and allows one to focus on quickly iterating on abstraction challenges. One of the most interesting challenges in grounded language learning is the interaction of language learning with learning to perceive. By factoring out perception, the benchmark does not allow perception to be influenced by the language. The benchmark would have been more comprehensive if visual perception was an integral part of it.*

We agree that visual perception is an important problem that warrants additional study in a SILG-like multi-environment evaluation benchmark. However, for our first iteration of SILG, we wanted to focus on the unique challenges presented by existing symbolic environments such as multi-hop reasoning and generalization to rich sets of procedurally generated dynamics. We agree that future versions of SILG should also investigate the use of environments with perception challenges.

*3. A related citation that is missing is GSCAN [1].*

Thank you for the suggested reference, we will include discussions of GSCAN in our revised manuscript.


 We thank the reviewer for their detailed comments. We are glad that you found our benchmark and ablations useful, and that our experiments were well-motivated and informative. We are also excited about the potential for SILG as a language-conditioned RL testbed for unified architectures. Please find our detailed response below.

*1. I did not see a mention of any public release of this benchmark and standardized evaluation protocol. I am assuming that this will be publicly released for the community unless there is reason not to do so.*

The benchmark will be released publicly, and we are currently deciding on the best way to introduce it (e.g. online leaderboard, github repo, etc.).

*2. Is there any reason to believe that a single model is suitable for all these environments? Given that the text, visual, and RL challenges are all quite different across these datasets, this might indeed cause certain design decisions to work better for one environment and not for another.*

There is no reason to believe that a unified model will outperform specialized models engineered to perform well on specific environments. However, it can be helpful to understand whether a particular modelling innovation is environment specific, or a more general technique that can scale to other language grounding tasks. SILG allows researchers to easily answer these questions by testing their methods across 5 unique environments.
Furthermore, we believe that SILG is a step towards developing more general language grounding advancements that are not environment-specific, which we believe is also an important future area of study.

*3. There are many multimodal methods for language and visual grounding that I believe this paper should also cover, beyond the simple attention-based models considered in this paper. It would be better to compare more closely to works such as "A survey of reinforcement learning informed by natural language" and be more thorough in the comparisons of proposed approaches in that paper as well.*

Thank you for the suggested reference, we will draw more comparisons to such work in our revisions.

*4. Why were these 5 environments chosen in particular? And why were only 'egocentric local convolution, recurrent state-tracking, entity-centric attention' classified as recent advances? It would be good to describe a more systematic system for choosing and expanding these environments and models as this benchmark is used. It would be even better to organize an open call for environments and models so that the research community can expand on both of these.*

We chose these environments because they include a diverse set of distinct challenges (detailed in table 1), while satisfying certain criteria (e.g. symbolic, interactive). We subsequently chose these modelling enhancements because they were proposed for (and resulted in key gains on) the environments included in this benchmark. We wanted to answer the question of how useful these enhancements are for environments other than those the addition was developed for. Thank you for suggesting the open call for environments, SILG is easily extensible and we look forward to incorporating additional environments in future releases.

*5. It would be good to add references to recent works in multimodal benchmarks in different domains such as…*

Thank you for the suggestions, we will include these references in our revised version.


This paper proposes a benchmark for symbolic interactive language grounding that integrates 5 existing environments into a unified gym-like setting. The authors converted tasks in different environments to unified action/state space, task descriptions, within-trial information, etc., and designed a baseline model that performs over all the benchmarks. The authors also provide various adjustments for the baseline model considering the different generalization tasks within each environment. The overall evaluation shows hints on potential directions on symbolic interactive language grounding tasks.  The SILG proposed in this paper integrates 5 existing symbolic generalization environments and serves as a unified benchmark for evaluating models' generalization capabilities over new dynamics, new layouts, new instr, etc. This benchmark has the potential to become the general testbed for symbolic interactive language grounding and could be beneficial for the field.

The paper is overall clearly written with detailed descriptions for model design. The provided possible adjustments to the backbone model summarize previous works' ideas on tackling each specific kind of generalization in different sub-environments. The analysis of the results makes sense and smoothly leads to the general challenges for modern interactive symbolic grounding tasks summarized in (L.255-259).

On the other hand, there are some concerns for the current benchmark which I hope the authors can address:

(1) Model's are separately trained on different sub-environments and evaluated separately as well. As a general test-bed, it should not stop at integrating all environments into one unified framework and model design. The current evaluation protocol doesn't provide clues on what models need to possess in general in order to solve all tasks in SILG (e.g. for all the (+) modules, which one is generally the best across sub-environments). A general evaluation score (e.g. average success rate over all environments) will help illustrate things needed for models better.

(2) As a constructive suggestion for building this general test-bed, it will also be good to see how we can design our models, or over what tasks can we pre-train on so that a model can be quickly adapted into all these sub-environments, for example in a more challenging few-shot setting. This is similar to the authors' intuition of leveraging (+bert), but since the current model is trained from scratch in each environment, the current evaluation is in fact showing models' ultimate potential for each task instead of a generalization capability across tasks. Compared to (1), this is an additional step to discover what ingredients are needed and how these ingredients could be obtained from other domains.

(2) The authors might want to clarify some details in the experiment section. There exists early stopping of different model-design throughout the experiment section in training (Fig.1 - Fig.7), which is not discussed by the authors in the manuscript. This leads to questions on the validity of the reported results since in some experiments (e.g. Fig.6, Fig.7), the training clearly hasn't converged and steps are different from model to model (e.g. +all vs. base).  Further descriptions or illustrations will be helpful for justifying these results.

 Please check the previous sections for concerns and constructive feedback.

The authors propose a new interactive language grounding benchmark by combining five different language grounding environments that pose different types of challenges; RTFM for multi-step multi-stage reasoning, Messenger for systematic generalisation in making entity-role assignments, Nethack for learning in partially observable environments with large state spaces, ALFWORLD for learning in environments with a large action space and Symbolic Touchdown tests for spatial reasoning. The authors also propose a new baseline model that combines recent advances in language grounding models and test it on their benchmark with different ablation tests. They use these results to analyse the challenges posed by the different environments to current methods while also assessing the benefits of the different methods used in the baseline model.  ### Pros:

- The paper proposes a new benchmark by combining existing environments in a common framework (similar to benchmarks like GLUE and SUPER GLUE, which are used widely for evaluating NLI models).
- I found the OpenAI Gym-like framework to be quite neat and convenient to use.
- The analysis of the various features of the baseline model by using ablations,  by conducting comparisons with human participants and then discussing the results made the benefits provided by different techniques quite clear.
- Similarly, the analysis of the challenges of solving the different environments gave a quantitative justification to the author's choice of environments.
- Overall, the proposed SILG benchmark provides an easy way to test for and iterate over agents that interactively learn language.

### Concerns /Suggestions:

- The paper was quite clear and easy to understand. However, I would have appreciated more clarity at some places: The types of tasks in each environment would be more clear if Figure 1 could be changed/appended with more information about the examples shown.
- I wasn't convinced about the justification given for using a purely symbolic setup in the benchmark. The authors provide a comparison to learning from visual observations and say that their benchmark allows "perception challenges to be factored out" and allows one to focus on quickly iterating on abstraction challenges. One of the most interesting challenges in grounded language learning is the interaction of language learning with learning to perceive. By factoring out perception, the benchmark does not allow perception to be influenced by the language. 
The benchmark would have been more comprehensive if visual perception was an integral part of it.

- The baseline models seem to struggle with systematic generalization in tasks like Messenger and Sym-touchdown. A related citation that is missing is GSCAN [1].

References:
[1] Ruis, L., Andreas, J., Baroni, M. Bouchacourt, D., and Lake, B. M. (2020). A Benchmark for Systematic Generalization in Grounded Language Understanding. Advances in Neural Information Processing Systems 33. The discussion of the limitations and societal impact by the authors was adequate.

Meta-review: Contributing a unified API and benchmark for language grounding in symbolically represented environments can potentially be valuable for the community. Such a benchmark can focus efforts, spark discourse about appropriate evaluation, and avoid bespoke, over-engineered models from dominating individual tasks and obscuring more general solutions. Reviewers all advocate for acceptance and the AC agrees. Authors are encouraged to revise the paper to address reviewer concerns -- especially to clarify how/why these environments were chosen (both these specific ones and why perceptual environments were excluded). Further, the details on Messenger's performance included in the response would be useful for future readers as well.

Decision: accept