Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Because existing language grounding work operates in single environments, it is difficult to determine whether model contributions apply across diverse settings. To remedy this, the authors present the Symbolic Interactive Language Grounding Benchmark (SILG), an environment simplifying and unifying five existing language grounding environments: RTFM, Messenger, NetHack, ALFWorld, and SymTD. The authors focus on symbolic environments over visual environments to abstract out the component of visual perception.
In addition, the authors provide a baseline model for SILG, Symbolic Interactive Reader (SIR). SIR's core architecture is compatible with all environments in SILG. They test SIR and several variants thereof containing recent modelling contributions, finding that while SIR achieves decent performance on the benchmark, it falls short of human expert playthroughs, suggesting room for future work. In addition, they find that most contributions except for recurrent state tracking are beneficial in only the environment for which they were originally developed. 
main_review: Contributions:
1. Introducing a multi-environment symbolic language grounding benchmark, SILG that combines five language-grounding environments under the same interface.

While the authors make a good argument for training/testing models on multiple, diverse environments instead of single environments, the five complex existing environments are significantly simplified. The value added for combining these existing benchmarks under one umbrella is unclear.


2. Providing a baseline model for this benchmark (SIR).

SIR consists of a FiLM^2 model which is provided with different types of inputs. The authors should simply say they use FiLM^2 as the baseline model. 


3. Results of variants of FiLM^2 (termed SIR) on these benchmarks.

It’s unclear what insights are gained from these results which would lead to better models. This is underscored by the fact that it’s hard to compare performance across such different benchmarks on different models.


===============================================================================

High-level review

i. Testing on multiple, diverse environments to analyze generalization of RL algorithms is an existing paradigm. It is not clear why we want to combine simplified versions of five existing environments into one benchmark. It’s also detrimental to keep the original names like (NetHack) for the simplified version; this will only sow confusion.

ii. Generalization is the main focus, but different models are used for different experiments and no cross-environment generalization is presented. This weakens the results significantly.

iii. Fundamental details are missing, such as what the dynamic text is for different games, making the results impossible to reproduce.

iv. A core result of the paper suggests that "algorithm X works on environment A but not environment B". This is misleading, as models are not comparable across environments (model size, environment size, hyperparameters).


More in the detailed review.

===============================================================================

Testing the generalization of existing algorithms to other environments is an interesting problem. However, in general, the method, model selection, and evaluation need to be better described and justified. There are several major issues with the manuscript as it stands, summarized here from the detailed review below with references to the detailed review:

- (1, 2, 12) The authors should discuss how to interpret the analysis of different model enhancements. It's not clear that different models, environments, and tasks are comparable, especially when the base models have different hyperparameters for different environments, and when models are trained on single environments. This reduces the impact of combining multiple environments into one benchmark. If the result is that models are trained/tested on single environments, it is not clear that the authors have demonstrated significant progress towards solving the motivating problem.

- (3, 5, 6, 11) As it stands, the work is not reproducible from reading the paper. The authors should report methodology with more detail, including environment simplification and model selection.

- (4, 8, 10) Reported results (Tables 2, 3) are difficult to interpret and lacking some information. 

I would vote for rejecting the publication based on the comments above. 

========================================================================================

Detailed comments:

Major:

1) Why can we compare the algorithmic contributions across different environments? There is no relationship between the environments, and the models are different for each environment. This makes results very difficult to interpret. 

2) The evaluation metric in comparing the performance of recent modelling contributions is not stated (it would help to add a couple words on the evaluation metric at the beginning of Section 4.1). I believe the evaluation metric is an algorithm's performance improvement over the base.

3) (Appendix G) How are model hyperparameters selected? Are learning hyperparameters the same for each environment? Tuning hyperparameters for each environment separately results in a different model per environment. 

4) Standard deviations in Tables 2, 3 are quite large and overlapping, which makes it hard to differentiate Base and Best. 

5)  Pg. 3 (NetHack): There are 23 fixed actions (Table 1). However, in the NetHack paper, there are 93 possible actions that players can take. How is the subset of actions chosen? In the expert playthroughs, do they also only access the 23 same fixed actions?

6) Pg. 3 (NetHack) "…in our expert playthroughs of SILG NetHack, we observe just over 100 unique words." How is this number relevant to the agent? How does the agent name objects? 

7) Lines 30-32, Figure 1, lines 57-58 state that SILG contains five language-grounding environments (RFTM, Messenger, NetHack, ALFWorld, SymTD). However, SILG does not contain RFTM nor NetHack, but rather one level of RFTM and three NetHack tasks. Like ALFWorld and SymTD, it is important to avoid stating that SILG contains RFTM and NetHack. This could be remedied by giving the simplified RFTM and NetHack versions a different name and stating their relationship with the original RFTM, NetHack environments. 

8) Please include the standard deviation for SOTA in Table 2 so that Base, Best, and SOTA are easier to compare. 

9) "Table 3 shows that policies learned in the symbolic environment transfer to the 3-D environment."-- For sake of comparison, the SOTAs on ALFRED and VisTD should be included. 

10) Though one of the main conclusions of the paper is that recurrent state-tracking is beneficial across environments (including SymTD), it does not result in the best transfer results to VisTD. If transfer to VisTD is the main practical usage of training with SymTD, then finding that recurrent state-tracking is best in SymTD may not be relevant.

11) A key challenge in reproducibility is that we don't know what the dynamics text and non-dynamics text are for each environment. It would be helpful to state for each environment where these texts are generated or found in instruction manuals, how they're processed, etc. 

12) Models differ per environment (thus incomparable) and are trained/tested on single environments. Training/testing on single environments was a motivation for the work. Moreover, environments are simplified versions of the originals. Then, how does using SILG improve from training individually on the original environments?

Minor:

14) How was the experimental subject recruited? If possible, it would be better to have more participants.

15) "This may be because it helps prevent local minima that cause repetitive actions". I would look into pg. 2 in the Resnet paper (https://arxiv.org/pdf/1512.03385.pdf) for a theoretical explanation for why H' may converge faster than $LSTM(H, S_{t-1})$. 

16) "Dynamics" and "non-dynamics text" is confusing terminology. It may help to italicize these to help the reader view them as being defined in lines 66-67. 

17) Line 160: how do we get 4 in the subscript?

18) Equations 5, 6: Perhaps replace the C, A in the subscripts or remove the subscripts? As they were being defined on the left and reused in the subscript.

19) Reference hops are mentioned several times throughout the paper, and are estimated using the number of sentences and "then"s in textual inputs. How do reference hops relate to how good a model is? 


Typos:

Appendix D: principle -> principal

Appendix C: dtextdimensional -> d-dimensional

limitations_and_societal_impact: Societal impact is well-addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides a benchmark that wraps multiple interactive language grounding environments in a shared symbolic environment paving the way for agents that can benchmark on multiple different tasks at once. A further shared architecture baseline, the Symbolic Interactive Reader, is benchmarked on each of the individual environments and is further ablated with respect to its components (state tracking, entity centric convolutions+attention, and pretrained language models). 
main_review: Pros:

The SILG benchmark itself is easily usable in the form of Gym wrappers.

SIR provides a unique baseline and an initial method for other researchers to build off of in creating such combined models.

The paper was clearly written and the choices for the architecture and all of the individual environments were justified. 


Cons/Questions:

Its not immediately clear to me why we want an architecture for NetHack to do well in ALFWorld. They are both trying to ground language in environment mechanics, yes, but the core issues faced are different. ALFWorld has much more of a focus on language (richer descriptions, text -> vision transfer, etc.) while NetHack is all about generalizing to unseen environment structures (a natural consequence of the procedural generation, without nearly as rich a language component). RTFM <-> NetHack or ALFWorld <-> Touchdown makes more immediate sense. A discussion regarding exactly what properties these individual environments are trying to foster in RL agents would appear (and are desirable) in a SILG environment agent would go a long way towards tying this paper together (expand on the Table 1 key challenges line perhaps).

As an RL benchmark, it would be really useful to have environment only benchmarks such as FPS and if any FPS is lost/gained over the underlying environments.
limitations_and_societal_impact: There is no Broader Impacts section provided. A brief section on the biases of the language present in the individual environments could be added given that the environments are trying to learn grounded language from these environments with potential for other downstream tasks.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 
This paper is about a novel benchmark to evaluate interactive language-grounding models called Symbolic Interactive Language Grounding benchmark (SILG). It regroups five different grounded language learning environments, each with its own key challenges. To perform well on SILG, a model would need to generalize to new dynamics, entities, and partially observed worlds, understand rich natural language and perform multi-step reasoning. The authors designed a common interface for researchers to evaluate their models across all of these environments. In the paper, in addition to describing SILG, the authors propose the first architecture, Symbolic Interactive Reader (SIR), that can be used for all of these environments. Experiments were conducted on SILG to study how several recent advances in language-conditioned RL generalize over these environments (FiLM, recurrent state-tracking, large pretrained LMs, etc). Empirically, it was shown that most of the proposed components are only useful for the environment they were proposed for.
main_review: 
## Overview

The proposed benchmark, SIGL, is a valuable contribution to the research community to track progress on interactive language-grounding. To the best of my knowledge, the proposed model, SIR, is novel and offers a good first baseline to compare against on this benchmark. The authors clearly described what are the key challenges and properties of each environment in SIGL. To me, this submission appears to be technically sound and each engineering decision is well motivated. The paper is easy to follow and well-written. I believe the paper provides enough information for me to build the model (Figure 2, Section 3, and Supplementary material) and try it on SIGL (once release). Overall, I agree there's a need for unifying environments used by the interactive language-grounding research community. Having standardized benchmarks help tracking research progress. I recommend this paper for acceptance.


## What I like about this paper

- The proposed benchmark unifies five language-grounding environments under a common interface. SIGL allows to quickly test new methods and see how much progress the community is making.
- The ablation study on the different recent advances in language-conditioned RL and how useful they are for environments other than the one they were proposed for.
- Table 1 about SIGL statistics showing the distinctive key properties for each environment.


## Concerns

- p.6 line 198: State-tracking does not help generalize to the evaluation distribution.

- Fig.6 seems to be missing +bert and +all lines.

- In ALFWORLD, "the names of which we concatenate into a symbolic world observation grid". I don't understand how the symbolic world observation grid looks like exactly.

-----
## Typos
- p.4: line 109: "is not complex, but [observation?] are 100 words on average", missing word?

limitations_and_societal_impact: The authors did discuss how they have adapted each environment to fit under a common API (e.g., convert text description to world representation). They also mentioned some issues human experts had with SymTD due to the symbolic representations discarding information such as color.

The authors have included a thorough impact statement in the appendix. They discussed the lack of interpretability that results from complex policies, especially those learned via RL (learning may result in unintuitive policies that achieve the object in surprising ways).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents SILG, a broad environment for language-conditioned multitask RL that unites several recently proposed symbolic language+RL environments, including RTFM, NetHack, Messenger, as well as instruction-following datasets including ALFWorld and a symbolic version of Touchdown. To unify such disparate environments, authors introduce a general framework for language conditioned RL, where agents operate given a partial observation of an environment, as well as language divided between "dynamics text", which consists of static, high-level language descriptions of the environment, and "non-dynamics text" which contains language specific to the agents' current observation and state, e.g. descriptions of the current observation, environment feedback, etc.

Authors then present a baseline model, SIR, a fairly standard language+RL model that conditions on text of varying levels of abstraction, applying successive FiLM2 layers to simulate multimodal reasoning between textual information and current observation. This entire architecture can then be trained without changes to each environment, which allows us to evaluate the generalizability of various modeling decisions in the language conditioned RL space. The authors show that many innovations proposed for certain environments (e.g. egocentric local convolution, forcing the building of representations from text dynamics) do not generalize across environments. SIR lags behind human performance on most of the environments, demonstrating potential for future work.
main_review: Overall, I like this paper: there is undoubtedly substantial engineering work that is put into making such disparate environments accessible under a common modeling/dataset/environment umbrella, which should certainly speed up prototyping and evaluation in language conditioned RL. I do see a few weaknesses of the paper as presented currently, however.

## Strengths

- This paper provides a broad framework for multi-task RL that conditions on language at varying levels of abstraction (dynamics/non-dynamics text). Providing a unified, simple environment for evaluation across all of these environments will undoubtedly be very useful for future work in this field.
- The paper shows quite nicely that several advancements developed for specific environments (perhaps unsurprisingly) fail to generalize reliably to other environments. This further highlights the need to come up with more general purpose language-conditioned RL techniques, in the same way that e.g. GLUE served as a good goal for more general-purpose NLU systems). Due to the unified environment and architecture, SILG should serve a similar role.

## Weaknesses

My main concerns have to do with the SymTD environment, the sparse description of the included environments, the lack of ablation studies on the base SIR model itself.

- Perhaps the most questionable part of the work is the SymTD environment, which appears to be a rather ad-hoc modification of an existing environment that has not yet been subject to the level of scrutiny of the existing environments, all of which are parts of peer-reviewed publications. There are several worrying aspects of SymTD:
    - An expert win rate of 60% is quite low and demonstrates significant noise in the environment. It could also be that many tasks have been converted incorrectly and are therefore impossible. What kinds of obstacles do human players face that result in such a low win rate?
    - Images are segmented to panoramas, but no evaluation about how accurate these panoramas are is presented. There are reasons to be suspect about the segmentation quality (e.g. because cityscapes are standard images, while touchdown is highly distorted panoramas).
- FiLM2 itself is a "recent grounded language RL modelling contribution" along the lines of those presented in 4.1, and several of the same questions asked in that section apply to the base SIR model: does FiLM2 generalize across environments? It would be good to have ablations of FiLM2, and more general ablations of the base SIR model (e.g. no language, of either kind).
- The descriptions of the environments included in SILG are extremely sparse (admittedly due to space constraints). The paper would benefit from clear examples of tasks across each environment, and a better characterization of the kinds of language information given in the environment and how they map onto the dynamics/non-dynamics text components of SIR. For example, for RTFM, presumably the game manual is categorized as dynamics text, but it's unclear whether the goal is dynamics text or non-dynamics text. Similarly, environments like ALFWorld and SymTD do *not* have high level game dynamics descriptions (I believe?)—is there simply no dynamics text in this setting? Touchdown only has one text field (L239) - is this dynamics or non-dynamics? One presentational suggestion is to add additional information in Table 1 that divides mean text len/vocab size by whether or not it's treated as dynamics text or not, and additionally a high level description of what kinds of language belong in which category (e.g. RTFM has game manual, ALFWorld has text description of environment, etc). (Maybe in Figure 1 as well?)
- Along these lines, the "relative position" feature for SIR is only briefly mentioned in L141, which left me confused. Why is this relative position to "a particular cell"? What particular cell is this, and why is it necessary?
- Some environments may be too easy: the inclusion of only the first stage of RTFM is rather odd, since we are already at human performance on this baseline (Table 2). What was the motivation for including only the first stage? (Similarly, only an easier subset of tasks was selected for Messenger and NetHack). Are the more difficult stages in these papers still compatible with both the SILG environment and the SIR framework? I would hope that authors will be able to include such harder environments in subsequent releases.

## Questions

- This was actually not super clear from the paper: SIR is a unified architecture, but it is trained in isolation on each individual environment, correct? Rather than training a single model across all environments.
    - (Assuming the above is true) SILG might enable some interesting questions regarding training across multiple environments and transfer learning. E.g. perhaps training on some environments actually enables faster learning/transfer to others, since there are general game-playing ideas common to each environment. Did authors try the (dramatic) attempt of training jointly across all environments at once?
- Table 2 lists performance of the best variant for each environment—what is the variant that leads to the best performance averaged across all environments? Presumably it's "+state"?
limitations_and_societal_impact: Yes, Appendix A.
review_title: null

Meta-review: Codifying benchmarks and bringing them under a single umbrella encourages better research. Particularly in an area like RL where results are often tenuous, depending on factors like random seeds. Larger and more comprehensive benchmarks ensure that methods really do improve on the state of the art.
 
While the idea is good, two major concerns came up during the discussion.
 
1. The work seems incomplete.
 
None of us could understand why the authors include only partial variants of existing benchmarks when creating SILG. This seems like a significant weakness that limits the SILG's contribution. The authors cite GLUE as their inspiration, but GLUE included other benchmarks wholesale; one didn't need to separately evaluate on them anymore because the final GLUE result already did so. By not including all of the games, as they were meant to exist, the authors sell their own approach short. We would love to see SILG published, with the full games included.
 
The authors put forward that they only use partial games because current methods can't deal with full games anyway and the resulting evaluation numbers would be too poor. Many methods exist for dealing with this, for example, in the retrieval community, methods usually don't retrieve the top result with perfect accuracy, and so we measure top-n (R@5, R@10, etc.) retrieval accuracy. The authors could do the same. Include the full games and then measure performance at 1% of the game, 10%, 20%, etc. This would make SILG a far more lasting contribution.
 
2. Essential details, that determine the technical validity of the benchmark and of the results, are completely missing.
 
Since the authors cite GLUE as their inspiration, it's worth having a second look at that paper. It very carefully defined what the inputs, outputs, and metrics are for every single benchmark rolled into GLUE. Unfortunately, almost all details to do with how models interact with the games are missing here.
 
Including such details is essential. They make the difference between a technically correct and a flawed manuscript. Imagine that the authors had crafted a mapping to one of the games in such a way that the game is now unplayable or far more difficult than it normally is. This is very easy to do.
 
As it stands, reviewers could not evaluate the technical correctness of the manuscript.
 
Even in terms of the novel conceptual contribution of the manuscript, it's really this mapping that we should be evaluating. It's the core of the method that brings the games together, and it’s missing.
 
One more minor point, the human experiment was generally seen as inadequate having only one subject.
 
I encourage the authors to continue with SILG despite this unfortunate news. The resulting benchmark and manuscript will be far more useful and impactful when fleshed out fully and when presented with enough detail to both reproduce the work and judge its technical correctness.
Decision: Accept (Poster)