Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a system for grounded, spatio-temporal, synthetic language learning.
Spatio-temporal, here, means descriptions of observations that describe spatial layout (e.g. front) and temporal descriptions (e.g. past).
The main results are 1) object identity is important for performance; 2) collabpsing object traces into a single token does not impact performance.
The paper studies different transformer architectures for solving the task with a specific eye on generalization 1) to hold out sentences, 2) to grammatical primitives.
The system is trained to learn a truth function that represents whether a given spatio-temporal statement is true.

main_review: Strength

The general research direction of learning grounded language specifically in tasks that are temporally extended is interesting and worthwhile.

The paper does give a good overview of more generally related literature such language learning in RL etc.

Authors do a good job at pointing out limitations of the work

The paper is written very clearly and is easy to follow.


Weaknesses

It's not clear to me that the main results merit publication. I think it's somewhat obvious that object identity plays a role in spatio-temporal tasks. Not sure what other explanation would be possible.

The paper learns on a synthetic language that is very structured. The language imo doesn't have exceptions and the free usage that characterises Natural language. Also the grammatical complexity of the synthetic language is quite reduced. It's not clear how important those choices are for the performance of the system and it's performance in more natural settings.

The setting is quite a toy environment. While it's illuminating as an example of the different aspects of the system, I am not sure that the result is already very interesting or surprising in the sense of solving the task. 


Questions/Comments

There is a huge are of AI that deals with models of spatio temporal and dynamic preposition reasoning and learning. Work by Tenbrink comes to mind. I think it's worthwhile to relate this paper to some of this work.

Similarly related work on compositional grounded language learning is missing in my view.

Truth functions for language represent a specific class of semantics. There are other forms of language use that are probably more important in real life. Such as the discrimination etc. I wonder how this system would deal with that. 

I am sort of surprised that the representation for sentences and words is one hot. Since most models these days train or use embeddings, I am not sure why this choice is important here. Embeddings may also remove the constraint that the dictionary is fixed.


limitations_and_societal_impact: YES
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. They do this by training a truth function that predicts whether the description matches the given history of observations or not. They study the impact of architectural choices for multimodal transformer and implement different attention computations between words and objects across space and time. They tested these models on generalization to held-out sentences and generalization to grammar primitives.
main_review: Originality and Significance:
-	This paper is the first step towards learning spatio-temporal concepts for embodied language grounding. They learn a truth function that predicts if a given sentence is correct for time-extended observations of an agent interacting with a collection of objects. They set up a task for this problem and provide the first baselines for it.

-	They define three transformer architectures: Unstructured Transformer, Spatial-First Transformer, and Temporal-First Transformer, and used these different architectures to aggregate observations and language. They demonstrate that it helps to process temporally-extended observations and language tokens side-by-side, shown by the scores of unstructured transformer.

Quality and Clarity:
-	The paper is clearly written, well-organized, and easy to follow.

limitations_and_societal_impact: Some limitations are described in the Conclusions sections.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new 2D environment that produces synthetic language describing spatio-temporal relations between objects within the environment. In the paper, the environment was used to generate interaction data using a bot. A synthetic language annotations describing the (spatio-temporal) interactions were generated. 

Given the data, the authors proceed by formulating the Embodied Language Grounding problem as learning a parametrized truth function of the spatio-temporal interactions. The parameterized truth function takes in the history of observations as well as a sentence describing the interactions and produces a boolean value signaling whether the sentence is true or false. Ground truth values are obtained by the data generation procedure.

The authors introduce a number of Transformer models and baseline LSTM models and perform two types of evaluation:
1. Testing the generalization capability of the models when the data is within the same distribution - by using % of the overall data as test data,
2. Testing the out-of-distribution generalization - by removing certain interactions from the training set, but adding them to the test set.

The authors show that Unstructured Transformer variant, which processes temporally-extended observations and language tokes side-by-side achieves a reasonable performance on almost all of the tasks. This in turn demonstrates it's ability to generalize. 
main_review: The paper introduces a novel way of studying Embodied Language Grounding through the means of a newly introduced environment, data generating procedure and an evaluation pipeline. The authors perform evaluation with the aim assessing generalization capabilities of a set of architectures. It presents a novel angle and a new approach to Embodied Language Grounding.

My main problem with the submission is clarity. The wording of the paper is very complicated and it took multiple passes to understand what the authors meant. Due to this, I am not entirely sure what the final conclusions on the generalization capabilities are, as well as what the novelty of the approach is, compared to the grounding strategies detailed in the introduction.

The way operations (reduce in particular) were explain was unclear to me until I went through code/math. So I would suggest formalizing the transformations using mathematical notation/pseudocode instead of explaining by words.

Due to this it is also difficult to assess significance of this work in a wider scope. Highlighting the main results in the conclusion and making parallels with related work mentioned in the introduction could help.

Some minor comments:
- It is not clear whether multiple linguistic observations ($W$) are generated for a single trace ($S$) in the dataset,
- `animal` objects can move independently, but I was not able to find in the paper how that behavior is implemented,
- Negative sentences are part of the dataset, but it is not clear how those are generated. The paper references Supplementary Section 2.1 for more information, but I wasn't able to find information on that part of the logic.

limitations_and_societal_impact: -
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work introduces a new embodied benchmark for grounding spatio-temporal language. The task is binary classification of whether a language description fits a temporal trajectory with multiple objects spanned spatially. Three transformer models are proposed: no aggregation (Unstructured Transformer, UT), spatial-first Transformer (SFT), and temporal-first Transformer (TFT). The models are evaluated on the benchmark with several generalization studies. The main finding is that maintaining object identity is instrumental to generalization, and aggregating an object trace in a single token has little influence.
main_review: Significance and Originality: Spatio-temporal language has been less studied in language grounding. However, as argued by authors, it is an important aspect of human language, and has an significant value to embodied robotics research. The benchmark, though simple and basic, fills an important position and can inspire future attention in this direction. 

Quality: The benchmark and baseline models are not complicated, yet the main findings (i.e. SFT is bad at generalization due to object mixing, several linguistic generalizations are hard) make sense as general findings about the problem. 

Clarity: The writing is neat, and the figures are clear.



limitations_and_societal_impact: - The language may be too synthetic. Is the English paraphrase in Figure 1 anyway used? Also, for the task of binary classification, random negatives might be too easy to figure out. Maybe some evaluation should be done with hard negatives, e.g. containing same basic concepts and objects, but organized in a different structure. This tests if the model is just pattern recognizing simple concept units, or is really understanding the compositional language.

- Why is binary classification an ideal task? Why not generating descriptions conditional on trajectories, which is harder and potentially more valuable?
review_title: null

Meta-review: To begin, I should clarify that during discussions, Reviewer MYUM stated that while their objection stands after reading other reviews, they see it as a corner case, and that they voted "marginally below threshold". I believe they forgot to update the rating on their review, so I'm interpreting their 4 as a 5+.

Simply put, this paper introduces a new setting for testing the grounded language learning capabilities of learning algorithms and architectures, whereby spatial and temporally extended relationships between objects must be captured by agents via (synthetic) language understanding. They propose a number of architectures which aim to help with this sort of language grounding, and compare them to sensible baselines. They evaluate against several forms of generalization, both on the linguistic and the relational front.

The main objection that could be levelled at this sort of environment is that the synthetic language is somewhat toy in that it is (as far as I understand) unambiguous, neatly structured, and has fairly homogenous semantics, but I think such testbeds are important if they serve as a starting point for research into more complex topics, such as the spatially and temporally extended relationships this work caters to. It's a neat contribution to language-informed/directed sequential decision making, and given the reviewer consensus post discussion, I don't think anyone is looking to champion the case for rejecting it. I recommend acceptance.
Decision: Accept (Poster)