Grounding Spatio-Temporal Language with Transformers

Abstract: Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to new sentences, 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents.


Reviews and author responses:

 We thank reviewer 8BQz for their comment. We are glad that our motivations for studying the truth function appear clear now. We perfected our introduction, making these motivations more salient.

We now provide clarifications on the hard negative experiments. For both of our control experiments (hard trajectories and hard descriptions), **we evaluate previously trained models on new test sets containing hard negatives**.
 
In the hard trajectories setup, by ''trajectories contain the object or the actions described by the description'', we mean that the negative trajectories are selected such that they contain either the action or the object included in the positive examples. Hence, if the description is "Grasp red cat" for instance, the hard negative examples will either include trajectories with red cats (not being grasped) or trajectories with an object being grasped (not a red cat).

In the hard descriptions setup, two models are consistently better than the others (UT and TFT have higher f1 scores on split 1 to 4 and their WA variants outperform others on split 5). 
As one can see in the table provided, UT and TFT performances on split 1 and 2 are on par with other splits indicating that they do not fail so badly. 


|                       |LSTM-FACTORED|LSTM-FLAT  |UT          |UT-WA      |SFT         |SFT-WA     |TT         |TT-WA      |
|------------------------|-------------|-----------|------------|-----------|------------|-----------|-----------|-----------|
|1. Object-attribute     |0.70 ± 0.05  |0.69 ± 0.10|**0.76 ± 0.05** |0.71 ± 0.03|0.76 ± 0.05 |0.69 ± 0.02|**0.73 ± 0.04**|0.71 ± 0.02|
|2. Predicate-object     |0.64 ± 0.05  |0.67 ± 0.05|**0.75 ± 0.08** |0.74 ± 0.06|0.69 ± 0.03 |0.71 ± 0.06|**0.70 ± 0.03**|0.77 ± 0.06|
|3. One-to-one relation  |0.32 ± 0.07  |0.39 ± 0.12|**0.71 ± 0.07**|0.63 ± 0.03|0.40 ± 0.22 |0.21 ± 0.07|**0.70 ± 0.05**|0.67 ± 0.04|
|4. Past spatial relation|0.71 ± 0.12  |0.60 ± 0.12|**0.76 ± 0.05** |0.59 ± 0.06|0.68 ± 0.09 |0.73 ± 0.02|**0.76 ± 0.05**|0.60 ± 0.07|
|5. Past predicate       |0.50 ± 0.22  |0.43 ± 0.24|0.22 ± 0.25 |**0.72 ± 0.13**|0.16 ± 0.21 |0.60 ± 0.08|0.11 ± 0.16|**0.64 ± 0.11**|

These results indicate that transformer models do have acquired word order knowledge and use them on split 1 and 2 even if the simple presence or absence of words is enough to extract the meaning of descriptions. A solution to make those models more robust to word order mutations would be to expose them to shuffled training descriptions during training in order to reinforce their word order relational computations.



This paper introduces a new 2D environment that produces synthetic language describing spatio-temporal relations between objects within the environment. In the paper, the environment was used to generate interaction data using a bot. A synthetic language annotations describing the (spatio-temporal) interactions were generated. 

Given the data, the authors proceed by formulating the Embodied Language Grounding problem as learning a parametrized truth function of the spatio-temporal interactions. The parameterized truth function takes in the history of observations as well as a sentence describing the interactions and produces a boolean value signaling whether the sentence is true or false. Ground truth values are obtained by the data generation procedure.

The authors introduce a number of Transformer models and baseline LSTM models and perform two types of evaluation:
1. Testing the generalization capability of the models when the data is within the same distribution - by using % of the overall data as test data,
2. Testing the out-of-distribution generalization - by removing certain interactions from the training set, but adding them to the test set.

The authors show that Unstructured Transformer variant, which processes temporally-extended observations and language tokes side-by-side achieves a reasonable performance on almost all of the tasks. This in turn demonstrates it's ability to generalize.   The paper introduces a novel way of studying Embodied Language Grounding through the means of a newly introduced environment, data generating procedure and an evaluation pipeline. The authors perform evaluation with the aim assessing generalization capabilities of a set of architectures. It presents a novel angle and a new approach to Embodied Language Grounding.

My main problem with the submission is clarity. The wording of the paper is very complicated and it took multiple passes to understand what the authors meant. Due to this, I am not entirely sure what the final conclusions on the generalization capabilities are, as well as what the novelty of the approach is, compared to the grounding strategies detailed in the introduction.

The way operations (reduce in particular) were explain was unclear to me until I went through code/math. So I would suggest formalizing the transformations using mathematical notation/pseudocode instead of explaining by words.

Due to this it is also difficult to assess significance of this work in a wider scope. Highlighting the main results in the conclusion and making parallels with related work mentioned in the introduction could help.

Some minor comments:
- It is not clear whether multiple linguistic observations ($W$) are generated for a single trace ($S$) in the dataset,
- `animal` objects can move independently, but I was not able to find in the paper how that behavior is implemented,
- Negative sentences are part of the dataset, but it is not clear how those are generated. The paper references Supplementary Section 2.1 for more information, but I wasn't able to find information on that part of the logic.
 -

 I would like to thank the authors for their detailed replies and apologize for the delay on my end. Authors have addressed all of the concerns raised in my initial review as well as a potential follow up with their reply to reviewer 8BQz. 

I believe that novelty presented in the paper, potential that it has through the experimental setup developed as part of this work, together with clarifications provided here deserve a larger score and merit publication. However, I do share the concerns pointed out by reviewer MYUM on the simplicity of the environment and believe that providing additional axes of complexity could significantly improve the paper and the confidence in the current results. 

I am raising the score to marginally above acceptance.

 Thanks for the rebuttal! Sorry I replied late.

I agree the truth function setup is important and appreciate the added discussion. 

For hard negative result:

- **Do you use hard negatives in training, or testing, or both?** The setup is important to state. I guess you evaluate previously trained models on new tests of hard negatives. 

- The 1st setup makes more sense to me, and is indeed a positive result to add to the paper. Just to make sure, by "trajectories contain the object or the actions described by the descriptions", you mean at least one element is in both hard negative and original?

- The 2nd setup is harder, and the result is interesting. Is one model solving all 5 splits, or five models? If one model, then by the argument it should pick up some word order knowledge and not fail so badly on split 1-2. If five models, it would be interesting to think about how to learn a model robust to all five challenges (as future work maybe).

I think the rebuttal is helpful to me and other authors, thanks again. If all matters here are clarified I'm happy to raise the score!

 ### Main response:

We thank reviewer MYUM for their constructive commentary and for clearly stating the strengths and weaknesses of our work. Reviewer MYUM assumes our main contribution is an experimental proof that ''object identity plays a role in Spatio-temporal tasks''. Although we discuss this result in our conclusion, our work targets a more general objective. Indeed, we view our paper as the first step towards exploring specifically how to ground Spatio-temporal concepts, providing a dataset, a set of baselines, and their analysis. We apologize if this contribution does not appear clear in the text and hope that, through discussion, we can agree on the scope of the paper. To our knowledge, our work is the first attempt to study specifically systematic generalization of spatio-temporal language. Therefore, our experiments rely on several simplifications pointed out by the reviewer. 

*''The setting is quite a toy environment [...] and the paper learns on a synthetic language that is very structured''*

We try to provide extra justification and details about our assumptions below. We propose three categories of arguments supporting our choices.

First, our setup offers several experimental conveniences:
- The setup allows us to have fine-grained control of the semantic of our predicates; especially time-extended predicates;
Although quite simplistic, our setup includes key features of Spatio-temporality (time-extended signature of predicates, Spatio-temporal references to moving objects);
- The synthetic language relies on a generative grammar that facilitates data collection  (no human in the loop);
- The synthetic language allows to precisely isolate different categories of meanings and its generative grammar is used to define the systematic splits on which generalization is tested.

Second, the results we obtain suggest possible future improvements:
- Observing significant differences between models provides a better understanding of the different impacts of inductive biases in Language Grounding architectures
- The fact that our best architecture does not obtain near-perfect performance in 3 of our 5 systematic splits indicates that there is room for improvement and that our proposed dataset offers interesting challenges for the future.

Third, studying language grounding and linguistic generalization with synthetic language has a long history in the literature:
- Synthetic language, contrary to natural language, is fully compositional and does not contain exceptions. This is, thus, the ideal setting for studying the compositional abilities of neural networks. 
- As discussed in our conclusion (l360) there exist techniques that enable the application of learning systems trained on synthetic language to natural language settings with varied grammatical structures and synonyms.

### Answers to questions and comments:

*“There is a huge area of AI that deals with models of spatio temporal and dynamic preposition reasoning and learning”*

We thank reviewer MYUM for very interesting and relevant pointers to the work of Tenbrink and colleagues, we will add detailed discussion of their work in our related work section. Here we will simply insist that their line of work is trying to formalize the use of spatial relations (and temporal relations) in language users. This approach is of particular relevance to our work. Nevertheless, we place ourselves in the context of an embodied agent executing actions over several timesteps and thus the aspect of *temporally-extended predicates* is very important in our setting. Since their work focuses on temporal relations between events only, we feel that temporally-extended predicates are not completely covered by this previous work. Similarly, we simplify the spatial reference frame by considering a default orientation of Temporal Playground, which circumvents part of the reference frame problem tackled by Tenbrink et. al.

*“Truth functions for language represent a specific class of semantics.”*

We agree with the reviewer that truth functions are only a part of language grounding, however it is a very important one. It comes down to learning the agreement of two variables, one in language space, the other in observation space, thus tying together these two modalities. This, in turn, can be used as a goal-achievement function for an embodied agent trying to reach a language goal. In this context, the agreement between the observation and the language instruction can be seen as a reward function, for instance to be used by an autonomous RL agent.
Additionally, the setting of binary classification allows us to control the generation of negative samples which we do in the additional experiments we provide in the response to reviewer 8BQz.

*“I am sort of surprised that the representation for sentences and words is one hot.”*

We use one-hots for individual words, which are then projected with a language encoder. This means we multiply our one-hot vectors with a learned weight matrix, which comes down to performing lookup of a specific row (or column depending on the notational convention) of this learned matrix. This is mathematically equivalent to training our own embeddings. Using pre-trained embeddings makes sense in a context where we need to be able to generalize to unseen synonyms, but since our vocabulary size is limited we did not consider this option.

 We thank reviewer 8BQz for their helpful feedback and for suggesting new experiments on our benchmark. In this response, we first justify the use of a multi-modal binary classification task over other tasks. We then discuss some new experimental evaluation of our models with hard negatives, as was suggested in the reviewer’s comments.

We agree that description generation from behavioral trajectories is valuable as it could, for instance, allow for the design of agents that could narrate their ongoing activities and explain their own behavior. However, we believe that learning a truth function in a binary classification setting remains a crucial and general problem. Indeed, within a Reinforcement Learning perspective, in which agents are trained to reach language goals, meanings are modeled via reward function [1,2].  If agents are able to convert Spatio-temporal descriptions provided by a Social Partner into rewards, they can self-train a policy that reproduces the trajectories they represent. This allows for the design of autonomous agents that learn from social/linguistic interactions with a caregiver, removing the burden of designing reward functions. On top of that, recent work [2] shows that the systematic generalization capacity of language conditioned reward functions is a crucial question to address as it enables goal imagination: in an open-ended learning setup, agents can generate new descriptions (never provided by a social partner) and leverage the generalization of the reward function to discover new interactions in their environment. Bearing this in mind, studying the language grounding of new spatio-temporal concept by modelling a truth function is valuable as improving the range of meanings that can be modelled could directly improve existing methods. We discuss this possible application at the end of our discussion (l361-367).

In the binary classification setting, reviewer 8BQz mentions that randomly selected negative examples might be too easy to figure out and suggests that testing our models on hard negatives could give more insight into the model’s ability to understand compositional language. We completely agree and ran new evaluations with hard negatives. More specifically, we investigated two control settings with different types of hard negatives:

- 1. Hard trajectories: negative pairs (trajectory, description) are selected so that the trajectories contain the object or the actions described by the descriptions.
- 2. Hard descriptions:  negative pairs (trajectory, description) are selected so that the trajectories match with the descriptions but the descriptions are mutated. Mutations are performed by randomly shuffling the order of words in the ground truth descriptions.

In the first setting with hard trajectories, we observe a slight decrease of performances on all 5 categories (drop is less than 5%), demonstrating that the models do in fact represent the meaning of the sentence and not simply the presence or absence of a particular object or predicate.

In the second setting with hard descriptions, we observe an important drop of performances (around 0.3 points across models) on the first two categories of systematic generalization (new object-attribute and new predicate-object combinations). Conversely, the three other categories of systematic generalization show similar performances as in the random negative setting. This result can be explained by the fact that, for the two first categories, word ordering does not impact the meaning of descriptions. Therefore, it is sufficient for the models to perform unit concept recognition in order to get high f1 scores in the train, and when tested on randomly re-shuffled sentences this partial permutation invariance prevents the model from considering these sentences as negative. This is not the case in the harder systematic splits 3 to 5, where the ordering of words does matter for deciding the sentence in the train set (think of the position of the token ‘was’, or the order of words in spatial relations). To summarize, the models do seem to recognize more than simple concept units, even if there is a partial effect of individual, independent recognition in the first and second systematic split.

[1] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosseini, P. Kohli, E. Grefenstette -  Learning to Understand Goal Specifications by Modelling Reward - ICLR 2019

[2] C. Colas, T. Karch, N. Lair, J.M. Dussoux, C. Moulin-Frier, P. Dominey, P.Y. Oudeyer - Language as a Cognitive Tool to imagine goals in Curiosity-Driven Exploration - Neurips 2020

 ### Main response

We thank reviewer viF6 for their thoughtful review. We first point out that other reviewers were satisfied with the clarity of the writing. Nevertheless, we are ready to take any steps necessary to ensure that the wording of the main text becomes clearer; particularly reformulating the introduction and the conclusion to make our contributions more salient and make our setting stand out from the existing literature. We first address the comment about clarity and then we respond in detail to the additional comments of the reviewer.

*“what the novelty of the approach is, compared to the grounding strategies detailed in the introduction [...] what the final conclusions on the generalization capabilities are”*

In this paper we introduce a new spatio-temporal language grounding task, which has been little-studied in the language grounding literature so far. We introduce spatial relations, and temporality. The temporal aspect of the language and semantics we introduce is novel, and is implemented in two ways: first we consider temporally-extended predicates, which describe actions happening over several time steps, and we introduce the past tense, indicated by the linguistic token ‘was’. This past modality can be both applied to predicates to describe an action that happened in the past and is no longer happening, or to spatial relations, to describe a spatial relation that was valid in the past but no longer valid in the current frame. To the best of our knowledge we are the first to introduce these aspects of temporality in a controlled embodied language grounding setting.

The temporal aspect of our language entails that we must use temporally-extended observations to decide the truth value of a predicate. This raises the question of which architectural inductive biases are best for tackling this problem. We answer this question by systematically considering architectural variants and measuring their generalization both in a setting where test sentences are drawn at random and in a setting where some word combinations are systematically excluded from the training set.

We conclude by observing that in all cases the Unstructured Transformer variant performs best in all kinds of generalization, closely followed by the Temporal-First Transformer. Adding word-aggregation does not seem to impact performance in a significant way in the random split, but does have an ambiguous effect in the systematic split experiment. This is useful for further work in multimodal language grounding, where unstructured architectures such as UT could be used if the size of the context allows it. This is good news since this architecture is very general; for instance it is trivially extensible to more input modalities.

### Detailed response to comments:

*“So I would suggest formalizing the transformations using mathematical notation/pseudocode instead of explaining by words”*

In a final version of the paper we will introduce the pseudocode notation at the beginning of the section on architectural choices and hope that this will make the architectural choices easier to follow.

*“It is not clear whether multiple linguistic observations (W) are generated for a single trace (S) in the dataset”*

After one episode of interaction between the scripted agent and the environment, a grammar-based system generates all grammatically valid sentences that are true of the given scene. This means that many sentences are generated after an episodic rollout. This allows us to define a mapping between the observations in our dataset and the set of all possible sentences according to the grammar. This mapping allows us to fetch positive and negative observations for a given sentence. We added additional explanation of this procedure in the supplementary material to clarify.

*“animal objects can move independently, but I was not able to find in the paper how that behavior is implemented,”*

At the beginning of an episode, a direction is sampled for each animal, and at each time step each animal’s position is incremented by a fixed amplitude along their preferred direction plus a uniformly sampled noise term. We added a more detailed description in the supplementary material.

*“Negative sentences are part of the dataset, but it is not clear how those are generated.”*

For each episode we generate the set of all valid and true sentences. Our grammar provides us with a way to generate all valid sentences regardless of their truth value. Thus, by exclusion, all sentences that are valid but not true in this episode must be false. This is the procedure by which we generate our negative examples. There are other ways to sample negative examples, for instance by making sure that the negative sentence in question contains linguistic references to the actual objects present in the scene. This is what we do in one supplementary experiment in the response to reviewer 8BQz. The model’s f1 scores are left unchanged in this condition. We thus conclude that our models do not simply rely on the recognition of the presence and absence of the objects in the sentence but actually learn to bind the predicate to the referred object.

 We are thankful for their kind words and would be happy to answer any of their questions that may arise from reading other reviews.

The paper presents a system for grounded, spatio-temporal, synthetic language learning.
Spatio-temporal, here, means descriptions of observations that describe spatial layout (e.g. front) and temporal descriptions (e.g. past).
The main results are 1) object identity is important for performance; 2) collabpsing object traces into a single token does not impact performance.
The paper studies different transformer architectures for solving the task with a specific eye on generalization 1) to hold out sentences, 2) to grammatical primitives.
The system is trained to learn a truth function that represents whether a given spatio-temporal statement is true.
  Strength

The general research direction of learning grounded language specifically in tasks that are temporally extended is interesting and worthwhile.

The paper does give a good overview of more generally related literature such language learning in RL etc.

Authors do a good job at pointing out limitations of the work

The paper is written very clearly and is easy to follow.


Weaknesses

It's not clear to me that the main results merit publication. I think it's somewhat obvious that object identity plays a role in spatio-temporal tasks. Not sure what other explanation would be possible.

The paper learns on a synthetic language that is very structured. The language imo doesn't have exceptions and the free usage that characterises Natural language. Also the grammatical complexity of the synthetic language is quite reduced. It's not clear how important those choices are for the performance of the system and it's performance in more natural settings.

The setting is quite a toy environment. While it's illuminating as an example of the different aspects of the system, I am not sure that the result is already very interesting or surprising in the sense of solving the task. 


Questions/Comments

There is a huge are of AI that deals with models of spatio temporal and dynamic preposition reasoning and learning. Work by Tenbrink comes to mind. I think it's worthwhile to relate this paper to some of this work.

Similarly related work on compositional grounded language learning is missing in my view.

Truth functions for language represent a specific class of semantics. There are other forms of language use that are probably more important in real life. Such as the discrimination etc. I wonder how this system would deal with that. 

I am sort of surprised that the representation for sentences and words is one hot. Since most models these days train or use embeddings, I am not sure why this choice is important here. Embeddings may also remove the constraint that the dictionary is fixed.

 YES

This paper proposes a new spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. They do this by training a truth function that predicts whether the description matches the given history of observations or not. They study the impact of architectural choices for multimodal transformer and implement different attention computations between words and objects across space and time. They tested these models on generalization to held-out sentences and generalization to grammar primitives.  Originality and Significance:
-	This paper is the first step towards learning spatio-temporal concepts for embodied language grounding. They learn a truth function that predicts if a given sentence is correct for time-extended observations of an agent interacting with a collection of objects. They set up a task for this problem and provide the first baselines for it.

-	They define three transformer architectures: Unstructured Transformer, Spatial-First Transformer, and Temporal-First Transformer, and used these different architectures to aggregate observations and language. They demonstrate that it helps to process temporally-extended observations and language tokens side-by-side, shown by the scores of unstructured transformer.

Quality and Clarity:
-	The paper is clearly written, well-organized, and easy to follow.
 Some limitations are described in the Conclusions sections.

The work introduces a new embodied benchmark for grounding spatio-temporal language. The task is binary classification of whether a language description fits a temporal trajectory with multiple objects spanned spatially. Three transformer models are proposed: no aggregation (Unstructured Transformer, UT), spatial-first Transformer (SFT), and temporal-first Transformer (TFT). The models are evaluated on the benchmark with several generalization studies. The main finding is that maintaining object identity is instrumental to generalization, and aggregating an object trace in a single token has little influence.  Significance and Originality: Spatio-temporal language has been less studied in language grounding. However, as argued by authors, it is an important aspect of human language, and has an significant value to embodied robotics research. The benchmark, though simple and basic, fills an important position and can inspire future attention in this direction. 

Quality: The benchmark and baseline models are not complicated, yet the main findings (i.e. SFT is bad at generalization due to object mixing, several linguistic generalizations are hard) make sense as general findings about the problem. 

Clarity: The writing is neat, and the figures are clear.


 - The language may be too synthetic. Is the English paraphrase in Figure 1 anyway used? Also, for the task of binary classification, random negatives might be too easy to figure out. Maybe some evaluation should be done with hard negatives, e.g. containing same basic concepts and objects, but organized in a different structure. This tests if the model is just pattern recognizing simple concept units, or is really understanding the compositional language.

- Why is binary classification an ideal task? Why not generating descriptions conditional on trajectories, which is harder and potentially more valuable? 

Meta-review: To begin, I should clarify that during discussions, Reviewer MYUM stated that while their objection stands after reading other reviews, they see it as a corner case, and that they voted "marginally below threshold". I believe they forgot to update the rating on their review, so I'm interpreting their 4 as a 5+.

Simply put, this paper introduces a new setting for testing the grounded language learning capabilities of learning algorithms and architectures, whereby spatial and temporally extended relationships between objects must be captured by agents via (synthetic) language understanding. They propose a number of architectures which aim to help with this sort of language grounding, and compare them to sensible baselines. They evaluate against several forms of generalization, both on the linguistic and the relational front.

The main objection that could be levelled at this sort of environment is that the synthetic language is somewhat toy in that it is (as far as I understand) unambiguous, neatly structured, and has fairly homogenous semantics, but I think such testbeds are important if they serve as a starting point for research into more complex topics, such as the spatially and temporally extended relationships this work caters to. It's a neat contribution to language-informed/directed sequential decision making, and given the reviewer consensus post discussion, I don't think anyone is looking to champion the case for rejecting it. I recommend acceptance.

Decision: accept