Grounding Spatio-Temporal Language with Transformers
Authors: Tristan Karch
Laetitia Teodorescu
Katja Hofmann
Clément Moulin-Frier
Pierre-Yves Oudeyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Language Grounding
Embodied Autonomous Agents

Abstract: Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to new sentences, 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents.


Grounding Spatio-Tempora, Language with Transformers
Tristan Karch Laetitia Teodorescu"
Katja Hofmann Microsoft Research Cambridge, UK
Clément Moulin-Frier Inria Flowers team Université de Bordeaux ENSTA ParisTech
Inria Flowers Team Université de Bordeaux firstname. last lastname@inria.ft
Pierre-Yver Oudeyer
Inria Flowers team Université de Bordeaux ENSTA Paris Tech
Abstract
Language is an interface to the outside world. In order for embodied agents it, language must be grounded in other, sensorimotor modalities. While there is extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts still largely uncharted. To make progress in this direction, here introduce novel -patio-temporal language grounding task where the goal is to learn the meaning of descriptions of behavioral traces of embodied agent. This achieved by training a truth function that predicts a description matches a given history observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role architectural biases this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guide guided autonomous embodied agents. We also release our code under open-source license well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.
Introduction
Building autonomous agents that learn represent and use language is long standing-goal in Artificial Intelligence 20 39 In developmental robotics 77, language considered a cornerstone of development that enables embodied cognitive agents to learn more efficiently through social interactions with humans or through cooperation with other autonomous agents. It is also considered essential to develop complex sensorimotor skills, facilitating the representation of behaviors and actions.
Embodied Language Grounding [50] is the field that studies how agents can align language with their behaviors in order to extract the meaning of linguistic constructions. Early approaches in
Equal Contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) Grow action with spatial or attribute reference to object
(b) Shoke action with spatio temporal reference to object
(c) Grasp past actions description with
spatio-temporal reference to object
shake
Traces Interactions
Sampled descriptions
Grow blue chameleun
SShaee red cat'
"Was graxp blue cactus'
Grow thing left tf
Shaae thing right of lamp'
"Was grosp thing top most"
Grow thing left moxt"
SShake thing was left of lamp'
'Was grasp dhing was right of parret
Grrasp green water"
Natural
English version
'You are currently growing the blue chamellon"
Youu are currently shaking some- thing that used to be at the left of the lomp'
"Yuu previously grasped some- thing that used to be at the right of the parrot'
Figure 1: Visual summary of the Temporal Playground environment: At each episode (column b and c). the actions of an agent (represented by hand) unfold in the environment and generate trace of interactions between objects and the agent body. Given such a trace, the environment automatically generates a set of synthetic linguistic descriptions that are true at the end of the trace. In (a) the agent grows an object which is described with spatial (underlined) or attribute (highlighted) reference. In (b) it shakes an object which is described with attribute, spatial spatio-temporal (underlined) reference. In (c) it has grasped an object (past action underlined) which is described with attribute, spatial or spatio-temporal (highlighted) reference. The natural english version is given for illustrative purposes.
developmental robotics studied how various machine learning techniques, ranging from neural networks 40 45 24] to non-negative matrix factorization [3), could enable the acquisition of
grounded compositional language 42 41. This line of work was recently extended using techniques
for Language conditioned Deep Reinforcement Learning (1.. Among these works we can distinguish mainly three language grounding strategies. The first one consists of directly grounding language in the behavior of agents by training goal-conditioned policies satisfying linguistic instructions The second aims at extracting the meaning of sentences from mental simulations (i.c. generative models) of possible sensorimotor configurations matching linguistic descriptions
40
45
23
22
The third strategy searches learn the meaning of linguistic constructs terms of outcomes that agents can observe in the environment. This is achieved by training truth function that detects if descriptions provided by an expert match certain world configurations. This truth function can be obtained via Inverse Reinforcement Learning 4u
32
3 or by training a multi- binary classifier 14. Previous work [4 3 has shown that access to this reward is enough for sucessfully grounding language in instruction followin agents.
While all the above-mentioned approaches consider language that describes immediate and instanta- neous actions, we argue that is also important for agents to grasp language expressing concepts that are relational and that span multiple time steps. We thus propose to study the grounding of new spatio-temporal concepts enabling agents to ground time extended predicates (Fig. [p) with complex spatio-temporal references objects (Fig. (D)) and understand both present and past tenses (Fig. I)) To do so we choose the third strategy mentioned above, i.e. to train a truth function that predicts when descriptions match traces of experience. This choice motivated by two important considerations. First, prior work showed that learning truth functions was key to foster generalization B, enabling agents to efficiently self-train policies via goal imagination [E]] and goal relabeling [12]. Hence the truth function is an important and self-contained component of larger learning systems. Second, this strategy allows to carefully control the distribution of experiences and descriptions perceived by the agent.
2

---

The Embodied Language Grounding problem has a relational structure. We understand the meaning of words by analyzing the relations they state in the world [ The relationality of spatial and temporal concepts has long been identified in the field of pragmatics
(see Supplementary Section C
for additional discussion). Furthermore actions themselves are relations between subjects and objects, and can be defined in terms of affordances of the agent [9 We acknowledge this and provide the right relational inductive bias [5] to our architectures through the use of Transformers [ We propose formalism unifying three variants of a multi-modal transformer inspired by Ding et al. 16 that implement different relational operations through the use of hierarchical attention. We measure the generalization capabilities of these architectures along three axes 1) generalization to new traces of experience; 2) generalization to randomly held out sentences; 3) generalization to grammar primitives, systematically held out from the training set as in Ruis et al. [77. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall. We also identify specifie relational operations that are key to generalize on certain grammar primitives.
Contributions. This paper introduces:
1. A new Embodied I Language Grounding task focusing on spatio-temporal language;
2. A formalism unifying different relational architectures based on Transformers expressed as a function of mapping and aggregation operations;
3. A systematic study of the generalization capabilities of these architectures and the identifica- tion of key components for their success on this task.
2 Methods
2.1 Problem Definition
We consider the setting of an embodied agent behaving in an environment. This agent interacts with the surrounding objects over time, during an episode of fixed length (7), Once this episode is over, an oracle provides exhaustive feedback in a synthetic language about everything that has happened. This language describes actions of the agent over the objects and includes spatial and temporal concepts. The spatial concepts are reference an object through its spatial relation with others (Fig. Ia), and the temporal concepts are the past modality for the actions of the agent (Fig. [e),, past modality for spatial relations (Fig. [T). and actions that unfold over time intervals. The histories of states of the agent's ' body and of the objects over the episode as well as the associated sentences are recorded in a buffer B. From this setting, and echoing previous work on training agents from descriptions, we frame the Embodied Language Grounding problem as learning parametrized truth function Ra over couples of observations traces and sentences, tasked with predicting whether a given sentence W is true of given episode history S or not. Formally, we aim to minimize:
[s,
r(,
where L denotes the cross-entropy loss and r denotes the ground truth boolean value for sentence W about trace S.
2.2 Temporal Playground
In the absence of any dedicated dataset providing spatio-temporal descriptions from behavioral traces of an agent, we introduce Temporal Playground (Fig. D an environment coupled with a templated grammar designed to study spatio-temporal language grounding. The environment is a 2D world, with procedurally-generaaee scene containing N = 3 objects sampled from 32 different object types belonging to 5 categories. Each object has continuous 2D position, a size, continuous color code specified by a 3D vector in RGB space, a type specified by one-hot vector, and a boolean unit specifying whether it is grasped. The size of the object feature vector (0) is |o] = 39. The agent's 's body has its 2D position in the environment and its gripper state (grasping or non-grasping) as features (body feature vector (b) of size |b| = 3). In this environment, the agent can perform various actions over the length (7) of an episode. Some of the objects (the animals) can move independently. Objects can also interact: if the agent brings food or water to an animal, will grow in size; similarly, if water is brought to plant, it will grow. At the end of an episode, generative grammar generates
3

---

sentences describing all the interactions that occurred. A complete specification of the environment as well as the BNF of the grammar can be found in Supplementary Section
Synthetic language. To enable a controlled and systematic study of how different types of spatio- temporal linguistic meanings can be learned, we argue is necessary to first conduct a systematic study with a controlled synthetic grammar. We thus consider synthetic language with a vocabulary of size 53 and sentences with maximum length of 8. This synthetic language facilitates the generation of descriptions matching behavioral traces of the agent. Moreover, it allows us to express four categories of concepts associated with specific words. Thus, the generated sentences consist in four conceptual types based on the words they involve:
Sentences involving basic concepts. This category of sentences talk about present-time events by referring to objects and their attributes. Sentences begin with the 'grasp' token combined with any object. Objects can be named after their category (eg. 'animal", 'thing' ) or directly by their type ('dog', 'door', 'algae', etc.). Finally, the color ( ('red', 'blue", green') of objects can also be specified.
Sentences involving spatial concepts. This category of sentences additionally involve one-to-one spatial relations and one-to-all spatial relations to refer to objects. An object can be 'left of another object (reference is made in relation to a single other object), or can be the 'top most' object (reference is made in relation with all other objects). Example sentences include graasp thing bottom of cal' or 'grasp thing right most'.
Sentences involving temporal concepts. This category of sentences involves talking about predicates and the past tense, without any spatial relations. The two temporal predicates are denoted with the words 'grow' and 'shake", The truth value of these predicates can only be decided by looking at the temporal evolution of the object's size and position respectively. A predicate is transposed at the past tense if the action describes was true at some point in the past and is no longer true in the present, this is indicated by adding the modifier 'was' before the predicate. Example sentences include 'was grasp red chameleon' (indicating that the agent grasped the red chameleon and then released it) and shake bush";
Sentences involving spatio-temporal concepts. Finally, we consider the broad class of spatio-temporal sentences that combine spatial reference and temporal or past-tense predi- cates. These are sentences that involve both the spatial and temporal concepts defined above. Additionally, there is case of where the spatial and the temporal aspects are entangled: past spatial reference. This happens when an object is referred to by its previous spatial relationship with another object. Consider the case of an animal that was at first on the bottom of a table, then moved top, and then is grasped. In this case could refer to this animal as something that was previously on the bottom of the table. We use the same 'was' modifier as for the past tense predicates; and thus we would describe the action as 'Grasp thing was bottom of table".
2.3 Architectures
In this section we describe the architectures used as well as their inputs. Let one input sample to model be I = (S, W), where (S,.))., represents the objects' and body's evolution, and (Wi), represents the linguistic observations. S has a spatial (or entity) dimension indexed by E [O...N] and temporal dimension indexed by t [1..T]; for any i, f, Si, is a vector of observational features. Note that by convention, the trace (So,e), represents the body's features, and the traces (Si,.),.ss represents the other objects' features. W is 2-dimensional tensor indexed by the sequence / E [1..L]: for any I, Wi &dw is one-hot vector defining the word in the dictionary. The output to models is a single scalar between O and 1 representing the probability that the sentence encoded by W is true in the observation trace S.
Transformer Architectures. To systematically study the influence of architectural choices on language performance and generalization in our spatio-temporal grounded language context, we define a set of mapping and aggregation operations that allows us to succinetly describe different models in unified framework. We define:

---

W
(c) Unstructured Transformer (UT)
$
Body Encoder
Object Encoder
Postioona Encodingg
W "Grusp thing left most"'
Language
Encoder
(d) Spatial Transformer (SFT)
(a) Input Encoding
reduce
redace
(h) Optional Wordl Aggregation (-WA)
(e) T'emporol-First Transformer (TFT)
Visual summary of the architectures used. We show the details of UT, SFT and TFT respectively in subfigures (c), (d), (e), as well as a schematic illustration of the preprocessing phase (a) and the optional word-aggregation procedure (b).
Figure 2:
An aggregation operation based a Transformer model, called reduce. reduce is parametrized function that takes 3 inputs: a tensor, a dimension tuple D over which to reduce and query tensor (that has have the size of the reduced tensor). R layers of Transformer are applied to the input-query concatenation and are then queried at the position corresponding to the query tokens. This produces an output reduced over the dimensions D. A casting operation called cast. cast takes as input 2 tensors A and B and a dimension d. A is flattened, expanded as to fit the tensor B in all dimensions except and concatenated along the d dimension.
A helper expand operation called expand that takes arguments tensor and integer and repeats the tensor n times.
Using those operations, define three architectures: Transformer, inspired by Ding et al. [16]. or UT); one with a spatial-first structural bias - objects and words are aggregated along the spatial dimension first (Spatial-Fine Transformer or SFT); and one with temporal-fire structural bias objects and words are aggregated along the temporal dimension first (Temporal-Fite -irst Transformer. TFT).
with particular bias (Unstructured
Before inputting the observations of bodics and objects S and the language W into any of the Trans- former architectures, they are projected more details). A positional encoding 46 is then added along the time dimension for observations and along the sequence dimension for language; and finally one-hot vector indicating whether the vector observational or linguistic is appended at the end. This produces the modified diservation-language tuple ($, W). We let:
common dimension (see Supplementary Section B.2 for
UT UT ($, W) == reduce(cast($, W, 0.g).
SFT W.q) == reduce (reduce ( castuucccccattti// $, 0, expand T)), 0)
TF W, W.4)= reduce(reducel cwweeeeaassattaaccaaaccc Š, 1), 1, expand N + 1)), 0,
where 7' is the number of time steps, N is the number of objects and g a learned query token. See Fig. 2 for an illustration of these architectures.
Note that SFT and TFT are transpose versions of each other: SFT is performing aggregation over space first and then time, and the reverse true for TFT. Additionally, we define a variant of each of these architectures where the words are aggregated before being related with the observations. We name these variants by appendding -WA (word-aggregation) to the name of the model (see Fig. (b)).
W +- reduce(Hk W, 0,,g)
5

---

We examine these variants to study the effect of letting word-tokens directly interact with object- through the self-atee self-attention layers vs simply aggregating all language tokens single embedding and letting this vector condition the processing of observations. The latter is commonly done in using the language embedding in FiLM layers [d for instance. Finding significant effect here would encourage using architectures which allow direct interactions between the word tokens and the objects they refer to.
the aunuuag--condttione RI. and language grounding literature [ 25 37
LSTM Baselines. We also compare some LSTM-based baselines on this task; their architecture is described in more detail Supplementary Section B.3
2.4 Data Generation, Training and Testing Procedures
We use a bot generate the episodes we train on. The data collected consists of 56837 trajectories of T = 30 time steps. Among the traces some descriptions are less frequent than others but we make sure to have least 50 traces representing each of the 2672 descriptions we consider. We record the observed episodes and sentences in a buffer, and when training a model we sample (S, r) tuples with one observation coupled with either a true sentence from the buffer or another false sentence generated from the grammar. More details about the data generation can be found in Supplementary Section B.I
For each of the Transformer variants (6 models) and the LSTM baselines (2 models) we perform an hyper parameter search using 3 seeds in order to extract the best configuration. We extract the best condition for each model by measuring the mean Fi on a testing set made of uniformly sampled descriptions from each of the categories define in section 202 We use the Fi score because testing sets are imbalanced (the number of traces fulfilling each description is low). We then retrain best configurations over 10 seeds and report the mean and standard deviation (reported as solid black lines in Fig. 3 and Fig. 4 of the averaged Fi score computed on each set of sentences. When statistical significance is reported in the text, is systematically computed using a two-tail Welch's 's t-test with null hypothesis μ1 μ2, at level a = 0.05 [33. Details about the training procedure and the hyper parameter search are provided in Supplementary Section B.4
3 Experiments and Results
3.1 Generalization abilities of models on non-systematic split by categories of meaning
In this experiment, we perform study of generalization to new sentences from known observations. We divide our set of test sentences in four categories based on the categories of meanings listed in Section 2.2 Basic, Spatial, Opatio-Temponn and Temporal. We remove 15% of all possible sentences in each category from the train set and evaluate the Fl score those sentences. The results are provided in Fig.
First, we notice that over all categories of meanings, all UT and TFT models, with or without word- aggregation, perform extremely well compared the LSTM baselines, with all these four models achieving near-perfect test performance on the Basic sentences, with very little variability across the 10 seeds. We then notice that all SFT variants perform poorly on all test categories, in line worse than the baselines. This is particularly visible on the spatio-temporal category, where the SFT models perform at 0.75 + 0.020 whereas the baselines perform at 0.80 + 0.019. This suggests that across tasks, it is harmful to aggregate each scene plus the language information into a single vector. This may be due to the fact that objects lose their identity in this process, since information about all the objects becomes encoded in the same vector. This may make it difficult for the network to perform computations about the truth value of predicate on a single object.
Secondly. we notice that the word word-aggregation condition seems to have little effect on the performance on all three Transformer models. We only observe significant effect for UT models on spatio- temporal concepts (p-value = 2.38e-10) This suggests that the meaning of sentences can be adequately summarised by single vector; while maintaining separated representations for each object is important for achieving good performance it seems unnecessary to do the same for linguistic input. However we notice during our hyperparameter search that our -WA models are not very robust to hyperparameter choice, with bigger variants more sensitive to the learning rate.

---

Thirdly, we observe that for our pest-performing models, the basic categories of meanings are the easiest, with a mean score of 1.0 + 0.003 across all UT and TFT models, then the spatial ones at 0.96 + 0.020, then the temporal ones at 0.96 = 0.009, and finally the spatio-temporal ones at 0.097. This effectively suggests, as we hypothesised, that sentences containing spatial relations or temporal concepts are harder to ground than those who do not.
Known sentences with novel observations. We also examine the mean performance of our models for sentences in the training set but evaluated on set of new observations: we generate a new set of rollouts on the environment, and only evaluate the model on sentences seen at train time (plots are reported in Supplementary Section D) We see the performance is slightly better in this case, especially for the LSTM baselines (0.82 0.033 versus 0.79 = 0.032), but the results are comparable in both cases, suggesting that the main difficulty for models lies in grounding spatio-temporae meanings and not in linguistic generalization for the type of generalization considered in this section.
Models UT
FACTORDD
SFT
TFT
LSTM-FUAT
UTMWN
SFRWWN
TTPAA
1.0
0.8
0.6
0.4
0.7
L.Basic
Spatial Sets of descriptions grouped by concepts
3. Temporal
Spatio temporal
Figure 3: F1 scores for all the models on randomly held-out sentences. Fi is measured on separated sets representing each category of concepts defined in Section 2.2
3.2 Systematic generalization withheld combinations of words
In addition to the previous generalization studies, we perform experiment in harder linguistic generalization setting where we systematically remove binary combinations in our train set. This is in line with previous work systematic generalization on deep learning models 20 five test sets to examine the abilitics of our models to generalize on binary combinations of words that have been systematically removed from the set of training sentences, but whose components have been seen before in other contexts. Our splits can be described by the set of forbidden combinations of words as:
37 26
We create
1. Forbidden object-attribute combinations. remove from the train set all sentences contain- ing 'red cat', "blue door' and 'green cactus'. This tests the ability of models to recombine known objects with known attributes;
2. Forbidden predicate-object combination. remove all sentences containing 'grow' and all objects from the "plant' category. This tests the model's ability to apply known predicate to a known object in a new combination;
3. Forbidden one-to-one relation. remove all sentences containing 'right of'. Since the "right" token is already seen as-is in the context of one-to-all relations ( ('right most") ) and other one-to-one relations are observed during training, this tests the abilities of models to recombine known directions with in a known template;
4. Forbidden past spatial relation. remove all sentences containing the contiguous tokens 'was left of This tests the abilities of models to transfer a known relation to the past modality, knowing other spatial relations the past;
5. Forbidden past predicate. remove all sentences containing the contiguous tokens 'was grasp". This tests the ability of the model to transfer a known predicate to the past modality, knowing that it has already been trained on other past-tense predicates.
7

---

To avoid retraining all models for each split, we create one single train set with all forbidden sentences removed and we test separately on all splits. We use the same hyperparameters for all models than in the previous experiments. The results are reported in Fig.
Models
LSTN-ALCOORED
UT
SFT
TFT TFTANA
LSTH-
UTWA
SFT
1.0
0.0
0.6
0.2
0.0
-.hhect-wttribute
2. Prodraat-chjeett
One-fo-dnc relation
4. Post spatial relutton
5. Past prodicate
Types of Systematic Generalization
Figure 4: F1 scores of all the models on systematic generalization splits. Fj is measured on separated sets representing each of the forbidden combinations of word defined above.
First we can notice that the good test scores obtained by the UT and TFT models on the previous sections are confirmed in on this experiment: they are the best performing models overall. We then notice that the first two splits, corresponding to new attribute-object and predicate-object combinations, are solved by the UT and TFT models, while the SFT models and the LSTM baselines struggle to achieve high scores. For the next 3 splits, which imply new spatial and temporal combinations, the scores overall drop significantly; we also observe much wider variability between seeds for each model, perhaps suggesting the various strategies adopted by the models to fit the train set have very different implications in terms of systematic generalization spatial and temporal concepts. This very high variability between seeds on systematic generalization scores are reminiscent of the results obtained on the gSCAN benchmark
Additionally, for split 3, which implies combining known tokens to form a new spatial relation, we observe significant drop in generalization for the word-aggregation (WA) conditions, consistent across models (on average across seeds, 0.14 0.093, -0.15 0.234 and -0.20 + 0.061 for UT, SFT and TFT resp. with p-values < 1e-04 for UT and SFT). This may be due to the fact that recombining any one-to-one relation with the known token right seen the context one-to-all relations requires a separate representation for each of the linguistic tokens. The same significant drop performance for the WA condition can be observed for UT and TFT split 4, which implies transferring a known spatial relation to the past.
However, very surprisingly, for split 5 which implies transposing the known predicate grasp the past tense we observe a very strong effect in the opposite direction: the WA condition seems to help generalizing to this unknown past predicate (from close-to-zero scores for all transformer models, the WA adds average 0.71 = 0.186, 0.45 + 0.178 and 0.52 + 0.183 points for UT, ST and TT resp. and p- p-values< 1e-05). This may be due the fact that models without WA learn a direct and systematic relationship between the grasp token and grasped objects, indicated in their features; this relation not modulated by the addition of the was modifier as a prefix to the sentence. Models do not exhibit the same behavior split 4, which has similar structure (transfer the relation lest of to the past). This may be due to the lack of variability in instantaneous predicates (only the grasp predicate); whereas there several spatial relations (4 one-to-one, 4 one-to-all).
Control experiment. We evaluate previously trained models on test set containing hard negative examples. The aim of this experiment is to ensure that models truly identify the compositional structure of our spatio-temporal concepts and do not simply perform unit concept recognition. We select negative pairs (trajectory, description) so that the trajectories contain either the object or the action described in the positive example. Results are provided in Fig. We observe a slight decrease of performances on all 5 categories (drop is less than 5%), demonstrating that the models do in fact represent the meaning the sentence and not simply the presence or absence of a particular object or predicate.
8

---

Models
LSTH-FLAT
1.0
0.5
0.4
0.2
0.0
1. -.jject-ttrrae eitribtte
2. Peedicele-cbject
One-to relation Types Systematic Generalization
Pest spetial reletion
5. Past, predicate
Figure 5: Control experiment: F1 scores for all the models on systematic generalization splits in the hard negative examples setting.
Related Work
The idea that agents should learn to represent and ground language in their experience of the world has a long history in developmental robotics (50
39 40
7 and was recently extended in the context
of Language Conditioned Deep Reinforcement Learning
3I 3. These recent approaches 22 tasks and are always using instructive language. Meanings typically refer to instantaneous actions and rarely consider spatial reference to objects (35. Although our environment includes object manipulations, we here tackle novel categories of meanings involving the grounding spatio-temporal concepts such as the past modality complex spatio-temporal reference to objects.
22
often consider navigation [T0 g or object manipulation [
We evaluate our learning architectures on their ability to generalise sets of descriptions that contain systematic differences with the training data so as to assess whether they correctly model grammar primitives. This procedure is similar to the gSCAN benchmark (37. This kind of compositional generalisation is referred as 'systematicity" by Hupkes et al. (26]. Environmental drivers that facilitate systematic generalization are also studied by Hill et 123 Although Hupkes al. 26 consider Language Grounding task. Ruis al. 3 consider an Embodied Language Grounding setup involving one form of time- extended meanings (adverbs), but do not consider the past modality and spatio-temporal reference to objects, and do not consider learning truth functions. Also, they do not consider learning architectures that process sequences of sensorimotor observations. To our knowledge. no previous work has conducted systematic generalization studies on an Embodied Language Grounding task involving spatio-temporal language with Transformers.
relational models in their work, they do not evaluate their performance
The idea that relational architectures are relevant models for Language Grounding has been previously explored in the context of Visual Reasoning. They were indeed successfully applied for spatial reasoning the visual question answering task CLEVR 38). With the recent publication of the video reasoning dataset CLEVRER 47, those models were extended and demonstrated abilities to reason over spatio-temporal concepts, correctly answering causal, predictive and counterfactual questions study, these works around CLEVRER do not aim to analyze ppatio-tempordl
[66. In contrast language and therefore do not consider time-extended predicates or ppatio-temporal reference to objects in their language, and do not study properties of systematic generalization over sets of new sentences.
5 Discussion and Conclusion
In this work, we have presented a first step towards learning Embodied Language Grounding of spatio-temporal concepts, framed as the problem learning a truth function that can predict if given sentence is true of temporally-extended observations of an agent interacting with collection of objects. We have studied the impact of architectural choices on successful grounding of our artificial spatio-temporal language. We have modelled different possible choices for aggregation of

---

observations and language as hierarchical Transformer architectures. We have demonstrated that in our setting. is beneficial process remporally-exteer extended observations and language tokens side-by- side, evidenced by the good score of our Unstructured Transformer variant. However, there seems aggregating temporal observations along the temporal
to be only minimal effect on performance dimension first - compared to processing all traces and the language in an unstructured manner as long as object identity is preserved. This can inform architectural design in cases where longer episode lengths make it impossible to store all individual timesteps for each object; our experiments provide evidence that a temporal summary can be used in these cases. Our experiments with systematic dimensions of generalization provide mixed evidence for the influence of summarizing individual words into a single vector, showing it can be detrimental to generalize to novel word combinations but also help prevent overgeneralization of relation between single word and single object without considering the surrounding linguistic context.
Limitations and further work. There several limitations of our setup which open important opportunities for further work. First, we have used a synthetic language that could be extended: for instance with more spatial relations and relations that are more than binary. Another axis for further research is using low-level observations. In our setting, we wanted to disentangle the effect of structural biases on learning spatio-temporal language from the problem of extracting objects
30 g in a consistent manner over time (object permanence needed, and it could allow to define richer attributes (related to material or texture) and richer temporal predicates (such as breaking, floating, etc). Finally, we use a synthetic language which is far from the richness of the natural language used by humans, but previous work has shown that natural language can be projected onto the subspace defined by synthetic language using the semantic embeddings learned by large language models [3 this opens up be a fruitful avenue for further investigation.
from low level observations 6 21
17
15
Further steps in this direction
A further interesting avenue for future work would be to use the grounding provided by this reward function to allow autonomous language-conditionel agents to target their own goals [14. In this sense, the truth function can be seen as goal-achievement function reward function. While generalization performance of our method is not perfect, the good overall fl scores of our architectures imply that they can be directly transferred a more complete RL setup to provide signal for policies conditioned on spatio-temporal language.
Broader Impact. This work provides step in the direction of building agents that better understand how language relates to the physical world; this can lead to personal robots that can better suit the needs of their owners because they can be interacted with using language. If successfully implemented. this technology can raise issues concerning automation of certain tasks resulting in loss of jobs for less-qualified workers.
Links. The source code as well as the generated datasets can be found at httph:tt/////t /tttp://github... com/ temporal -laggugge transformers
Acknowledgments and Disclosure of Funding
Tristan Karch is partly funded by the French Ministère des Armées - Direction Générale de I' l'Armement. Laetitia Teodorescu supported by Microsoft Research through its PhD Scholar- ship Programme. This work was performed using HPC resources from GENCI-IDRIS (Grant 2020-40091011996
References
[1] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer. Mohamed Chetouani, and Olivier Sigaud. Grounding Language to Autonomously-Acquree Skills via Goal Generation. In ICLR 2021- Ninth International Conference on Learning Representation, Vienna Virtual, Austria, May 2021. http:tss: fnriaa.fr/hal-01221111
[2] James F. Allen. Towards general theory of action and time. Artificial Intelligence, 23(2): 123-154, 1984. ISSN 0004-3702 doi: http:://wooooogo//....////041010
URI.
10

---

[3] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. In International Conference on Learning Representations, 2019. URL https: cestps://openrevieee wttprrrvviiwwee//frrrere net/forum? id=HixsSjC9Ya
[4] John A. Bateman, Joana Hois, Robert Ross, and Thora Tenbrink. A linguistic ontology of space for natural language processing. Artificial Intelligence. 172414):1027-1071,, 2010. ISSN 0004-
3702. doi: https://doi.org/100. 1016/j wwt.:/i....tttt2111..... art.iitt 00.0000.000........0.000 URL http://wwww.cieeceerrrrr...
com httpa//c//errrtaa///////////////
[5] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo. Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning. and graph networks, 2018.
[6] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion, 2019.
[7] Angelo Cangelosi, Giorgio Metta, Gerhard Sagerer, Stefano Nolfi, Chrystopher Nehaniv, Kerstin Fischer, Jun Tani, Tony Belpaeme, Giulio Sandini, Francesco Nori, Luciano Fadiga, Britta Wrede, Katharina Rohlfing, Elio Tuci, Kerstin Dautenhahn, Joe Saunders, and Amne Zeschel. Integration of action and language knowledge: A roadmap for developmental robotics. IEEE Transactions on Autonomous Mental Development, 2(3):167-195, 2010. doi: 10.1 109/TAMD 2010.2053034.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
[9] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. inteddattention architectures for misk-oriented language grounding, 2018.
[10] David L. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI'11, page 859-865. AAAI Press, 2011.
[11] Maxime Chevalier-Boisee Boisverr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform study the sample efficiency of grounded language learning, 2019.
[12] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 225-232. 2020. doi: 10.1109//S114700. 2020.9308603.
[13] Cédric Colas, Olivier Sigaud, and Pierre-Yver Oudeyer. A hitchhiker's guide statistical comparisons of reinforcement learning algorithms, 2019.
[14] Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Fries Pe- ter Ford Dominey, and Pierre Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020.
[15] Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanbur,, Marta Garnelo, and Murray Shanahan. Alignnet: Unsupervised entity alignment, 2020.
[16] David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architec- tures, 2020.
11

---

[17] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations, 2020.
[18] Dedre Gentner and Jeffrey Loewenstein. Relational language and relational thought, 2002.
[19] James J. (James Jerome) Gibson. The senses considered as perceptual systems. Allen Unwin, London, 1968.
[20] Arthur M. Glenberg and Michael Kaschak. Grounding language action. Psychonomic Bulletin & Review, 93)):55--565, September 2002. ISSN 1531-5320. doi: 10.3758/
BF03196313. URL httpp:t:/t.ooooo///////o 0tt.//00.7758//F3196333
[21] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference, 2020.
[22] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps. Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world, 2017.
[23] Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent, 2020.
[24] Xavier Hinaut, Maxime Petit, Gregoire Pointeau, and Peter Ford Dominey. Exploring the acquisition and production of grammatical constructions through human-robor interaction with echo state networks. Frontiers neurorobotics, 8:16, 2014.
[25] David Yu- Yu-Tung Hui, Maxime Shevalier-Boisver, Dzmitry Bahdanau, and Yoshua Bengio. Babyai 1.1, 2020.
[26] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise?, 2020.
[27] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning, 2016.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[29] Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the composi- tional skills sequence to-nicee----eunncee recurrent networks, 2018.
[30] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
[31] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language, 2019.
[32] Olivier Mangin, David Filliat, Louis Ten Bosch, and Pierre- Yves Oudeyer. Mca-nmf: Multi- modal concept acquisition with non-negative matrix factorization. PloS one, 10(10):e0140732, 2015.
[33] Alana Marzoev, Samuel Madden, M. Frans Kaashock, Michael Cafarella, and Jacob Andreas. Unnatural language processing: Bridging the gap between synthetic and natural language data, 2020.
[34] Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro Dudík, and Patrick Shafto, Interactive learning from activity description, 2021.
12

---

[35] Rohan Paul, Jacob Arkin, N. Roy, and T. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Robotics: Science and Systems, 2016.
[36]
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer, 2017.
[37] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A benchmark for systematic generalization in grounded language understanding. 2020.
[38] Adam Santoro, David Raposo. David G. T. Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning, 2017.
[39] I.. Steels. Semiotic dynamics for embodied agents. IEEE Intelligent Systems, 21(3):32-38 2006. doi: 10.1 10910099...00 2006.58..
[40] Yuuya Sugita and Jun Tani. Learning semantic combinatoriality from the interaction between linguistic and behavioral processes. Adaptive behavior, 13(1):33-52, 2005.
[41] Jun Tani. Exploring robotic minds: actions, symbols, and consciousness as self self-organizing dynamic phenomena. Oxford University Press, 2016.
[42] Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh. Symbol emergence in robotics: a survey. Advanced Robotics, 30(1)-12): 706-728, 2016.
[43] Thora Tenbrink. Space, Time, and the Use of Language: An Investigation Relationships. De Gruyter Mouton, 2008. ISBN 978-3-11-01188229, doi: do111001511//983111199988 URL https://doi.org /1tp:///...1111/1111111111111110
[44] Thora Tenbrink. Reference frames of space and time in language. Journal of Pragmatics, 43(3): 704-722, 2011. ISSN 0378-2166. doi: httpa//tww......c..cc..c..aaaaaa
URL The
https://www.. http://wwww.cceenceeeerrtt.
Language of Space and Time.
[45] Elio Tuci, Tomassino Ferrauto, Ame Zeschel, Gianluca Massera, and Stefano Nolfi. An experiment on behavior generalization and the emergence of linguistic compositionality in evolving robots. IEEE Transactions on Autonomous Mental Development, 3(2):176-189, 2011.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
[47] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, 2020.
[48] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizii, Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In International Conference on Learning Representations, 2021. URL https //openreview.o Copparrerieev.eetffruuk.aaaaeaee ret:/forum?iddMM22f//bbbb
[49] Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals, 2020.
[50] Rolf A. Zwaan and Carol J. Madden. Embodied Sentence Comprehension, page 224-245. Cambridge University Press, 2005. doi: 10.0017/CH097701114499996
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes], we propose a summary of our results supporting our
contributions in Section 5
(b) Did you describe the limitations of your work? [Yes], in the concluding Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes], in a the
Broader Impact paragraph in the Conclusion section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes], the code is given supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]. see Sections B.] and 3.2
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes], all reported scores were averaged over 10 seeds and std is reported in all plots (Fig- 3 Fig. of Section 3 and in the main text. We also performed statistical tests to measure significance.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] details are given in Supplementary Section D.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "y1NqLjFZaC", "reviewer": "Reviewer_Zjb3", "summary": "The study presents a new synthetic dataset for the Embodied Language Grounding task, which specifically targets spatio-temporal aspects of language. The authors further test a number of transformer-based models on the task, identifying a number of architectural properties that are likely to be conducive to specific types of generalization (three such types were considered).\n\nThe paper strikes me as a well-rounded and a high-quality contribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The paper addresses a highly relevant problem\n- The paper is sufficiently novel and original\n- The paper is well written and is a pleasure to read\n- Experiments are thorough and sufficient to support the claims being made\n\n## Weaknesses\n\n- The paper focuses on a simulated dataset, which limits direct practical applications\n- The models seem to achieve near-ceiling performance in a number of conditions\n\nOverall, I enjoyed reading this paper and I believe that it is a strong contribution that may aid in the challenging and important task of Language Grounding.\n\nThe main concern I have is that the proposed dataset may be too easy to be a \"driving force\" behind architecture development. In the most challenging conditions, at least some of the models considered in the paper already seem to reach near-ceiling performance. \n\nI believe that expanding the artificial language so as to be able to present an even more trying challenge would have made the paper noticeably stronger. Ideally, a simulated dataset should offer a continuum of tasks (from extremely easy to extremely difficult). \n\nThis concern is not too crucial, since there are conditions that are not yet solved, and hence there is room for improvement in model development. Hopefully, by the time the dataset is solved completely, its modified version will be developed and released.\n\nRegardless of the concern voiced above, I believe that the experimental results will be interesting for a large number of researchers, and that the dataset, although not always challenging enough, may still be a good starting point for development of grounded linguistic agents.\n\n## Typos\n\n362 on -> own\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Broader impact and limitations are discussed in sufficient depth.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "whN_CoX-94", "reviewer": "Reviewer_xTkz", "summary": "In this paper, the authors introduce a spatio-temporal language grounding task. In their setting, the agent interacts with the surrounding objects over time. Once the episode is over, an oracle provides exhaustive feedback in a synthetic language about everything that has happened. They use the transformer model to solve the task. They also test the generalization ability of their models, including the generalization to randomly held-out sentences and the generalization to grammar primitives.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is good that the paper considers the spatial-temporal language descriptions in the language grounding tasks. However, the motivation is not clear and the claimed contributions are minor.\n\nThe authors claim that they \"study the grounding of new spatio-temporal concepts enabling agents to ground time extended predicates with complex spatio-temporal references to objects and understand both present and past tenses\". This sounds confusing. As I know, there are some language grounding papers that involve both spatial and temporal concepts. See detailed limitations below.\n\nThe task definition is not clear. What is the difference between the proposed Embodied Language Grounding task from existing works? What is the advantage of the new task? Why is it more challenging than existing language Grounding tasks?\n\nThe details of how to get the trajectories and languages are missing. See detailed limitations below. \n\nFurthermore, if you can generate linguistic descriptions given trajectories by the oracle function, it seems you can also get the trajectories given the linguistic descriptions using scripts, especially that the environment is very simple and not diverse. It seems not hard to hand-code the action sequences given some linguistic descriptions. Then why do you still need to train a model for language grounding? If it is impossible to get the action sequences by hand-coding, could you explain why?\n\nIt is good that the authors show the generalization experiments. Is it because of the Transformer model? If so, what's the novelty of your method? The good generalization ability of Transformer has been shown in many papers.\n\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The motivation is not clear and the claimed contributions are minor.\nThe authors claim that they \"study the grounding of new spatio-temporal concepts enabling agents to ground time extended predicates with complex spatio-temporal references to objects and understand both present and past tenses\". This sounds confusing. As I know, there are some language grounding papers that involve both spatial and temporal concepts, such as \n\n-- ALFRED A Benchmark for Interpreting Grounded Instructions for Everyday Tasks\n-- Are We There Yet? Learning to Localize in Embodied Instruction Following\n-- FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning\n-- Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments\n-- IQA: Visual Question Answering in Interactive Environments\n-- PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World\n\nThe Problem Definition section and Figure 1 are confusing. \nHow did you get the synthetic linguistic descriptions? How does the oracle provide exhaustive feedback?\nThe tasks are very simple. It seems that the linguistic descriptions are script generated and are not diverse enough.\nHow did you control that the generated trajectories make sense? What if the agent repeats one action or alternatively executes two actions? It is unclear how do you get good trajectories and how does the oracle work.\n\nThe writing needs to be improved.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "pOWdcMTdy9t", "reviewer": "Reviewer_3aA5", "summary": "This paper provides a study of spatio temporal language grounding. It introduces a new environment called the Temporal Playground that generates synthetic language descriptions of the traces of the actions performed by an embodied agent. Although the relative search space of synthetic language is limited, it provides a set of experiments to demonstrate model generalization to aspects like unseen word combinations. A study of variations on transformer architectures for this task is also provided. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe motivation is quite clear, lots of vision+language sequential decision making problems but not many have really focused on the spatio-temporal aspects of the language being used.\n\nThe paper is clearly written and I could follow along at all steps, experiment and architecture choices are clearly defined.\n\nThe study of providing temporal bias first vs spatial bias first in the transformer architecture (as opposed to just the unstructured transformer) is potentially valuable to other domains involving sequential decision making.\n\n\nCons/Questions:\n\nThe selection of which predicates involve temporal concepts and which don't seem rather arbitrary. A notion of time units in terms of steps could perhaps help here. Else one could make the argument that an action such as \"grasp\" is also a temporal concept in addition to being a spatial one in all cases (and not only when presented as a past tense predicate).\n\nMany of the conclusions regarding the linguistic components being tested (e.g. lines 230-231) are likely to be complicated by factors such as the size of the synthetic vocabulary and the number of novel descriptions. Even within the limited vocab used, an ablation showing performance of the models on the experiments such as within Section 3.1 as a function of the size of the synthetic vocab used would be critical towards validating such claims.\n\nSimilarly to the above concern, the tests of systematic generalization on withheld combinations of words only tests on a one to three forbidden training samples. A much more useful experiment here would be to ablate this as a function of the number of forbidden training word combination (1, 5, 10...). As it is, it is difficult to tell whether the models are truly generalizing to these unseen combinations or simply finding proxy features in the surrounding text.\n\n(Minor) A zeroshot eval on description traces from \"real language\" (something like the Room2Room dataset https://bringmeaspoon.org/ or existing prodedural text understanding datasets) would be very useful as a sanity check. I'm not asking for super high performance, just that it would better ground a reader's judgement on how the synthetic eval numbers match up when evaluated on a larger language observation space.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Though plenty of downstream application based concerns exist for such embodied agents, at its current stage - transfer to them is not yet possible. There is thus limited societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns for this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BGxtXHw5cY2", "reviewer": "Reviewer_P8h7", "summary": "This paper proposes a new dataset for reasoning about vision, language, and time. The dataset (Temporal Playground) consists of several objects moving around an image paired with a sentence that may or may not describe the image. The task is to determine whether the description is true or false about the moving image. The data is synthetically generated.\n\nThe paper also proposes several architectures to solve this task and dataset, and experiments with several settings including how the inputs are combined and how to represent sentences in the model.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: As mentioned in this paper, there are many datasets that describe video captions on real videos (e.g., HowTo100M of Miech et al. 2019), and datasets which are completely synthetically generated based on CLEVR (e.g., CLEVRER of Yi et al. 2020) that focus on image reasoning. I don't understand the distinction between CLEVRER and this proposed dataset, as CLEVRER requires spatial and temporal reasoning, and though I'm not familiar with all of the evaluation splits of the dataset, one could imagine resplitting it to evaluate different kinds of generalization (e.g., unseen sentences/questions). It isn't clear what this proposed dataset adds that existing datasets don't already have. Since the task is fairly specific, I would like to see a more in-depth discussion on the possible applications of this data (and what if offers over existing datasets). \n\nThe experiments comparing SFT and TFT could have wider applicability beyond this dataset to other video processing work. I'm not very familiar with existing work in video processing, and not sure if this kind of experiment has been done before.   \n\nQuality: The dataset is synthetically generated, so it is not clear how work on this data will extend to natural language (is the \"Natural English version\" in Figure 1 ever used?). The performance on the dataset is already very impressive, and I would have liked to see more detailed error analysis about what the models are failing on. The motivation behind some of the choices in the dataset (e.g., the number of object types and properties, including interaction between object types, etc.) is not well-stated. \n\nResults showing a small difference between using and not using WA don't seem very meaningful here, because the language is relatively simple and synthetically generated (i.e., the conclusion that language can be summarized into a single vector wouldn't necessarily hold in the real world). It would be interesting to see baselines that provide only one of the two inputs (video or text, not both) to the model. \n\nThe claim that \"Meanings typically refer to instantaneous actions and rarely consider spatial reference to objects\" is not true. There are plenty of challenging grounded instruction-following datasets that require resolving complex spatial relationships between objects and following long, ordered instructions, e.g., Touchdown (Chen et al. 2019). \n\nClarity: The paper is relatively easy to understand. The intro was fairly low-level near the end (not sure what a \"grammar primitive\" is in this context). I was confused about what the \"agent\" was in Section 2.2 -- how was this controlled? I would have preferred to see the equations in lines 177--178 written in standard matrix operation notation rather than a pytorch-like syntax.\n\nSignificance: The results on SFT vs. TFT may be applicable to other systems which process video data. Other researchers may use this dataset in addition to similar datasets like CLEVRER. It's not clear to me the main contribution of this work over existing works, and I worry about its limitations as a synthetically generated dataset.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the main limitation (in my opinion) which is the synthetic nature of the dataset. \n\nThe societal impact of the work is briefly discussed. I don't have specific suggestions for it, but at the very least I would like a deeper discussion about what this dataset in particular offers over existing ones for real-world applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the task of predicting truthfulness of a summary statement given an agent’s behavioral trace in an environment, as a way of measuring a model’s understanding of spatio-temporal concepts. The reviewers had differing opinions about the paper — while they all agreed that the task of spatio-temporal grounding is interesting and impactful to solve, there were several concerns around the motivation of the paper being unclear, the synthetic nature of the task and comparison to existing benchmarks like ALFRED, Room2Room, etc. The author response addressed some questions around the motivation, but the question of how this (synthetic) benchmark would connect to more realistic settings remains unclear. One of the reviewers even suggested re-using data from slightly less synthetic tasks like ALFRED as a step towards demonstrating this connection. On reading the paper myself, I think that the paper has good ideas, but feels too preliminary and in the current form, leaves unanswered important questions such as the comparison with existing datasets and how developing models on this may correlate with real-world tasks (for example, the TFT models are almost at 100% F1 score which begs the question of whether the task is too easy to test all aspects of spatio-temporal grounding). 
Decision: Accept (Poster)