Grounding Spatio-Temporal Language with Transformers
Authors: Tristan Karch
Laetitia Teodorescu
Katja Hofmann
Clément Moulin-Frier
Pierre-Yves Oudeyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Language Grounding
Embodied Autonomous Agents

Abstract: Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to new sentences, 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents.


Grounding Spatio-Tempora, Language with Transformers
Tristan Karch Laetitia Teodorescu"
Katja Hofmann Microsoft Research Cambridge, UK
Clément Moulin-Frier Inria Flowers team Université de Bordeaux ENSTA ParisTech
Inria Flowers Team Université de Bordeaux firstname. last lastname@inria.ft
Pierre-Yver Oudeyer
Inria Flowers team Université de Bordeaux ENSTA Paris Tech
Abstract
Language is an interface to the outside world. In order for embodied agents it, language must be grounded in other, sensorimotor modalities. While there is extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts still largely uncharted. To make progress in this direction, here introduce novel -patio-temporal language grounding task where the goal is to learn the meaning of descriptions of behavioral traces of embodied agent. This achieved by training a truth function that predicts a description matches a given history observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role architectural biases this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on classes of generalization: 1) generalization to randomly held-out sentences; 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guide guided autonomous embodied agents. We also release our code under open-source license well as pretrained models and datasets to encourage the wider community to build upon and extend our work in the future.
Introduction
Building autonomous agents that learn represent and use language is long standing-goal in Artificial Intelligence 20 39 In developmental robotics 77, language considered a cornerstone of development that enables embodied cognitive agents to learn more efficiently through social interactions with humans or through cooperation with other autonomous agents. It is also considered essential to develop complex sensorimotor skills, facilitating the representation of behaviors and actions.
Embodied Language Grounding [50] is the field that studies how agents can align language with their behaviors in order to extract the meaning of linguistic constructions. Early approaches in
Equal Contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) Grow action with spatial or attribute reference to object
(b) Shoke action with spatio temporal reference to object
(c) Grasp past actions description with
spatio-temporal reference to object
shake
Traces Interactions
Sampled descriptions
Grow blue chameleun
SShaee red cat'
"Was graxp blue cactus'
Grow thing left tf
Shaae thing right of lamp'
"Was grosp thing top most"
Grow thing left moxt"
SShake thing was left of lamp'
'Was grasp dhing was right of parret
Grrasp green water"
Natural
English version
'You are currently growing the blue chamellon"
Youu are currently shaking some- thing that used to be at the left of the lomp'
"Yuu previously grasped some- thing that used to be at the right of the parrot'
Figure 1: Visual summary of the Temporal Playground environment: At each episode (column b and c). the actions of an agent (represented by hand) unfold in the environment and generate trace of interactions between objects and the agent body. Given such a trace, the environment automatically generates a set of synthetic linguistic deseriptions that are true at the end of the trace. In (a) the agent grows an object which is described with spatial (underlined) or attribute (highlighted) reference. In (b) it shakes an object which is described with attribute, spatial spatio-temporal (underlined) reference. In (c) it has grasped an object (past action underlined) which is described with attribute, spatial or spatio-temporal (highlighted) reference. The natural english version is given for illustrative purposes.
developmental robotics studied how various machine learning techniques, ranging from neural networks 44 45 24] to non-negative matrix factorization [3), could enable the acquisition of
grounded compositional language 42 41. This line of work was recently extended using techniques
for Language conditioned Deep Reinforcement Learning (1.. Among these works we can distinguish mainly three language grounding strategies. The first one consists of directly grounding language in the behavior of agents by training goal-conditioned policies satisfying linguistic instructions The second aims at extracting the meaning of sentences from mental simulations (i.c. generative models) of possible sensorimotor configurations matching linguistic descriptions
40
45
23
22
The third strategy searches learn the meaning of linguistic constructs terms of outcomes that agents can observe in the environment. This is achieved by training truth function that detects if descriptions provided by an expert match certain world configurations. This truth function can be obtained via Inverse Reinforcement Learning 4u
32
3 or by training a multi- binary classifier 14. Previous work [4 3 has shown that access to this reward is enough for sucessfully grounding language in instruction followin agents.
While all the above-mentioned approaches consider language that describes immediate and instanta- neous actions, we argue that is also important for agents to grasp language expressing concepts that are relational and that span multiple time steps. We thus propose to study the grounding of new spatio-temporal concepts enabling agents to ground time extended predicates (Fig. [p) with complex spatio-temporal references objects (Fig. (D)) and understand both present and past tenses (Fig. I)) To do so we choose the third strategy mentioned above, i.e. to train a truth function that predicts when descriptions match traces of experience. This choice motivated by two important considerations. First, prior work showed that learning truth functions was key to foster generalization B, enabling agents to efficiently self-train policies via goal imagination [E]] and goal relabeling [12]. Hence the truth function is an important and self-contained component of larger learning systems. Second, this strategy allows to carefully control the distribution of experiences and descriptions perceived by the agent.
2

---

The Embodied Language Grounding problem has a relational structure. We understand the meaning of words by analyzing the relations they state in the world [ The relationality of spatial and temporal concepts has long been identified in the field of pragmatics
(see Supplementary Section C
for additional discussion). Furthermore actions themselves are relations between subjects and objects, and can be defined in terms of affordances of the agent [9 We acknowledge this and provide the right relational inductive bias [5] to our architectures through the use of Transformers [ We propose formalism unifying three variants of a multi-modal transformer inspired by Ding et al. 16 that implement different relational operations through the use of hierarchical attention. We measure the generalization capabilities of these architectures along three axes 1) generalization to new traces of experience; 2) generalization to randomly held out sentences; 3) generalization to grammar primitives, systematically held out from the training set as in Ruis et al. [77. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall. We also identify specifie relational operations that are key to generalize on certain grammar primitives.
Contributions. This paper introduces:
1. A new Embodied I Language Grounding task focusing on spatio-temporal language;
2. A formalism unifying different relational architectures based on Transformers expressed as a function of mapping and aggregation operations;
3. A systematic study of the generalization capabilities of these architectures and the identifica- tion of key components for their success on this task.
2 Methods
2.1 Problem Definition
We consider the setting of an embodied agent behaving in an environment. This agent interacts with the surrounding objects over time, during an episode of fixed length (7), Once this episode is over, an oracle provides exhaustive feedback in a synthetic language about everything that has happened. This language describes actions of the agent over the objects and includes spatial and temporal concepts. The spatial concepts are reference an object through its spatial relation with others (Fig. Ia), and the temporal concepts are the past modality for the actions of the agent (Fig. [e),, past modality for spatial relations (Fig. [T). and actions that unfold over time intervals. The histories of states of the agent's ' body and of the objects over the episode as well as the associated sentences are recorded in a buffer B. From this setting, and echoing previous work on training agents from descriptions, we frame the Embodied Language Grounding problem as learning parametrized truth function Ra over couples of observations traces and sentences, tasked with predicting whether a given sentence W is true of given episode history S or not. Formally, we aim to minimize:
[s,
r(,
where L denotes the cross-entropy loss and r denotes the ground truth boolean value for sentence W about trace S.
2.2 Temporal Playground
In the absence of any dedicated dataset providing spatio-temporal descriptions from behavioral traces of an agent, we introduce Temporal Playground (Fig. D an environment coupled with a templated grammar designed to study spatio-temporal language grounding. The environment is a 2D world, with procedurally-generaaee scene containing N = 3 objects sampled from 32 different object types belonging to 5 categories. Each object has continuous 2D position, a size, continuous color code specified by a 3D vector in RGB space, a type specified by one-hot vector, and a boolean unit specifying whether it is grasped. The size of the object feature vector (0) is |o] = 39. The agent's 's body has its 2D position in the environment and its gripper state (grasping or non-grasping) as features (body feature vector (b) of size |b| = 3). In this environment, the agent can perform various actions over the length (7) of an episode. Some of the objects (the animals) can move independently. Objects can also interact: if the agent brings food or water to an animal, will grow in size; similarly, if water is brought to plant, it will grow. At the end of an episode, generative grammar generates
3

---

sentences describing all the interactions that occurred. A complete specification of the environment as well as the BNF of the grammar can be found in Supplementary Section
Synthetic language. To enable a controlled and systematic study of how different types of spatio- temporal linguistic meanings can be learned, we argue is necessary to first conduct a systematic study with a controlled synthetic grammar. We thus consider synthetic language with a vocabulary of size 53 and sentences with maximum length of 8. This synthetic language facilitates the generation of descriptions matching behavioral traces of the agent. Moreover, it allows us to express four categories of concepts associated with specific words. Thus, the generated sentences consist in four conceptual types based on the words they involve:
Sentences involving basic concepts. This category of sentences talk about present-time events by referring to objects and their attributes. Sentences begin with the 'grasp' token combined with any object. Objects can be named after their category (eg. 'animal", 'thing' ) or directly by their type ('dog', 'door", 'algae', etc.). Finally, the color ( ('red', 'blue", green') of objects can also be specified.
Sentences involving spatial concepts. This category of sentences additionally involve one-to-one spatial relations and one-to-all spatial relations to refer to objects. An object can be 'left of another object (reference is made in relation to a single other object), or can be the 'top most' object (reference is made in relation with all other objects). Example sentences include graasp thing bottom of cal' or 'grasp thing right most'.
Sentences involving temporal concepts. This category of sentences involves talking about predicates and the past tense, without any spatial relations. The two temporal predicates are denoted with the words 'grow' and 'shake", The truth value of these predicates can only be decided by looking at the temporal evolution of the object's size and position respectively. A predicate is transposed at the past tense if the action describes was true at some point in the past and is no longer true in the present, this is indicated by adding the modifier 'was' before the predicate. Example sentences include 'was grasp red chameleon' (indicating that the agent grasped the red chameleon and then released it) and shake bush";
Sentences involving spatio-temporal concepts. Finally, we consider the broad class of spatio-temporal sentences that combine spatial reference and temporal or past-tense predi- cates. These are sentences that involve both the spatial and temporal concepts defined above. Additionally, there is case of where the spatial and the temporal aspects are entangled: past spatial reference. This happens when an object is referred to by its previous spatial relationship with another object. Consider the case of an animal that was at first on the bottom of a table, then moved top, and then is grasped. In this case could refer to this animal as something that was previously on the bottom of the table. We use the same 'was' modifier as for the past tense predicates; and thus we would describe the action as 'Grasp thing was bottom of table".
2.3 Architectures
In this section we describe the architectures used as well as their inputs. Let one input sample to model be I = (S, W), where (S,.))., represents the objects' and body's evolution, and (Wi), represents the linguistic observations. S has a spatial (or entity) dimension indexed by E [O...N] and temporal dimension indexed by t [1..T]; for any i, f, Si, is a vector of observational features. Note that by convention, the trace (So,e), represents the body's features, and the traces (Si,.),.ss represents the other objects' features. W is 2-dimensional tensor indexed by the sequence / E [1..L]: for any I, Wi &dw is one-hot vector defining the word in the dictionary. The output to models is a single scalar between O and 1 representing the probability that the sentence encoded by W is true in the observation trace S.
Transformer Architectures. To systematically study the influence of architectural choices on language performance and generalization in our spatio-temporal grounded language context, we define a set of mapping and aggregation operations that allows us to succinetly describe different models in unified framework. We define:

---

W
(c) Unstructured Transformer (UT)
$
Body Encoder
Object Encoder
Postioona Encodingg
W "Grusp thing left most"'
Language
Encoder
(d) Spatial Transformer (SFT)
(a) Input Encoding
reduce
redace
(h) Optional Wordl Aggregation (-WA)
(e) T'emporol-First Transformer (TFT)
Visual summary of the architectures used. We show the details of UT, SFT and TFT respectively in subfigures (c), (d), (e), as well as a schematic illustration of the preprocessing phase (a) and the optional word-aggregation procedure (b).
Figure 2:
An aggregation operation based a Transformer model, called reduce. reduce is parametrized function that takes 3 inputs: a tensor, a dimension tuple D over which to reduce and query tensor (that has have the size of the reduced tensor). R layers of Transformer are applied to the input-query concatenation and are then queried at the position corresponding to the query tokens. This produces an output reduced over the dimensions D. A casting operation called cast. cast takes as input 2 tensors A and B and a dimension d. A is flattened, expanded as to fit the tensor B in all dimensions except and concatenated along the d dimension.
A helper expand operation called expand that takes arguments tensor and integer and repeats the tensor n times.
Using those operations, define three architectures: Transformer, inspired by Ding et al. [16]. or UT); one with a spatial-first structural bias - objects and words are aggregated along the spatial dimension first (Spatial-Fine Transformer or SFT); and one with temporal-fire structural bias objects and words are aggregated along the temporal dimension first (Temporal-Fite -irst Transformer. TFT).
with particular bias (Unstructured
Before inputting the observations of bodics and objects S and the language W into any of the Trans- former architectures, they are projected more details). A positional encoding 46 is then added along the time dimension for observations and along the sequence dimension for language; and finally one-hot vector indicating whether the vector observational or linguistic is appended at the end. This produces the modified diservation-language tuple ($, W). We let:
common dimension (see Supplementary Section B.2 for
UT UT ($, W) == reduce(cast($, W, 0.g).
SFT W.q) == reduce (reduce ( castuucccccattti// $, 0, expand T)), 0)
TF W, W.4)= reduce(reducel cwweeeeaastattaaccaaacca Š, 1), 1, expand N + 1)), 0,
where 7' is the number of time steps, N is the number of objects and g a learned query token. See Fig. 2 for an illustration of these architectures.
Note that SFT and TFT are transpose versions of each other: SFT is performing aggregation over space first and then time, and the reverse true for TFT. Additionally, we define a variant of each of these architectures where the words are aggregated before being related with the observations. We name these variants by appendding -WA (word-aggregation) to the name of the model (see Fig. (b)).
W +- reduce(Hk W, 0,,g)
5

---

We examine these variants to study the effect of letting word-tokens directly interact with object- through the self-atee self-attention layers vs simply aggregating all language tokens single embedding and letting this vector condition the processing of observations. The latter is commonly done in using the language embedding in FiLM layers [d for instance. Finding significant effect here would encourage using architectures which allow direct interactions between the word tokens and the objects they refer to.
the aunuuag--condttione RI. and language grounding literature [ 25 37
LSTM Baselines. We also compare some LSTM-based baselines on this task; their architecture is described in more detail Supplementary Section B.3
2.4 Data Generation, Training and Testing Procedures
We use a bot generate the episodes we train on. The data collected consists of 56837 trajectories of T = 30 time steps. Among the traces some descriptions are less frequent than others but we make sure to have least 50 traces representing each of the 2672 descriptions we consider. We record the observed episodes and sentences in a buffer, and when training a model we sample (S, r) tuples with one observation coupled with either a true sentence from the buffer or another false sentence generated from the grammar. More details about the data generation can be found in Supplementary Section B.I
For each of the Transformer variants (6 models) and the LSTM baselines (2 models) we perform an hyper parameter search using 3 seeds in order to extract the best configuration. We extract the best condition for each model by measuring the mean Fi on a testing set made of uniformly sampled descriptions from each of the categories define in section 202 We use the Fi score because testing sets are imbalanced (the number of traces fulfilling each description is low). We then retrain best configurations over 10 seeds and report the mean and standard deviation (reported as solid black lines in Fig. 3 and Fig. 4 of the averaged Fi score computed on each set of sentences. When statistical significance is reported in the text, is systematically computed using a two-tail Welch's 's t-test with null hypothesis μ1 μ2, at level a = 0.05 [33. Details about the training procedure and the hyper parameter search are provided in Supplementary Section B.4
3 Experiments and Results
3.1 Generalization abilities of models on non-systematic split by categories of meaning
In this experiment, we perform study of generalization to new sentences from known observations. We divide our set of test sentences in four categories based on the categories of meanings listed in Section 2.2 Basic, Spatial, Opatio-Temponn and Temporal. We remove 15% of all possible sentences in each category from the train set and evaluate the Fl score those sentences. The results are provided in Fig.
First, we notice that over all categories of meanings, all UT and TFT models, with or without word- aggregation, perform extremely well compared the LSTM baselines, with all these four models achieving near-perfect test performance on the Basic sentences, with very little variability across the 10 seeds. We then notice that all SFT variants perform poorly on all test categories, in line worse than the baselines. This is particularly visible on the spatio-temporal category, where the SFT models perform at 0.75 + 0.020 whereas the baselines perform at 0.80 + 0.019. This suggests that across tasks, it is harmful to aggregate each scene plus the language information into a single vector. This may be due to the fact that objects lose their identity in this process, since information about all the objects becomes encoded in the same vector. This may make it difficult for the network to perform computations about the truth value of predicate on a single object.
Secondly. we notice that the word word-aggregation condition seems to have little effect on the performance on all three Transformer models. We only observe significant effect for UT models on spatio- temporal concepts (p-value = 2.38e-10) This suggests that the meaning of sentences can be adequately summarised by single vector; while maintaining separated representations for each object is important for achieving good performance it seems unnecessary to do the same for linguistic input. However we notice during our hyperparameter search that our -WA models are not very robust to hyperparameter choice, with bigger variants more sensitive to the learning rate.

---

Thirdly, we observe that for our pest-performing models, the basic categories of meanings are the easiest, with a mean score of 1.0 + 0.003 across all UT and TFT models, then the spatial ones at 0.96 + 0.020, then the temporal ones at 0.96 = 0.009, and finally the spatio-temporal ones at 0.097. This effectively suggests, as we hypothesised, that sentences containing spatial relations or temporal concepts are harder to ground than those who do not.
Known sentences with novel observations. We also examine the mean performance of our models for sentences in the training set but evaluated on set of new observations: we generate a new set of rollouts on the environment, and only evaluate the model on sentences seen at train time (plots are reported in Supplementary Section D) We see the performance is slightly better in this case, especially for the LSTM baselines (0.82 0.033 versus 0.79 = 0.032), but the results are comparable in both cases, suggesting that the main difficulty for models lies in grounding spatio-temporae meanings and not in linguistic generalization for the type of generalization considered in this section.
Models UT
FACTORDD
SFT
TFT
LSTM-FUAT
UTMWN
SFRWWN
TTPAA
1.0
0.8
0.6
0.4
0.7
L.Basic
Spatial Sets of descriptions grouped by concepts
3. Temporal
Spatio temporal
Figure 3: F1 scores for all the models on randomly held-out sentences. Fi is measured on separated sets representing each category of concepts defined in Section 2.2
3.2 Systematic generalization withheld combinations of words
In addition to the previous generalization studies, we perform experiment in harder linguistic generalization setting where we systematically remove binary combinations in our train set. This is in line with previous work systematic generalization on deep learning models 20 five test sets to examine the abilitics of our models to generalize on binary combinations of words that have been systematically removed from the set of training sentences, but whose components have been seen before in other contexts. Our splits can be described by the set of forbidden combinations of words as:
37 26
We create
1. Forbidden object-attribute combinations. remove from the train set all sentences contain- ing 'red cat', "blue door' and 'green cactus'. This tests the ability of models to recombine known objects with known attributes;
2. Forbidden predicate-object combination. remove all sentences containing 'grow' and all objects from the "plant' category. This tests the model's ability to apply known predicate to a known object in a new combination;
3. Forbidden one-to-one relation. remove all sentences containing 'right of'. Since the "right" token is already seen as-is in the context of one-to-all relations ( ('right most") ) and other one-to-one relations are observed during training, this tests the abilities of models to recombine known directions with in a known template;
4. Forbidden past spatial relation. remove all sentences containing the contiguous tokens 'was left of This tests the abilities of models to transfer a known relation to the past modality, knowing other spatial relations the past;
5. Forbidden past predicate. remove all sentences containing the contiguous tokens 'was grasp". This tests the ability of the model to transfer a known predicate to the past modality, knowing that it has already been trained on other past-tense predicates.
7

---

To avoid retraining all models for each split, we create one single train set with all forbidden sentences removed and we test separately on all splits. We use the same hyperparameters for all models than in the previous experiments. The results are reported in Fig.
Models
LSTN-ALCOORE
UT
SFT
TFT TFTANA
LSTH-
UTWA
SFT
1.0
0.0
0.6
0.2
0.0
-.hhect-wttribute
2. Prodraat-chjeett
One-fo-dnc relation
4. Post spatial relutton
5. Past prodicate
Types of Systematic Generalization
Figure 4: F1 scores of all the models on systematic generalization splits. Fj is measured on separated sets representing each of the forbidden combinations of word defined above.
First we can notice that the good test scores obtained by the UT and TFT models on the previous sections are confirmed in on this experiment: they are the best performing models overall. We then notice that the first two splits, corresponding to new attribute-object and predicate-object combinations, are solved by the UT and TFT models, while the SFT models and the LSTM baselines struggle to achieve high scores. For the next 3 splits, which imply new spatial and temporal combinations, the scores overall drop significantly; we also observe much wider variability between seeds for each model, perhaps suggesting the various strategies adopted by the models to fit the train set have very different implications in terms of systematic generalization spatial and temporal concepts. This very high variability between seeds on systematic generalization scores are reminiscent of the results obtained on the gSCAN benchmark
Additionally, for split 3, which implies combining known tokens to form a new spatial relation, we observe significant drop in generalization for the word-aggregation (WA) conditions, consistent across models (on average across seeds, 0.14 0.093, -0.15 0.234 and -0.20 + 0.061 for UT, SFT and TFT resp. with p-values < 1e-04 for UT and SFT). This may be due to the fact that recombining any one-to-one relation with the known token right seen the context one-to-all relations requires a separate representation for each of the linguistic tokens. The same significant drop performance for the WA condition can be observed for UT and TFT split 4, which implies transferring a known spatial relation to the past.
However, very surprisingly, for split 5 which implies transposing the known predicate grasp the past tense we observe a very strong effect in the opposite direction: the WA condition seems to help generalizing to this unknown past predicate (from close-to-zero scores for all transformer models, the WA adds average 0.71 = 0.186, 0.45 + 0.178 and 0.52 + 0.183 points for UT, ST and TT resp. and p- p-values< 1e-05). This may be due the fact that models without WA learn a direct and systematic relationship between the grasp token and grasped objects, indicated in their features; this relation not modulated by the addition of the was modifier as a prefix to the sentence. Models do not exhibit the same behavior split 4, which has similar structure (transfer the relation lest of to the past). This may be due to the lack of variability in instantaneous predicates (only the grasp predicate); whereas there several spatial relations (4 one-to-one, 4 one-to-all).
Control experiment. We evaluate previously trained models on test set containing hard negative examples. The aim of this experiment is to ensure that models truly identify the compositional structure of our spatio-temporal concepts and do not simply perform unit concept recognition. We select negative pairs (trajectory, description) so that the trajectories contain either the object or the action described in the positive example. Results are provided in Fig. We observe a slight decrease of performances on all 5 categories (drop is less than 5%), demonstrating that the models do in fact represent the meaning the sentence and not simply the presence or absence of a particular object or predicate.
8

---

Models
LSTH-FLAT
1.0
0.5
0.4
0.2
0.0
1. -.jject-ttrrae eitribtte
2. Peedicale-cbject
One-to relation Types Systematic Generalization
Pest spetial reletion
5. Past, predicate
Figure 5: Control experiment: F1 scores for all the models on systematic generalization splits in the hard negative examples setting.
Related Work
The idea that agents should learn to represent and ground language in their experience of the world has a long history in developmental robotics (50
39 40
7 and was recently extended in the context
of Language Conditioned Deep Reinforcement Learning
3I 3. These recent approaches 22 tasks and are always using instructive language. Meanings typically refer to instantaneous actions and rarely consider spatial reference to objects (35. Although our environment includes object manipulations, we here tackle novel categories of meanings involving the grounding spatio-temporal concepts such as the past modality complex spatio-temporal reference to objects.
22
often consider navigation [T0 g or object manipulation [
We evaluate our learning architectures on their ability to generalise sets of descriptions that contain systematic differences with the training data so as to assess whether they correctly model grammar primitives. This procedure is similar to the gSCAN benchmark (37. This kind of compositional generalisation is referred as 'systematicity" by Hupkes et al. (26]. Environmental drivers that facilitate systematic generalization are also studied by Hill et 123 Although Hupkes al. 26 consider Language Grounding task. Ruis al. 3 consider an Embodied Language Grounding setup involving one form of time- extended meanings (adverbs), but do not consider the past modality and spatio-temporal reference to objects, and do not consider learning truth functions. Also, they do not consider learning architectures that process sequences of sensorimotor observations. To our knowledge. no previous work has conducted systematic generalization studies on an Embodied Language Grounding task involving spatio-temporal language with Transformers.
relational models in their work, they do not evaluate their performance
The idea that relational architectures are relevant models for Language Grounding has been previously explored in the context of Visual Reasoning. They were indeed successfully applied for spatial reasoning the visual question answering task CLEVR 38). With the recent publication of the video reasoning dataset CLEVRER 47, those models were extended and demonstrated abilities to reason over spatio-temporal concepts, correctly answering causal, predictive and counterfactual questions study, these works around CLEVRER do not aim to analyze ppatio-tempordl
[66. In contrast language and therefore do not consider time-extended predicates or ppatio-temporal reference to objects in their language, and do not study properties of systematic generalization over sets of new sentences.
5 Discussion and Conclusion
In this work, we have presented a first step towards learning Embodied Language Grounding of spatio-temporal concepts, framed as the problem learning a truth function that can predict if given sentence is true of temporally-extended observations of an agent interacting with collection of objects. We have studied the impact of architectural choices on successful grounding of our artificial spatio-temporal language. We have modelled different possible choices for aggregation of

---

observations and language as hierarchical Transformer architectures. We have demonstrated that in our setting. is beneficial process remporally-exteer extended observations and language tokens side-by- side, evidenced by the good score of our Unstructured Transformer variant. However, there seems aggregating temporal observations along the temporal
to be only minimal effect on performance dimension first - compared to processing all traces and the language in an unstructured manner as long as object identity is preserved. This can inform architectural design in cases where longer episode lengths make it impossible to store all individual timesteps for each object; our experiments provide evidence that a temporal summary can be used in these cases. Our experiments with systematic dimensions of generalization provide mixed evidence for the influence of summarizing individual words into a single vector, showing it can be detrimental to generalize to novel word combinations but also help prevent overgeneralization of relation between single word and single object without considering the surrounding linguistic context.
Limitations and further work. There several limitations of our setup which open important opportunities for further work. First, we have used a synthetic language that could be extended: for instance with more spatial relations and relations that are more than binary. Another axis for further research is using low-level observations. In our setting, we wanted to disentangle the effect of structural biases on learning spatio-temporal language from the problem of extracting objects
30 g in a consistent manner over time (object permanence needed, and it could allow to define richer attributes (related to material or texture) and richer temporal predicates (such as breaking, floating, etc). Finally, we use a synthetic language which is far from the richness of the natural language used by humans, but previous work has shown that natural language can be projected onto the subspace defined by synthetic language using the semantic embeddings learned by large language models [3 this opens up be a fruitful avenue for further investigation.
from low level observations 6 21
17
15
Further steps in this direction
A further interesting avenue for future work would be to use the grounding provided by this reward function to allow autonomous language-conditionel agents to target their own goals [14. In this sense, the truth function can be seen as goal-achievement function reward function. While generalization performance of our method is not perfect, the good overall fl scores of our architectures imply that they can be directly transferred a more complete RL setup to provide signal for policies conditioned on spatio-temporal language.
Broader Impact. This work provides step in the direction of building agents that better understand how language relates to the physical world; this can lead to personal robots that can better suit the needs of their owners because they can be interacted with using language. If successfully implemented. this technology can raise issues concerning automation of certain tasks resulting in loss of jobs for less-qualified workers.
Links. The source code as well as the generated datasets can be found at httph:tt/////t /tttp://github... com/ temporal -laggugge transformers
Acknowledgments and Disclosure of Funding
Tristan Karch is partly funded by the French Ministère des Armées - Direction Générale de I' l'Armement. Laetitia Teodorescu supported by Microsoft Research through its PhD Scholar- ship Programme. This work was performed using HPC resources from GENCI-IDRIS (Grant 2020-40091011996
References
[1] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer. Mohamed Chetouani, and Olivier Sigaud. Grounding Language to Autonomously-Acquree Skills via Goal Generation. In ICLR 2021- Ninth International Conference on Learning Representation, Vienna Virtual, Austria, May 2021. http:tss: fnriaa.fr/hal-01221111
[2] James F. Allen. Towards general theory of action and time. Artificial Intelligence, 23(2): 123-154, 1984. ISSN 0004-3702 doi: http:://wooooogo//....////041010
URI.
10

---

[3] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. In International Conference on Learning Representations, 2019. URL https: cestps://openrevieee wttprrrvviiwwee//frrrere net/forum? id=HixsSjC9Ya
[4] John A. Bateman, Joana Hois, Robert Ross, and Thora Tenbrink. A linguistic ontology of space for natural language processing. Artificial Intelligence. 172414):1027-1071,, 2010. ISSN 0004-
3702. doi: https://doi.org/100. 1016/j wwt.:/i...ttttt211...... art.iitt 00.0000.000........0.000 URL http://wwww..cieeeeerrre..
com httpa//c//errrtaa///////////////
[5] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo. Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning. and graph networks, 2018.
[6] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion, 2019.
[7] Angelo Cangelosi, Giorgio Metta, Gerhard Sagerer, Stefano Nolfi, Chrystopher Nehaniv, Kerstin Fischer, Jun Tani, Tony Belpaeme, Giulio Sandini, Francesco Nori, Luciano Fadiga, Britta Wrede, Katharina Rohlfing, Elio Tuci, Kerstin Dautenhahn, Joe Saunders, and Amne Zeschel. Integration of action and language knowledge: A roadmap for developmental robotics. IEEE Transactions on Autonomous Mental Development, 2(3):167-195, 2010. doi: 10.1 109/TAMD 2010.2053034.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
[9] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. inteddattention architectures for misk-oriented language grounding, 2018.
[10] David L. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI'11, page 859-865. AAAI Press, 2011.
[11] Maxime Chevalier-Boisee Boisverr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform study the sample efficiency of grounded language learning, 2019.
[12] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 225-232. 2020. doi: 10.1109//S114700. 2020.9308603.
[13] Cédric Colas, Olivier Sigaud, and Pierre-Yver Oudeyer. A hitchhiker's guide statistical comparisons of reinforcement learning algorithms, 2019.
[14] Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Fries Pe- ter Ford Dominey, and Pierre Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration, 2020.
[15] Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanbur,, Marta Garnelo, and Murray Shanahan. Alignnet: Unsupervised entity alignment, 2020.
[16] David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architec- tures, 2020.
11

---

[17] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations, 2020.
[18] Dedre Gentner and Jeffrey Loewenstein. Relational language and relational thought, 2002.
[19] James J. (James Jerome) Gibson. The senses considered as perceptual systems. Allen Unwin, London, 1968.
[20] Arthur M. Glenberg and Michael Kaschak. Grounding language action. Psychonomic Bulletin & Review, 93)):55--565, September 2002. ISSN 1531-5320. doi: 10.3758/
BF03196313. URL httpp:t:/t.ooooo//////to 0tt.//00.7758//F3196333
[21] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference, 2020.
[22] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps. Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world, 2017.
[23] Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent, 2020.
[24] Xavier Hinaut, Maxime Petit, Gregoire Pointeau, and Peter Ford Dominey. Exploring the acquisition and production of grammatical constructions through human-robor interaction with echo state networks. Frontiers neurorobotics, 8:16, 2014.
[25] David Yu- Yu-Tung Hui, Maxime Shevalier-Boisver, Dzmitry Bahdanau, and Yoshua Bengio. Babyai 1.1, 2020.
[26] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise?, 2020.
[27] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning, 2016.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[29] Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the composi- tional skills sequence to-nicee----eunncee recurrent networks, 2018.
[30] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
[31] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language, 2019.
[32] Olivier Mangin, David Filliat, Louis Ten Bosch, and Pierre- Yves Oudeyer. Mca-nmf: Multi- modal concept acquisition with non-negative matrix factorization. PloS one, 10(10):e0140732, 2015.
[33] Alana Marzoev, Samuel Madden, M. Frans Kaashock, Michael Cafarella, and Jacob Andreas. Unnatural language processing: Bridging the gap between synthetic and natural language data, 2020.
[34] Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro Dudík, and Patrick Shafto, Interactive learning from activity description, 2021.
12

---

[35] Rohan Paul, Jacob Arkin, N. Roy, and T. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Robotics: Science and Systems, 2016.
[36]
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer, 2017.
[37] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A benchmark for systematic generalization in grounded language understanding. 2020.
[38] Adam Santoro, David Raposo. David G. T. Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning, 2017.
[39] I.. Steels. Semiotic dynamics for embodied agents. IEEE Intelligent Systems, 21(3):32-38 2006. doi: 10.1 10910099...00 2006.58..
[40] Yuuya Sugita and Jun Tani. Learning semantic combinatoriality from the interaction between linguistic and behavioral processes. Adaptive behavior, 13(1):33-52, 2005.
[41] Jun Tani. Exploring robotic minds: actions, symbols, and consciousness as self self-organizing dynamic phenomena. Oxford University Press, 2016.
[42] Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh. Symbol emergence in robotics: a survey. Advanced Robotics, 30(1)-12): 706-728, 2016.
[43] Thora Tenbrink. Space, Time, and the Use of Language: An Investigation Relationships. De Gruyter Mouton, 2008. ISBN 978-3-11-01188229, doi: do1110.1511//983111199988 URL https://doi.org 11tp://....1111/1111111111111110
[44] Thora Tenbrink. Reference frames of space and time in language. Journal of Pragmatics, 43(3): 704-722, 2011. ISSN 0378-2166. doi: httpa//tww......c..cc..c..aaaaaa
URL The
https://www.. http://wwww.cceenceeeerrtt.
Language of Space and Time.
[45] Elio Tuci, Tomassino Ferrauto, Ame Zeschel, Gianluca Massera, and Stefano Nolfi. An experiment on behavior generalization and the emergence of linguistic compositionality in evolving robots. IEEE Transactions on Autonomous Mental Development, 3(2):176-189, 2011.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
[47] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, 2020.
[48] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizii, Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In International Conference on Learning Representations, 2021. URL https //openreview.o Copparrerieev.eetforuuk.aaaaeaee ret:/forum?iddMM22f//bbbb
[49] Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals, 2020.
[50] Rolf A. Zwaan and Carol J. Madden. Embodied Sentence Comprehension, page 224-245. Cambridge University Press, 2005. doi: 1010017/CH097701114499996
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes], we propose a summary of our results supporting our
contributions in Section 5
(b) Did you describe the limitations of your work? [Yes], in the concluding Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes], in a the
Broader Impact paragraph in the Conclusion section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes], the code is given supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]. see Sections B.] and 3.2
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes], all reported scores were averaged over 10 seeds and std is reported in all plots (Fig- 3 Fig. of Section 3 and in the main text. We also performed statistical tests to measure significance.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] details are given in Supplementary Section D.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "dBbNWGk8ojU", "reviewer": "Reviewer_MYUM", "summary": "The paper presents a system for grounded, spatio-temporal, synthetic language learning.\nSpatio-temporal, here, means descriptions of observations that describe spatial layout (e.g. front) and temporal descriptions (e.g. past).\nThe main results are 1) object identity is important for performance; 2) collabpsing object traces into a single token does not impact performance.\nThe paper studies different transformer architectures for solving the task with a specific eye on generalization 1) to hold out sentences, 2) to grammatical primitives.\nThe system is trained to learn a truth function that represents whether a given spatio-temporal statement is true.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\nThe general research direction of learning grounded language specifically in tasks that are temporally extended is interesting and worthwhile.\n\nThe paper does give a good overview of more generally related literature such language learning in RL etc.\n\nAuthors do a good job at pointing out limitations of the work\n\nThe paper is written very clearly and is easy to follow.\n\n\nWeaknesses\n\nIt's not clear to me that the main results merit publication. I think it's somewhat obvious that object identity plays a role in spatio-temporal tasks. Not sure what other explanation would be possible.\n\nThe paper learns on a synthetic language that is very structured. The language imo doesn't have exceptions and the free usage that characterises Natural language. Also the grammatical complexity of the synthetic language is quite reduced. It's not clear how important those choices are for the performance of the system and it's performance in more natural settings.\n\nThe setting is quite a toy environment. While it's illuminating as an example of the different aspects of the system, I am not sure that the result is already very interesting or surprising in the sense of solving the task. \n\n\nQuestions/Comments\n\nThere is a huge are of AI that deals with models of spatio temporal and dynamic preposition reasoning and learning. Work by Tenbrink comes to mind. I think it's worthwhile to relate this paper to some of this work.\n\nSimilarly related work on compositional grounded language learning is missing in my view.\n\nTruth functions for language represent a specific class of semantics. There are other forms of language use that are probably more important in real life. Such as the discrimination etc. I wonder how this system would deal with that. \n\nI am sort of surprised that the representation for sentences and words is one hot. Since most models these days train or use embeddings, I am not sure why this choice is important here. Embeddings may also remove the constraint that the dictionary is fixed.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "YES", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "OHszmL3jXr", "reviewer": "Reviewer_3pgx", "summary": "This paper proposes a new spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. They do this by training a truth function that predicts whether the description matches the given history of observations or not. They study the impact of architectural choices for multimodal transformer and implement different attention computations between words and objects across space and time. They tested these models on generalization to held-out sentences and generalization to grammar primitives.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and Significance:\n-\tThis paper is the first step towards learning spatio-temporal concepts for embodied language grounding. They learn a truth function that predicts if a given sentence is correct for time-extended observations of an agent interacting with a collection of objects. They set up a task for this problem and provide the first baselines for it.\n\n-\tThey define three transformer architectures: Unstructured Transformer, Spatial-First Transformer, and Temporal-First Transformer, and used these different architectures to aggregate observations and language. They demonstrate that it helps to process temporally-extended observations and language tokens side-by-side, shown by the scores of unstructured transformer.\n\nQuality and Clarity:\n-\tThe paper is clearly written, well-organized, and easy to follow.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are described in the Conclusions sections.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "6wENG0BGJj", "reviewer": "Reviewer_viF6", "summary": "This paper introduces a new 2D environment that produces synthetic language describing spatio-temporal relations between objects within the environment. In the paper, the environment was used to generate interaction data using a bot. A synthetic language annotations describing the (spatio-temporal) interactions were generated. \n\nGiven the data, the authors proceed by formulating the Embodied Language Grounding problem as learning a parametrized truth function of the spatio-temporal interactions. The parameterized truth function takes in the history of observations as well as a sentence describing the interactions and produces a boolean value signaling whether the sentence is true or false. Ground truth values are obtained by the data generation procedure.\n\nThe authors introduce a number of Transformer models and baseline LSTM models and perform two types of evaluation:\n1. Testing the generalization capability of the models when the data is within the same distribution - by using % of the overall data as test data,\n2. Testing the out-of-distribution generalization - by removing certain interactions from the training set, but adding them to the test set.\n\nThe authors show that Unstructured Transformer variant, which processes temporally-extended observations and language tokes side-by-side achieves a reasonable performance on almost all of the tasks. This in turn demonstrates it's ability to generalize. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a novel way of studying Embodied Language Grounding through the means of a newly introduced environment, data generating procedure and an evaluation pipeline. The authors perform evaluation with the aim assessing generalization capabilities of a set of architectures. It presents a novel angle and a new approach to Embodied Language Grounding.\n\nMy main problem with the submission is clarity. The wording of the paper is very complicated and it took multiple passes to understand what the authors meant. Due to this, I am not entirely sure what the final conclusions on the generalization capabilities are, as well as what the novelty of the approach is, compared to the grounding strategies detailed in the introduction.\n\nThe way operations (reduce in particular) were explain was unclear to me until I went through code/math. So I would suggest formalizing the transformations using mathematical notation/pseudocode instead of explaining by words.\n\nDue to this it is also difficult to assess significance of this work in a wider scope. Highlighting the main results in the conclusion and making parallels with related work mentioned in the introduction could help.\n\nSome minor comments:\n- It is not clear whether multiple linguistic observations ($W$) are generated for a single trace ($S$) in the dataset,\n- `animal` objects can move independently, but I was not able to find in the paper how that behavior is implemented,\n- Negative sentences are part of the dataset, but it is not clear how those are generated. The paper references Supplementary Section 2.1 for more information, but I wasn't able to find information on that part of the logic.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "675AV2Lxky", "reviewer": "Reviewer_8BQz", "summary": "The work introduces a new embodied benchmark for grounding spatio-temporal language. The task is binary classification of whether a language description fits a temporal trajectory with multiple objects spanned spatially. Three transformer models are proposed: no aggregation (Unstructured Transformer, UT), spatial-first Transformer (SFT), and temporal-first Transformer (TFT). The models are evaluated on the benchmark with several generalization studies. The main finding is that maintaining object identity is instrumental to generalization, and aggregating an object trace in a single token has little influence.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance and Originality: Spatio-temporal language has been less studied in language grounding. However, as argued by authors, it is an important aspect of human language, and has an significant value to embodied robotics research. The benchmark, though simple and basic, fills an important position and can inspire future attention in this direction. \n\nQuality: The benchmark and baseline models are not complicated, yet the main findings (i.e. SFT is bad at generalization due to object mixing, several linguistic generalizations are hard) make sense as general findings about the problem. \n\nClarity: The writing is neat, and the figures are clear.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The language may be too synthetic. Is the English paraphrase in Figure 1 anyway used? Also, for the task of binary classification, random negatives might be too easy to figure out. Maybe some evaluation should be done with hard negatives, e.g. containing same basic concepts and objects, but organized in a different structure. This tests if the model is just pattern recognizing simple concept units, or is really understanding the compositional language.\n\n- Why is binary classification an ideal task? Why not generating descriptions conditional on trajectories, which is harder and potentially more valuable? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: To begin, I should clarify that during discussions, Reviewer MYUM stated that while their objection stands after reading other reviews, they see it as a corner case, and that they voted "marginally below threshold". I believe they forgot to update the rating on their review, so I'm interpreting their 4 as a 5+.

Simply put, this paper introduces a new setting for testing the grounded language learning capabilities of learning algorithms and architectures, whereby spatial and temporally extended relationships between objects must be captured by agents via (synthetic) language understanding. They propose a number of architectures which aim to help with this sort of language grounding, and compare them to sensible baselines. They evaluate against several forms of generalization, both on the linguistic and the relational front.

The main objection that could be levelled at this sort of environment is that the synthetic language is somewhat toy in that it is (as far as I understand) unambiguous, neatly structured, and has fairly homogenous semantics, but I think such testbeds are important if they serve as a starting point for research into more complex topics, such as the spatially and temporally extended relationships this work caters to. It's a neat contribution to language-informed/directed sequential decision making, and given the reviewer consensus post discussion, I don't think anyone is looking to champion the case for rejecting it. I recommend acceptance.
Decision: Accept (Poster)