Grounded and well-rounded: a methodological approach to the study of cross-modal and cross-lingual grounding
Authors: Timothee Mickus
Elaine Zosa
Denis Paperno
Venue: EMNLP 2023 Findings (2023)
Keywords: grounding
crosslingual grounding
multimodality
TL;DR: We establish a methodological framework for studying what are the qualitative effects of grounding on text generation systems, which we measure both at a global dataset level and on the word level.

Abstract: Grounding has been argued to be a crucial component towards the development of more complete and truly semantically competent artificial intelligence systems. Literature has  divided into two camps: While some argue that grounding allows for qualitatively different generalizations, others believe it can be compensated by mono-modal data quantity. Limited empirical evidence has emerged for or against either position, which we argue is due to the methodological challenges that come with studying grounding and its effects on NLP systems.

In this paper, we establish a methodological framework for studying what the effects are---if any---of providing models with richer input sources than text-only. The crux of it lies in the construction of comparable samples of populations of models trained on different input modalities, so that we can tease apart the qualitative effects of different input sources from quantifiable model performances. Experiments using this framework reveal qualitative differences in model behavior between cross-modally grounded, cross-lingually grounded, and ungrounded models, which we measure both at a global dataset level as well as for specific word representations, depending on how concrete their semantics is.

Grounded and Well-rounded: A Methodological Approach to the Study of Cross-modal and Cross-lingual Grounding
Timothee Mickus
Elaine Zosa
Denis Paperno Utrecht University d. paperno@uu. nl
University of Helsinki
Silo AI
timothee. mi mwwkss@ellsskii....
elaine zosa@slzosa@sllo..a
Abstract
programming code, and to video game controls (e.g., Reed et al., 2022: Ni et al., 2021; Huang et al., 2021). These approaches to multimodal- ity are often practically motivated: the initial re- search question of Reed et al. (2022) is specifically whether a single model can handle multiple types of inputs, that of Shichman et al. (2023) is specifi- cally whether pretrained language models could be co-opted for human-robot instructions.
Grounding has been argued to crucial com- ponent towards the development of more plete and truly semantically competent artificial intelligence systems. Literature has divided into two camps: While some argue that ground- ing allows for qualitatively different general- izations, others believe it can be compensated by mono-modal data quantity. Limited empiri- cal evidence has emerged for or against either position, which argue is due the method- ological challenges that come with studying grounding and its effects on NLP systems.
This research field is vibrant, and novel insights have been plenty: To date, researchers have put forth theoretical arguments that we expect qualita- tive differences among models due grounding, and have demonstrated practical use-cases where handling multiple types of inputs is desirable and feasible. There however a dearth of empirical studies bridging the two: dearth of studies assess- ing whether, when it comes to language modeling, grounding systematically yields different textual predictions. In this paper, therefore take step back and focus what evidence there might be that grounding leads to an observable qualita- tive difference in model behavior, beyond the trivial quantitative expectation that richer input might lead to better downstream performance.
this paper, we establish a methodological framework for studying what the effects are- if any-of providing models with richer in- put sources than text-only. The crux it lies in the construction of comparable samples of populations of models trained on different in- put modalities, so that we can tease apart the qualitative effects of different input sources from quantifiable model performances. Experi- ments using this framework reveal qualitative differences in model behavior between cross- modally grounded, tross-lingually grounded, and ungrounded models, which we measure both at a global dataset level as well as for spe- cific word representations, depending on how concrete their semantics is.
This sort of endeavor difficult set up, due to interrelated confounding factors one confronted with. Firstly, data for different modalities tend to not be easily comparable-imen ically different from text. As such it is difficult to assess whether models trained on different modal- ities are of similar quality-and and therefore useful points of comparison are hard come by. Sec- ondly, there somewhat of lack when it comes defining what reasonable expectations with respect to grounding ought to be and how to measure them. However, none of these hurdles insurmount- able. and in the present paper we propose a method- ological framework for studying the effects of grounding. This framework is built upon a dis- tinction between two notions of grounding (section
1 Introduction
are after all rad-
Some researchers have explicitly argued that achieving natural, human-like linguistic behavior requires a richer input scheme than pure text, and specifically that perceptual input is necessary. This has been one of the arguments of the well-known position paper on the inherent limitations of lan- guage modeling (Bender and Koller, 2020). Luck- ily, recent developments have prepared the ground for empirical computational testing of this line of argumentation: pretrained Transformer-ban cls have been adapted to handle different modali- ties, ranging from natural language text to images,
mod-

---

3.1): one weaker and one stronger, depending on whether we can disentangle input representation from task processing. On a more practical level, we focus on multimodal and multilingual data, and define a procedure to construct samples of models so that they are broadly comparable. We apply this approach to cross-lingually grounded (Tiedemann, 2018), cross- moos-lydally grounded, and ungrounded models so as to better characterize how these dif- ferent groups of models compare and contrast with each other.
using neural networks as models, also exist: see for instance the work of Khorrami and Räsänen (2021); Nikolaus and Fourtassi (2021). related trend in computational semantics relates specific aspects of meaning to situated information (Ebert et al., 2022; Ghaffari and Krishnaswamy, 2023, e.g.). We also refer the reader to the survey of Chrupata (2022) recent visually grounded mod- els of spoken language from the NLP and cognitive science communities.
The works cited above focus primarily on the- oretical aspects of grounding; other works take a more practical angle. These latter fall into two broad categories: (i) works that probe for specific aspects of grounding, e.g. Patel and Pavlick (2022); Tenney et al. (2019); Hwang et al. (2021); and (ii) works tackling engineering challenges and oppor- tunities that come with systems handling multiple channels of inputs (e.g., Reed et al., 2022; Ni et al., 2021; Li et al., 2020; Jia et al., 2021; Kim et al., 2021; Shichman et al., 2023). In recent years, most research in this area has converged on Transformer- based systems which align inputs from different channels into shared semantic space enable multimodal interaction.
Our experiments' demonstrate that access to richer and more diverse inputs impacts model behavior, even when factoring in model werformance-wiichh we observe both on global, date level and more narrowly with respect lexicon of concrete and abstract words.
2
Related Work
The idea that language data alone do not suffice to build full-fledged semantic representations has been discussed by numerous philosophers, with various areas of focus. One controversial land- mark work is Searle (1980), which introduces thought experiment based on translation to argue that systems that only ever deal with symbols can- not gain understanding of their surroundings. Jack- son (1982) discusses the need of perceptual input; Harnad (1990) provides a thought experiment for monolingual situations.
Most research on grounding involves signal from different modalities, most commonly text and im- ages. Cross-lingual grounding, on the other hand, is still an under-researched area, even though al- ready Hjelmslev (1943) remarked the usefulness of bilingual lexicons study fine-grained seman- tic differences. The idea that mooss-lingual stud- ies could provide insights into semantics eventu- ally informed practical applications, e.g. Dyvik (2004) used translations as a source of semantic information that can elucidate the relations be- tween words that are hard to specify based on monolingual corpus. This idea also underlies works that use multilingual corpora to automati- cally construet language-specific WordNets (e.g.. Fišer, 2009). More recently, Tiedemann (2018) demonstrates how a model trained on a massively multilingual corpus can give rise to an 'interlin- gua' space that enables translation between lan- guage pairs that are not in the training corpus, link- ing cross- rings-lingual grounding to another core con- cept long studied in machine translation (Richens, 1956).
This line of thought has led to more modern theory-focusee approaches, i.e., NLP scientists try- ing repurpose the cognitive science concept of grounding to make sense of the behavior of neu- ral language models. One particular discourse- provoking piece that of Bender and Koller (2020), which questions whether modern language models can be expected to display some form of under- standing despite lacking grounding. It is worth echoing some remarks of Chandu et al. (2021), who outline that, while cognitive science defines grounding as establishing a common ground for communication, the NL.P community has adopted a specific, more restricted definition of grounding as linking concepts from different sources such as text (single and multiple languages), images, video and speech, and so on.
The opposite trend, viz. researchers interested in perceptual information in human development
There is nonetheless an important difference be- tween cross-modal and cross-lingual grounding:?
'Code and data for our experiments are available at gi github. wwwp://wwwwiiiiiiiiittteeeeeeeee wix.-k/tddd---dddff...
"We are indehted to an anomyous reviewer of this work

---

While the iormer generally understood as linking symbols to non-symbolic external data such per- ceptual data, Tiedemann's (2018) proposal slightly alters the definition of grounding as possessing rep- resentations that can "resolve language-interun ambiguities" ()1). These two positions are not as incompatible as one might initially think: For in- stance Harnad (1990) does concede that symbolic meaning can be derived from "[grounding] in first language and in real world experience and knowledge" ($2.2). In that respect, Tiedemann's view can be understood as positing that the avail- ability of a first language is already sufficient for grounding, regardless of real world experience and knowledge.
To clarify, we can reframe weak grounding as simply satisfying that system has access to some other modality, on top of the language processes. Any system with multiple input channels, that sense is weakly grounded. However, some of the recent literature suggests grounded systems should have further nontrivial properties than simply being able to (meaningfully) process alternative modality input: For instance, properly grounded system might be less likely to hallucinate about perceptu- ally salient properties even if there no informa- tion about them in the specific input. These sup- plementary properties are what we expect strong grounding to capture. All strongly grounded sys- tems are weakly grounded, but system can be weakly grounded without being strongly grounded. As such, one of the questions we explore in the paper is whether strong grounding, i.e. nontrivially different language-only processing by grounded systems, can be observed in a controlled setting.
3 Methodology
3.1 Theoretical framework
We start by reframing the question of grounding follows: are agents that have access to richer inputs functionally different? There are two ways in which this could be the case.
3.2 Comparable tasks
Our intent is to ensure that whatever difference we find can only be imputed the different types of inputs the models receive, As such, the first factor for us to control is that ensuring that our models are exposed to comparable data.
The first case to consider that functional differ- ences can be accurately described solely in terms of what input a model can receive. Under such a sce- nario, it may sometimes be possible to factorize the function implemented by grounded model into sub-functions g o f. where the first sub-function would map input data coming from different chan- nels into common semantic space, and the sec- ond sub-function y would perform the model's task proper. This leads to a weaker notion of grounding, where the effects of exposing a model to varied in- put sources are limited to requiring the model to learn map between different input spaces. ?
We work on the Vatex dataset (Wang al., 2019), which contains video features, Chinese cap- tions and English captions. 5 Each datapoint corre- sponds to one video, ten Chinese and ten English Thit captions. This allows us to define three tasks: a captioning task (C), where we generate English captions from video features, a translation task (T), where we generate English captions from Chinese captions, and a paraphrasing task (P) where we gen- erate English captions from other English captions.
The second possibility is that the functional dif- ferences cannot be subsumed to learning a mapping from different input sources to a common repre- sentational spaces, and that there is non-trivial processing to be done that is dependent on the spe- cific input source. This therefore corresponds to a stronger notion of grounding, whereby we assume that the handling of an input of given source can- not be neatly disentangled from the processing of the information conveys-simply put, that there are things that can be shown but not 5
All three of these tasks have the same output space, i.c., they all aim at generating English cap- tions. We also ensure that the intrinsic ambiguity of the examples is the same across tasks by selecting only one English and Chinese captions per data- point serve sources, and using all ten English captions as possible targets. This ensures that the three tasks are as comparable as possible, such that the only factor to explain variation is the type of input the models receive.
Owing practical considerations, re-split the Vatex dataset. We use the official Vatex vali-
for this point.
"This for instance explicitly the approach espoused by interlingua-batt based approaches to machine translation.
"Note that in the case of cross-lingual grounding, this would entail some form of linguistic relativism.
Phttps://eric- xw. ww..iw.................. thub. htt/vvaxx-websit.. about.htm html

---

dation set as a held out test set, since no labels are made publicly available on the official test set. We furthermore select 1000 datapoints from the official training set to create a validation set.
we only consider nulti-modal models that involve lask inputs. These models provide an alternative take on how to implement grounding, where we attempt to enrich the environment that the model has access to when making its predictions.
3.3 Different implementations of grounding
3.4 Comparable models
As interest lies in providing a better understand- ing of the effects of cross-lingus and cross-modal grounding, we consider three ways of combining inputs and training models in these tasks. Figure I illustrates the different setups we use in this paper. Single-task setups involve training models on only one of the three tasks P. T, C. Mod- els trained in this fashion provide useful base- lines against which to compare grounded mod- els. We expect single-task models to shed some light as to what differences we should expect sim- ply due to the difference in inputs and modalities- independently of proper grounding.
Having defined our tasks and how to combine them, we turn to the actual training of model populations. All of these models are defined sequence-to-sequence models with Transformer de- coders (Vaswani et al., 2017), and are trained with Adafactor to generate English captions. So as to ensure that models are strictly comparable, we rely on pretrained encoder representations (namely the 3D convolutional segment-level video features pro- vided with Vatex for C, vectors from bert-base for P and from bert-base-chiness for T; Devlin et al., 2019), and learn only decoder parameters. This allows us to contrast models with the exact same parameter counts and architecture, bypassing concerns of how to properly account for the needs of handling datapoints of different structure using single common architecture.
Multi-task setups involve training models such that they can handle two more of the tasks. As such, a multi-task model trained on P and T can either generate an English caption from another Lin glish caption, or generate an English caption from a Chinese caption. Consistent with the focus of this work, we only consider multi-task setups that in- volve the P task: our interest lies in characterizing the effects of adding input sources of other modal- ities or languages, hence we take the P task base case. These models are intended to be repre- sentative of approaches such as Reed et al. (2022), which are trained to handle varied input sources using the same parameters. They will also prove especially convenient for teasing apart evidence for and against the stronger and weaker notions of grounding we have outlined in Section 3.1, as we can contrast the functional behavior of a single-task model trained on P to that of multi-task model pre- sented with inputs. For notational convenience, we denote a multi-task model trained with features from tasks X and Y as a XvY model.
However, this approach introduces another con- founding factor, namely the intrinsic quality of the encoder representations. We have no strict guar- antee that the Vatex video features and the BERT embeddings capture their inputs equally accurately. We respond to this concern by constructing samples of models that we know to be of equal performance posteriori, by measuring their accuracy on held- out validation split. In practice, we greedily select one model from each of the different groups we wish to compare such that the selected models have the most similar accuracy, and iteratively repeat this greedy selection until a sample large enough has been selected."
So that we can ensure that our models display various degrees of accuracy- and that we can there- fore overcome any quality differential due the encoders- we consider two factors likely to impact their performance. The first is simply the number of training epuchs: For cach model that we train, we save checkpoints after 5. 10, 15, 20 and 25 training epochs; all checkpoints are then considered as dis- tinct models when constructing comparable groups.
Multi-modal setups involve training models to generate English captions using inputs from more than one task at once. For instance, multi-modal model trained on P and C inputs will use as input both English caption and video, and will use both jointly generate a novel English caption; which we denote as PAC model. 6 As previously,
"As such, we can ensure that they maximize the p-value on Kruskal- Wallis H-test of accuracy scores where samples correspond to the different groups we wish to compare. We select samples of 40 models, which in our experiments always guarantced a p-value > 0,5 on a H-test.
"Whereas a PVC multi-task model either makes a predic- tion le using a video as input or make prediction úP using a caption as input, a PAC multi-modal model uses hoth a video and a caption as inputs to produce single prediction frac-

---

Single-taak
Multi-lskk
P
PvT captors
EN cupsoon
Video
caciiione
captions
EN captions
capilons
Video features
Video featurns
captions
Inpult spaces
caplions
Anpur spaces
EN capiiona
Output spece
P.C
PACAT
PAT
Input spacos
Multi-modal
Figure 1: Setups for training the models used in this study.
As generally expect models from later epochs to outperform models from earlier epochs, this allows to introduce more diverse models at no cost. The second consists in adding Gaussian noise to the encoder representations. We control the strength of this noise by means of scalar E {0, 0.5, 1, 1.5): we substitute every encoder repre- sentation with noisy version X x nz where z randomly sampled standard Gaussian vector z (V, We expect that higher noise levels n deteriorate the quality of the encoder representa- tions, as subtler topological relations the encoder embedding space might be distorted.
mark that in principle, we have reason to expect qualitatively different predictions from models that reach equivalent accuracy: all of these models are trained predict the same outputs and are con- fronted to equally ambiguous inputs.
To answer this, we construct three samples of models to compare: (i) sample where com- pare P, C, and T single-task models; (ii) sam- ple where we compare single-task P models multi-task PvC, PVT, and PvCVT models; and (iii) a sample comprised of single-task P models and multi-modal PAC, PAT, and PACAT models. We then compute agreement rates for every pair of models MM,M2 in each sample-i.e., how of- ten they make the same prediction, given the same target prefix. For multi-task models, use the P features to derive predictions, both for constructing the sample and computing agreement.
We train models with 5 different seeds for each level of noise n and training setup from Section 3.3.
3.5 Implementation details
All the decoders we train have 6 layers, heads per multihead sublayer, hidden dimensions of 512, and latent feedforward dimensions of 2048, or approxi- mately 64M parameters. For multi-modal models, also learn simple linear projection for input features, so as to allow the model to handle fea- tures of varying dimensionality, component scale and orientation across the different tasks; features for all input spaces are then simply concatenated as attention banks. Models optimize a cross entropy loss, using AdaFactor (Shazeer and Stern, 2018) and batch size of 1024 over 25 epochs.
The distributions of agreement scores are shown in Figure 2. We observe that most comparisons involving models of the same setups tend to yield higher agreement scores than most comparisons of models from different setups. This leads to a very salient contrast for single-task models in Fig- ure 2a: Pairs of models of the same setup yield higher agreements than pairs of different setups (Mann Whitney U, < 10 -566 common lan- guage effect size f = 0.88). This also holds for multi-modal models in Figure 2b (p < 2- 171
f = 0.62) and even multi-task models in Figure 2c (p 10-6 ʃ 0.52) though the effect is much weaker and mostly noticeable through third quartiles. For single-task models, we also see that comparing two models trained on text yields higher scores than comparing text and video models.
4 Difference in behavior
The first question we address follows: are justified in expecting that different input sources should lead to qualitatively different predictions, after having controlled for model accuracy? Re-

---

Singlo task models
Muttimoodal modal models
seme tasis
same tasks
diferant
0.80
agrosment
aginammnt
0.70
0.70
0.60
0.50
1/1
PaCiPaa
PiPAT
PIPACAT
Paci GAA
PPAC
PACPAC
PATIPA
PACATPAT
PACATRAAA TAT
task
par
(a) Single-task sample
(b) Multi-modal sample
models
0.00
0.75
agrocmore
0.65
JAIT tooks
difiemnt
PPNT
PRYCUT
PPvc
PyGyTIPyT
PvTRVT
PxcRvT
PrCytPPGVT
PoCPPCOT
PNOPPG
baok
(c) Multi-task sample
Figure Distribution of agreement scores for every pair of setups (task pairs sorted by median agreement score)
To sum up, even after controlling for accuracy, we can highlight qualitative difference in behav- ior: models of the same setup make more similar predictions than models of different setups. The contrast is starkest with single-task models; and while this effect tends be less pronounced in multi-modal models and very subtle in multi-task models, we can still observe It should not be sur- prising that models in the multi-task sample barely display a difference: recall that these models all use the same inputs (P features) on top of being conditioned on the same target prefix. That we can register difference at all after controlling for accu- racy suggests that the use of different input sources impacts the learned parameters. This effect cannot be subsumed mere difference in inputs that translates into different outputs; rather, it has to be construed as guiding models towards different behaviors. Referring back to our theoretical frame- work in Section 3.1, this is evidence in favor of the stronger notion of grounding that we outlined.
tion as Transformer decoders, we can further high- light the importance of the different input sources in model behavior by focusing on source-attention patterns. We compute the attention paid to features from P, C, and T, and contrast whether similar at- tention patterns entail similar behaviors. While attention-basse based overviews are known to have lim- itations (Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Vázquez et al., 2022), they do offer a practical insight into the models' computation.
In practice, for every multi-modal model and every item in our test set, we can compute the average attention per feature type in P. C, T. As such, every multi-modal model has a correspond- ing 3D vector describing its general attention pat- tern. These are displayed in Figure 3a. Given that the attention-patten one, they furthermore lie the positive quadrant of the & unit-sphere; i.e., the equilateral triangle with summits (1,0,0), (0,1,0), (0,0, 1). A visual- ization of the corresponding vector population is shown Figure 3b: each point corresponds to a
vectors' components
In multi-modal models, given their implementa-

---

in Table 1. If look at global results, or re- sults within given setup, we find the expected anti-correlation; however results appear contrary to our assumptions when comparing across setups. particular, comparing PAC and PAT multi-modal models yields a correlation instcad of the expected anti-correlation. In short, studying the attention patterns of the models explains some, but not all of the observed variation in behavior across mod- els. This lends further credit the stronger no- tion of grounding we outline in Section 3.1. If we were to assume the weaker notion instead, then anti- correlations should arise across different modalities as well: The presence or absence of a given input type should not perturb the underlying processing function g. Instead, what we observe is that the presence absence of a given input type can alter model behavior beyond what we can account for by solely looking at the distribution over input types, This suggests that we cannot disentangle input rep- resentation mapping from processing- i.e., that the stronger notion of grounding more appropri- ately describe the facts at hand. This corroborates our earlier observation with respect to multi- task models that grounding should not be characterized as providing different inputs the same function; rather, grounded models correspond to a different type of function altogether.
1.0
PACAT
80
02 mosttttte
1020
(a) 3D view
PACAT PaT
0.4
0.2
(b) 2D projection
Figure 3: 3D attention patterns of multi-modal sample
different model, hues correspond to input features; the three summits P. T and C marked with a cross would correspond to the position of models only attend to P. T or C input features respectively.
All multi-modal models exhibit different behav- iors: models trained on P and C tend attend to paraphrasing features rather than captioning fea- tures; models trained on and T tend to display more balanced patterns; attention patterns for mod- els trained with all three input spaces P, T and C vary more along how much they attend C and T fea- tures rather than how much they attend to P features. Remark that some models trained with C features may entirely ignore captioning features, whereas the same never holds for paraphrase and translation input features. This likely highlights that the NLG task at hand tends to be more straightforward using cross- cross-lingual, rather than cross- modal, features.
5 Concreteness
We have established that there is some evidence in favor of a stronger notion of grounding, that is to say, that expect the functions described by grounded models to differ across setups. We now turn whether we can connect these qualitative functional differences with other prior observations. Namely, we focus on concreteness (Pezzelle et al., 2021; Tikhonov et al., 2023), and study how it might affect model parameters (viz., the input em- bedding layer) as well as model activations (viz., the last hidden state before vocabulary prediction).
"We have replicated these findings with KL-divergences. however other statistical indicators need not align with this be- havior. In particular, differences in terms of entropy are rarely statistically significant, tentatively suggesting that grounding does not reduce uncertainty. In detail, we only observe a signif- icant difference for single-task models (Kruskal- Wallis Htstst:: p < 0.011) 1), which is owed to C models having lower entropy than P or T models (Mann-Whitney U-tests: P vs Cp < 0.04, 0.63; C vs T. < 0.0031, f = 0.31. This could be a side effect of selecting checkpoints at different number of epochs: If some models have been optimized for longer, they may have converged onto more peaked distributions. We lcave a proper evaluation of this aspect for future work.
Beyond these general observations, we focus on whether we can establish a correlation between distance in attention patterns and pairwise agree- ment scores. We expect anti-correlation, since lesser distance between attention patterns ought to correspond greater degree of agreement. We consider three measures of vector dissimilarity: co- sine distance, viz. 1 - cos (a1, b2), as well as L and & distances. Corresponding results are shown

---

6
Setups
p-val
p-val
p-val
PAC
PAC
0.26
8.57 10
0.18 1.99 1.99-10
0.27
2.04-10
PAC
PAT
0.11
7.25-10
0.07
3.23 10
0.09
2.51-10
PAC
PACAT
0.17
2.62-10-11
-0.11
1.98-10-11
-0.19
1.52-10 102-10-14
PAT
PAT
0.26
1.44-10
0.26
1.38-10
-0.6
1.38-10
PAT
PACAT
0.47
0.91
0.40
PACAT
PACAT
0.27
2.46-10-14
-0277
3.57-10" 14
00.2
7.98-10--2
overall
0.26
4.98-10
111
-0.88
1.88-10
124
-0.28
2.35-10"
335-10-15
same setup
0.18 1.07-10-16
0.18 1.99-10-1
0.19
5.29-10
Table 1:
Spearman correlation between attention patterns and agreement rates in the multi-modal sample. overall row: across all sampled multi-modal models.; same setup row: only comparisons involving models of the setup (P/P, PAC/PAC, PAT/PAT and PACAT/PACAT),
We use concreteness ratings of words from Brys- baert et al. (2014). These ratings were gathered through crowd sourcing where annotators were in- structed to rate how concrete the meaning of each word is on the scale of 1-5. Higher ratings char- acterize concrete words, which refer to percepti- ble entity and can be experienced directly through the senses (experience-based) Conversely, words with low ratings are abstract, referring to concepts that cannot be directly experienced; instead, their meaning has to be derived through other words danguage-baseede For simplicity, we binarize this dataset by taking the 1000 lowest rated words to construct a set of abstract words, and similarly the 1000 highest rated to construct concrete set.
embeddings than the P T models With the embeddings from the last hidden state, however, C has lower scores than P or T This shows that the input embedding layer of C models is better able to separate abstract and concrete words compared to models that received only text features (P and T models). This ability separate abstract from concrete words is reinforced in the clustering ex- periment where C models form purer clusters with both the input and last hidden state embeddings than and as shown by the higher purity and inverse purity scores. 12
In the multi-task models, PVC and PVCVT mod- els tend to have higher silhouette scores than P or PVT models in both cases where we use embed- dings from the input and last hidden state layers. In the clustering experiment, PVCVT tends form purer clusters than all other models when using the embedding layer representations but not when us- ing the last hidden state. As discussed previously, we do not expect models in the multi-task setup to display stark differences in behavior since that they all receive P inputs to generate predictions in the same output space. It is therefore somewhat surprising that models that receive video features (PVC and PvCVT) still behave differently from the models that receive only text (P and PVT).
We compare the embeddings of concrete and ab- stract words learned by models trained on the dif- ferent setups that we outlined. We consider two sce- narios: In the first, we assess whether concrete and abstract word sets cluster into two (and only two) distinct groups of embeddings: More precisely. we evaluate the compactness of each group through their silhouette scores. In the second, relax the assumption that embeddings need form only two clusters: instead, we perform clustering using affinity propagation (Frey and Dueck, 2007), a clus- tering method that does not rely on a predefined number of clusters, and evaluate the purity of the formed clusters." 9
The multimodal models exhibit a somewhat dif- ferent behavior. All models have similar silhouette scores and purity scores when using the embed-
Table 2 shows the silhouette and purity scores for the single-task, multitask, and multimodal models. In the single-task models, we observe that C modells-odelt that use video features exclusively-ccomsisteelll exhibit behavior differ- ent from P and T models. The C models have the higher silhouette scores when using the input
"Mann-Whitney Whitney U tests: P vs Cp < 7.4 10-5; C < 1.2 1.2-10-3 10-3 "P Cp 8.9 10 T p 3.1 10
ISSuch low inverse purity scores nonetheless suggest that concreteness not the most important factor when comes shaping representation spaces grounded models.
'Input embedding: P ys PvCVT p < 3.6 10-1, PVC vs PVCVT p < 2.8 10-2, PvCVT vs PVT < 6.00-00 10 I Last hidden state: P ys PVC p < 211 10-2; P PVCVT P < 7.1 10-3
"Wo use scikit learn (Pedregosa et al., 201 1) with damping to 0.90 and default values for all other parameters.

---

Silhouette
Purity
Inverse Purity
Model
imput
last hidden state
input
last hidden state
input
last hidden state
Single-task
0.021(2000008
0.054(00.007)
0. 0.746C:0.021)
0.887(::0.005)
0.062(=0.015)
0.04%
0.026(=0.002 002)
0.760(#0.013) 0.049(=0.006)
0.890(=00005)
0.080 025)
0.06
0.02M(200008
0.056(=0.006)
0. 0.748(+0.016)
0.884(2000055
0.060(#0.014)
0.047(=0.006)
Multitask
0.023 0.40)
0.051(=0.005)
0.757(+0.017)
0.888(1000005)
(073(+0.021)
0.050
PvC
0.024(+0.004)
(056(-0.008)
0.752(
0.061(:0 0.000(+0.021) 0.891()00007)
PVCVT
0.027(2:0000 0.4)
0.055(=0.007)
0.765(+0.016)
0.889(1:00000 006)
0.073(#0.00 0166)
0.05%(=0.009)
PVT
0.024(:1.004)
0.053(=0.008)
0.754(10.019)
0.889(10.006)
0.069(10.016)
0.050(00.007)
Multimodal
0.025( 0.025(+0.004)
0.0000- 0.007)
0.759( 0.002)
0.8900 F0.006)
0.074(+0.023)
0.054 0.010)
PAC
0.052(=0.006) 0.026(40.003)
0.891(+0.007) 0.761(+0.018)
0.073(+0.019)
0.05((=0.006)
PACAT
0.024(20.004)
0.759(#0.017) 0.056(00.007)
0.072(#0.032) 000007)
0.048(=0.005)
PXT
00.06(+0.004)
0.051(=0.008)
0.765(+0.017)
0.886(+0.007)
0.065(+0.014)
0.006(=0.005)
Table 2: Silhouette and cluster purity scores for single-task, multitask, and multimodal models. Cells in bold correspond optima per embedding and setup when the difference with the second best value is significant.
dings from the input layer. But with the last hidden state embeddings, PACAT and PAC models show higher silhouette and purity scores, respectively, than the models that do not incorporate video fea- tures in their inputs.' 14 This suggests that models that have to process multimodal features (visual and textual) in order to generate a prediction in unimodal (text-only) output space behave differ- ently from models that process inputs from the same modality. Overall, our experiments in this section show that models that receive video or text and vidco features learn parameters that are subtly different from models that only receive text inputs despite the fact that they are trained to generate predictions in the same output space.
eating abstract and concrete words.
The methodology we outlined in Section 3.4 allowed us to select groups of models so as to guar- antee they are of similar accuracy, thereby teasing apart variation due to performance from that due to the type of input sources model receive. This methodology can easily be adapted any other situation where one wishes to control for a spe- cific variable. our specific case, it allows us to smooth over the necessary differences between text and video features and provide a more principled outlook on grounding. We believe the adoption of methodologies such as the one proposed here to be invaluable to studies on grounding: They allow for systematic empirical comparisons which are of- ten missing, as most studies focus on foundational models or theoretical arguments.
6 Conclusion
Beyond these methodological contributions, the core finding our experiments suggest is that the effects of grounding on neural network models can- not be subsumed to simply learning adequate rep- resentations: Rather, observe results consistent with stronger notion of grounding, where we can- not disentangle the effects of learning to represent inputs of different modalities or languages from that of learning how to solve the task at hand.
In the present work, we have proposed to study cross-lingual and cross cross-modal grounding by con- structing samples of comparable models.
Our experiments have revealed how populations of models using different types of input sources make diffferntt textual predictions. At a global dataset level, we have observed that models tend to agree less when they come from different setups than when they share the same training conditions (Section 4). Adopting more linguistically moti- vated approach, we have also discussed how the representations concrete and abstract words are impacted by grounding (Section 5): in particular, we can highlight that our captioning- task was the most impactful when it came to cleanly delin-
A second conclusion that emerges from our ex- periments is that the effects of cross- loossllnngul ground- ing are empirically distinct from that of cross- modal grounding. Cross-lingually grounded mod- els were found in several instances to be more closely in line with models trained on a paraphrase- like task. better characterization of the difference between cross -ingull and cross cross-moda modal grounding is one direction we hope to address in future work.
"P vs PACAT < 2.3 - 10-3; PAC vs PACAT p < 1.5 2: vs PAT < 1.6 1

---

Limitations
generally known english word lemmas. Behavior research methods, 06:904-911.
One crucial limitation of the proposed upproach is that it ignores how scaling up can impact the phe- nomena we study. Scaling up, in terms of number of parameters and volume of data a model is ex- posed to, can lead to behaviors that need not be ob- servable in our more restricted experiments. Repli- cating our experiments to larger models, while in principle possible, poses practical challenges: The experiments we have conducted here entail train- ing hundreds of models. This is likely not real- istically feasible for parameter-intensive models, which are often the core focus of current discus- sions about grounding. The emergence of ground- ing in large NLP models is therefore left untouched in the present article.
Khyathi Raghavi Chandu, Yonatan Bisk, and Alan Black. 2021. Grounding 'grounding" in NLP. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 4283 4305, On- line. Association for Computational Linguistics.
Grzegorz Chrupata. 2022. Visually grounded models of spoken language: A survey of datasets, architec- tures and evaluation techniques. Journal of Artificial Intelligence Research, 73:673-707.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-A 1171-4186, Minneapolis, Minnesota. Association for Computational Linguisties.
Lastly, the proposed approach remains limited in scope: We have only focused on one dataset, one architecture, two languages and two modalities. While we have striven to make our experimental protocol as broadly replicable and adaptable as pos- sible, any conclusions drawn in the present paper need not carry to other setups.
Helge Dyvik. 2004. Translations as semantic mirrors: from parallel corpus to wordnet, pages 309 326. Brill, Leiden, The Netherlands.
Dylan Ebert, Chen Sun, and Ellie Pavlick. 2022. Do trajectories encode verb meaning? In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2860-2871. Seattle, United States. Association for Computational Linguistics.
Ethics Statement
This work complies with the ACL Ethics Policy. We believe the present work does not involve signif- icant ethical risks and that the results do not entail a broader ethical impact.
Darja Fišer. 2009. Leveraging parallel corpora and existing wordnets for automatic construction of the slovene wordnet. In Human Language Technology. Challenges of the Information Sociery: Third Lan- guage and Technology Conference. LTC 2007, Poz- nan, Poland, October 5-7, 2007, Revised Selected Papers 3, pages 359-368. Springer.
A Acknowledgments
This work is part of the FoTran project, funded by the European Research Coun- cil (ERC) under the EU's Horizon 2020 re- search and innovation program (agreement No 771113). We also thank the CSC-IT Center for Science Ltd., for computational re- sources. This work is also supported by the ICT 2023 project anncertainty-wwar neural language models" funded by the Academy of Finland (grant agreement No 345999).
Brendan Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. science, 315(5814):972-977 976.
Sadaf Ghaffari and Nikhil Krishnaswamy. 2023. Grounding and distinguishing conceptual vocabulary through similarity learning in embodied simulations.
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1):335 346.
Louis Hjelmslev. 1943.
Omkring sprogteoriens grundlaggelse (Prolegomena to a Theory of Lan- guage).
References
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5185-5198, Online. Association for Computational Linguistics.
Po- Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander G Hauptmann. 2021. Multilingual multimodal pre-traine pre-training for zero- shot cross- mooss-inguual transfer of aision-language models, In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2443-2459.
Mare Brysbaert, Amy Beth Warriner, and Victor Ku- perman. 2014. Concreteness ratings for 40 thousand

---

Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In AAAL
Transactions of the Association for Computational Linguistics, 1-1563-1579,
Scott Reed, Konrad Zolna, Emilio Parisotto, Ser- gio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals. Mahyar Bordbar, and Nando de Freitas. 2022. A generalist agent. Transactions on Machine Learning Research. Featured Certification.
Frank Jackson. 1982. Epiphenomenal qualia. The Philo- sophical Quarterly (1950-). 32(127):127-
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li. and Tom Duerig. 2021. Scaling up visual and vision- lasion-angguae representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916, PMLR.
Richard H. Richens. 1956. Preprogramming for mechanical translation. Mechanical Translation, 3(1):20 25.
Khazar Khorrami and Okko Räsänen. 2021. Can
phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? a com- putational investigation. Language Development Re- search, 1 (1):123-199)
John R. Searle. 1980. Minds, brains, and programs. Behavioral and Brain Sciences, 3:417-424.
Sofia Serrano and Noah A. Smith. 2019. Is attention in- terpretable? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3931-29511 Florence, Italy. Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: iision-and-lagguee transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning, pages 5583-5594. PMLR.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4604. PMLR.
Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semanties aligned pre-training for vision-language tasks. In Computer Vision-EC ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16. pages 121-137. Springer.
Mollie Shichman, Claire Bonial, Austin Blodgett, Tay- lor Hudson, Francis Ferraro, and Rachel Rudinger. 2023. Use defines possibilities: Reasoning about object function to interpret and execute robot instruc- tions. In Proceedings of the 15th International Con- ference on Computational Semantics (IWCS 2023).
Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang. Dongdong Zhang, and Nan Duan. 2021. M3p: Learning universal repre- sentations via multitask multilingual multimodal pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3977-3986.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from con- text? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Mitja Nikolaus and Abdellah Fourtassi. 2021. Mod- cling the interaction between pereeption-based and production- based learning in children's 's carly acqui- sition of semantic knowledge. In Proceedings of the 25th Conference on Computational Natural Lan- guage Learning, pages 391 107.
Jörg Tiedemann. 2018 Emerging language spaces learned from massively multilingual corpora. In Pro- ceedings of the Digital Humanities in the Nordic Countries 3rd Conference (DHN 2018). volume 2084 of CEUR Workshop Proceedings, pages 188-197, Unknown. CEUR Workshop Proceedings. Digi- tal humanities in the Nordic Countries DHN2018. DHN2018: ; Conference date: 07-03-2018 Through 09-03-2018.
Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In Interna- tional Conference on Learning Representations.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 282822-2830..
Aleksey Tikhonov, Lisa Bylinina, and Denis Paperno. 2023. Leverage points in modality shifts: Comparing language-only and multimodal word representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Sandro Pezzelle, Ece Takmaz, and Raquel Fernández, 2021. Word representation learning in multimodal pre-trained transformers: An intrinsic evaluation.

---

Raúl Vázquez, Hande Celikkanat, Vinit Ravishankar, Mathias Creutz, and Jörg Tiedemann. 2022. closer look parameter contributions when training neural language and translation models. In Proceedings the 29th International Conference on Computational Linguistics, pages 4788-4800, Gyeongju, Republic of Korea. International Commitice Computational Linguistics.
Xin Wang. Jiawei Wu, Junkun Chen, Lei Li, Yuan- Fang Wang, and William Yang Wang. 2019. Vatex; large-scale, high-quality multilingual dataset for jideo-and-languuge research. In The IEEE Interna- tional Conference Computer Vision (ICCV).
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing DAMNLP-IJCNLL pages 1-200, Hong Kong, China. Association for Computational Linguistics.


Reviews:
[{"review_id": "uqmgYCgYcf", "reviewer": "Reviewer_dJYX", "summary": "", "questions": "a) You note that the experimental setup presented focuses on two languages (Chinese & English) and two modalities (text and video).  Do you have any thoughts on how your results might change for variations in either variable?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review and give comments on our work. \n\n**Regarding reproducibility:** \n- For the hyperparameters, we use cross entropy loss with Ada Factor optimizer and a batch size of 1024. All the decoders we trained have 6 layers, 8 head per multihead sublayer, latent dims of 512, and latent feedforward dims of 2048. In total, the models have approximately 64M parameters. This information will be included in the final version of this paper. We will also release our code, including the model architectures for the experiments conducted in the paper. \n- The VaTex dataset is publicly available at https://eric-xw.github.io/vatex-website/download.html, we use the validation set as the test set and split the training set into training/validation sets. We will add the link to the dataset in the final version of the paper.\n\n**Regarding the changes in language pairs:** we expect that the more similar two language pairs are, the closer they will behave to an ungrounded model. We are uncertain on how different modality pairs would behave and this is something that we would like to investigate in future work.\n\n**As for definitions,** the final version of our paper will incorporate a more thorough explanation of the weak and strong notions of grounding, along with explicit definitions for grounding and cross-lingual grounding borrowed from Chandu et al (2021) and Tiedemann (2018).\n\n**Regarding Figure 3,** the vertices of the triangle (marked by x) correspond to the point where features from only one modality are attended to. In other words, the point marked P would correspond to (0, 0,1), T to (0, 1, 0) and C to (1, 0, 0) in a 3D space. Figure 3 presents the plane that contains these three points, since all attention vectors lie on this plane. What we want to convey here is that for PCT models, features from English text, Chinese text, and video are attended to to some degree. We will provide a 3D plot of Figure 3 in the final version to make this clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed experiments are a novel way to address the issue of comparability across different modalities.", "reasons_to_reject": "It would be great to add more information about the model explicitly (e.g., parameters) for reproducibility.", "questions_to_authors": "a) You note that the experimental setup presented focuses on two languages (Chinese & English) and two modalities (text and video).  Do you have any thoughts on how your results might change for variations in either variable?", "paper_topic_and_main_contributions": "This paper discusses an experimental setup for testing grounding in cross-modal (text & video) and cross-lingual (Chinese & English) settings.  Code and dataset will be shared.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line052 To -> to\n\nGiven the diversity of the EMNLP audience, I think it would be helpful to have clear statement of what 'grounding' is - perhaps before the provided definition in line121 onwards. \n\nline136 survey of Chrupała (2022) > survey by Chrupała (2022)\nline315 grredy -> greedy\n\nIt's difficult to interpret figure 3 without labelled axes."}}, {"review_id": "O7FgJT9urB", "reviewer": "Reviewer_CJ8b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review and comment on our work.\n\nWe are happy to clarify the **distinction between the weak and the strong senses of grounding**, on which we rely in the paper.\n\nIn the weaker sense, a system for processing language X is grounded if it can accept input in a different modality than language X. The most common example of the other modality is visual, although this is not the only possibility.\n\nHowever, some of the recent literature suggests grounded systems should have further nontrivial properties than simply being able to (meaningfully) process alternative modality input. \n\nSo a strongly grounded system for processing language X not only is equipped with the capacity to handle an alternative modality, but also processes input in language X in a different way than a comparable ungrounded system. For instance, a grounded system might be less likely to hallucinate about perceptually salient properties even if there is no information about them in the specific input.\n\nA system can be (weakly) grounded without being strongly grounded. For instance, it can process text by passing it through an ungrounded system, and process images by generating their captions and then passing them through an ungrounded system.\n\nOne of the questions we explore in the paper is whether strong grounding, i.e. nontrivially different language-only processing by grounded systems, can be observed in a controlled setting.\n\nThe final version of our paper will incorporate a similar explanation of the weak and strong notions of grounding.\n\n**As to Figure 2,** we elected to use a density plot to visualise the spread of the agreement values. We intend to switch to violin plots to make these figures less confusing for readers.\n\n**As to Figure 3,** the vertices of the triangle (marked by x) correspond to the point where features from only one modality are attended to. What we want to convey here is that for PCT models, features from English text, Chinese text, and video are attended to to some degree. We would be happy to provide a 3D plot of Figure 3 in the final version to make this more clear.\n\n**As for missing details:**\n- For combining features, we concatenated the features  and passed the concatenated vector to a linear layer. We will clarify the information conveyed in footnote 5 in the final version. \n- We will report the sizes of the models in the final version. All the decoders we trained have 6 layers, 8 head per multihead sublayer, latent dims of 512, and latent feedforward dims of 2048. In total, the models have approximately 64M parameters. \n- Regarding the “two notions of scales” our intent was to stress that 1/ our models are smaller than widely-used foundation LLMs used nowadays and we expect that such large models might display emergent behaviour not seen in smaller models, and 2/ replicating these experiments with LLMs will also pose practical challenges because our proposed experimental framework involves updating the parameters of hundreds of models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is important to understand how deep neural networks merge/fuse information from different data sources, so the overall research direction of the paper is worthwhile.", "reasons_to_reject": "The paper lacked clarity. I found it difficult to understand what was happening beyond the empirical fact that the different model types behaved differently.\n\n-Figures 2 (Which model runs are covered in this distribution? Is it really a continuous density function?) and 3 (what are the axes?) are difficult to interpret. \n\n-I'm not sure I fully understand or buy into the difference between weak and strong grounding. Why wouldn't it be possible for data in different modalities to be represented in the same vector space, but contain different information, thus leading to different model output behavior. An image and a sentence can both be mapped to a rich space where a vector represents a distribution over possible captions/rewrites. The transformation from raw input into that space would still be different depending on the modality.\n\n-Some details are missing from the implementation. For example, are the different feature types in the multi-modal scenario simply concatenated? What are the network sizes involved in each scenario? Is this what the authors refer to as \"matters of scale on two different axes\" in the Limitations section?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper examines the effects of multi-modal learning in transformers. This is couched in the notion of \"grounding\", which refers to making inferences about an English phrase, given access to additional information like an associated image or equivalent Chinese phrase.\n\nThe authors compare single task models, multi-task models (in which each training sample input is either an image and English phrase, or a Chinese phrase), and multi-modal models (where the different data types are concatenated in the same training sample). All the models output in the same space (English text).\n\nLooking at metrics based on the output agreement between these configurations, the authors suggest that multi-modal models are \"strongly\" rather than \"weakly\" grounded - though I found the distinction confusing (see below).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YIV7p14bcd", "reviewer": "Reviewer_yRLh", "summary": "", "questions": "l. 375 \"p < 10e−256\" This p-value is smaller than one over the number of atoms in the observable universe - such small p-values are usually suspicious, have you checked the assumptions of your test?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read this paper and provide their thoughtful comments and insights.\n\nWe are happy to clarify the **distinction between the weak and the strong senses of grounding**, on which we rely in the paper.\nIn the weaker sense, a system for processing language X is grounded if it can accept input in a different modality than language X. The most common example of the other modality is visual, although this is not the only possibility.\n\nHowever, some of the recent literature suggests grounded systems should have further nontrivial properties than simply being able to (meaningfully) process alternative modality input. \n\nSo a strongly grounded system for processing language X not only is equipped with the capacity to handle an alternative modality, but also processes input in language X in a different way than a comparable ungrounded system. For instance, a grounded system might be less likely to hallucinate about perceptually salient properties even if there is no information about them in the specific input.\n\nA system can be (weakly) grounded without being strongly grounded. For instance, it can process text by passing it through an ungrounded system, and process images by generating their captions and then passing them through an ungrounded system.\n\nOne of the questions we explore in the paper is whether strong grounding, i.e. nontrivially different language-only processing by grounded systems, can be observed in a controlled setting.\n\nThe final version of our paper will incorporate a similar explanation of the weak and strong notions of grounding.\n\n**With regards to lexical concreteness,** we agree that concreteness is on scale and the dataset we use in our experiments are annotated on this assumption. On the other hand, we strongly expect polar opposites on the concreteness scale to be clearly distinct. This motivates us to take the words from the top and bottom of the scale in order to demonstrate more clearly whether a model can distinguish between words from the extreme ends of the scale.\n\n**Regarding cross-lingual grounding,** it has been posited (Tiedemann, 2018) that translation-–the task we use to demonstrate cross-lingual grounding in our experiments—is an implicitly grounded process because it consists of producing an utterance in a different language that is communicatively equivalent to the source sentence. \n\nFor example, consider the English word \"sheet.\" It can have different French translations, depending on whether we are talking of bed sheets (_draps_) or sheets of paper (_feuille_). Training a model to translate from French to English is likely to foster such semantic distinctions: such a model is likely to learn that _feuilles_—sheets are likely to be written on or put into envelopes, but not slept on or washed, which in turn should shape the probabilities the model assigns to the corresponding words. In contrast, such distinctions might not be as easily available to monolingual models. On the other hand, similar distinctions are expected to be fostered by captioning tasks: models trained to generate captions should learn that the symbol \"sheet\" applies to two broad classes of stimuli, one stereotypically including large cloth rectangles, the other stereotypically including small paper rectangles. \n\nIn other words, a cross-lingual training signal is expected to foster otherwise unavailable semantic distinctions in a manner similar to what we expect of a cross-model training signal. We agree that cross-lingual grounding is less well-established than cross-modal grounding. Our experiments provide some basis to support the concept of cross-lingual grounding and we hope that readers will find these results valuable and encourage more exploration in this notion of grounding. \n\n**As for Figure 2:** We thank the reviewer for the suggestion. Our intent was to indeed visualise the spread, rather than the average agreement, in line with our use of U-tests (which are tests of the full distribution). We intend to switch these visualisations for violin plots in the final version of this paper.\n\n**As for the p-value line 375:** We thank the reviewer for this remark. U-tests make minimal statistical assumptions, mainly that observations are independent. We would be interested in any suggestion for a more appropriate statistical treatment; nonetheless, we consider that the reported p-value, while it may be inflated, is low enough to be  indicative of a significant difference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] I think that comparing (a) single-task, (b) multi-task (models trained to solve a task with different input sources) and (c) multi-modal (models trained to solve a task relying on different input sources at once) models while controlling for accuracy is a nice idea. In particular, I find the comparison between (b) and (c) particularly interesting.\n\n[2] A lot of care was put into constructing several candidate models for each task, by training for different numbers of epochs or adding gaussian noise to encoder representations. \n\n\n[3] Several follow-up analyses try to understand the qualitative differences observed in the main experiment. These include an analysis of the attention weights.", "reasons_to_reject": "[1] \"Weak\" and \"strong\" notions of grounding are defined very loosely in an intuitive way; I would have appreciated a more rigorous definition.\n\n[2] The two experiments (the agreement one and the concreteness one) conceptualize the effects of grounding in two rather opposed ways: the first one controls for accuracy and looks at the agreement patterns between models, while the second considers concreteness ratings as a ground truth, and tests which kinds of models are closer to the ground truth. \n\n[3] Concreteness is a lexical dimension that is inherently continuous and graded, and I don't think that dichotomizing it is a good idea. \n\n[4] ll 154 - 176: It is not clear to me what you mean by \"cross-lingual **grounding**\". In the introduction you refer to Searle and Harnad, but that framework only applies to extra-linguistic grounding. Models that process symbolic input in several languages still lack grounding. Also, the literature on multilingual modeling is not well discussed; there are some citations that are arguably not relevant (e.g., multilingual Wordnets), and some important missing ones. \n\n[5] Figure 2: I feel there's a need clearer visualizations. The lines are very thin and it is difficult to see the colors. Further, it would be very helpful if line patterns were grouped (say, dotted lines for consistent pairs as C/C, P/P, etc.). Also: why density plots? What you're interested in is arguably the average agreement per condition, but density plots give the illusion that the y-axis (density) is what matters (which here it's not the case).", "questions_to_authors": "l. 375 \"p < 10e−256\" This p-value is smaller than one over the number of atoms in the observable universe - such small p-values are usually suspicious, have you checked the assumptions of your test?", "paper_topic_and_main_contributions": "The authors test whether grounding (i.e., providing the model with richer data than only text) causes qualitative differences in the linguistic abilities of language models. The study considers two types of grounding, namely multilingual (English-Chinese) and multimodal (text-video). They found that that is indeed the case; models of the same type agree more than models of different types (e.g., multimodal models produce similar predictions to multimodal models, after controlling for accuracy). Furthermore, multimodal models better capture the concreteness of words.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 317 \"grredy\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies whether grounding (defined as providing the model with data in modalities that are richer than just text) causes qualitative differences in behaviour of language models as measured by differences in attention patterns and comparisons of embeddings of the models. Reviewers state that the additional definitions of grounding introduced here (“weak” vs. “strong”) are not defined clearly enough, the quantitative analysis should be strengthened and better metrics and visualisations should be used to fully back up the claims made in the paper about whether models that have weak vs. strong vs. no grounding do differ in their behaviours. This paper has useful findings but would be good to have a deeper evaluation analysis to more thoroughly answer the questions here.
Decision: Accept-Findings