Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning
Authors: Yizhen Zhang
Minkyu Choi
Kuan Han
Zhongming Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: language learning
visual grounding
visual relational reasoning
cross-modal contrastive learning
grounded cognition

Abstract: In natural language processing, most models try to learn semantic representations merely from texts. The learned representations encode the “distributional semantics” but fail to connect to any knowledge about the physical world. In contrast, humans learn language by grounding concepts in perception and action and the brain encodes “grounded semantics” for cognition. Inspired by this notion and recent work in vision-language learning, we design a two-stream model for grounding language learning in vision. The model includes a VGG-based visual stream and a Bert-based language stream. The two streams merge into a joint representational space. Through cross-modal contrastive learning, the model first learns to align visual and language representations with the MS COCO dataset. The model further learns to retrieve visual objects with language queries through a cross-modal attention module and to infer the visual relations between the retrieved objects through a bilinear operator with the Visual Genome dataset. After training, the model’s language stream is a stand-alone language model capable of embedding concepts in a visually grounded semantic space. This semantic space manifests principal dimensions explainable with human intuition and neurobiological knowledge. Word embeddings in this semantic space are predictive of human-defined norms of semantic features and are segregated into perceptually distinctive clusters. Furthermore, the visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based on images, texts, or their combinations.

Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning
Yizhen Zhang!? Minkyu Choi', Kuan Han', and Zhongming Liu
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109 Department of Neurological Surgery, University of Califomna San Francisco, San Francisco, CA 94143 Department of Biomedical Engineering, University of Michigan, Ann Arhor, MI 48109 {zhyz, cminkyu, kuanhan, zmliu] edu
Abstract
In natural language processing, most models try to learn semantic representa- tions merely from texts. The learned representations encode the "distributional semantics" but fail to connect to any knowledge about the physical world. In contrast, humans leam language by grounding concepts perception and action and the brain encodes "grounded semantics" for cogmition. Inspired by this notion and recent work vision-language learning, we design two-stream model for grounding language learning in vision. The model includes a VGG-based visual stream and Bert-based language stream. The two streams merge into a joint representational space. Through cross-modal contrastive learning. the model first learns to align visual and language representations with the MS COCO dataset. The model further learns retrieve visual objects with language queries through a cross-modal attention module and to infer the visual relations between the retrieved objects through bilinear operator with the Visual Genome dataset. After training, the model's 's language stream a stand-alone language model capable of embedding concepts in visually grounded semantic space. This semantic space manifests principal dimensions explainable with human intuition and neurohiological knowl- edge. Word embeddings in this semantic space are predictive of human-defined norms of semantic features and are segregated into perceptually distinctive clusters. Furthermore, the visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based images, texts, their combinations.
Introduction
Humans take much longer time name a colored word when the color and the word mismatch (e.g..
"red" shown in green) than when they match (e.g., "red" shown in red) [1]. This effect is example of rich psychological evidence suggesting that humans learn language by grounding meanings to knowledge about the world [2, 3J. In contrast, most models in natural language processing (NLP) [4-7] encode "distributional semantics" " [8] learned from texts only. Put yourself as machines in thought experiment for the "Chinese Room Argument" [9]. Imagine that you have to learn Chinese from scratch as your first language. All that you have is a Chinese-to-Chine be able to relate word to other words based textual distributions. It is, however, impossible to learn word meanings without any additional explanation in reference to the physical world [10].
dictionary. You might
A language model may learn concepts from texts paired with sensory data, such as images. Joint vision- languaae learning has been explored for image captioning [11], visual question answering [12), and pre-training vision models with weak supervision [13, 14]. In line with these studies, we
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

train a language model and a vision model jointly to match images and texts. We further analyze the semantic space obtained with the visually grounded language model. this space, semantic embeddings are found to be organized and clustered by visual attributes, predictive of human-defined norms of semantic features, useful for compositional language understanding and mross-modal image search. We expect this visually grounded language model to also be useful for understanding the computational basis of grounded cognition [15, 16].
V
Mard
Linear head
CNN
Embestier
Attention
Solf
f
M Muster
Contrastive Learning
Visual Input
M, V.
Mas
S(V
A bird sits on a parking meter with several parked cars in the street.
Linear head
Matrr
Language Input
Figure I: Visual grounding of natural language (see Section 3.1). The visual and language streams take image and its caption as input, respectively. The inner-product between the visual feature maps and the contextual word embeddings forms the 3D match-map that highlights the matching between visual and language content. The similarity score calculated from the match-map (see Eqq used to evaluate the cross-modal contrastive loss.
2 Background and Related Work
2.1 Distributional vs. grounding hypothesis
In the distributional hypothesis [17], words that occur in similar contexts carry similar meanings. This hypothesis has motivated influential machine learning models to learn word embeddings from large text corpora [4, 18). However, learned word embeddings are not straightforward to interpret [8, 19]. Alternatively, the symbol grounding hypothesis suggests that word is connected to its meaning by relating to referent in the physical world [10, 20). line with this hypothesis, earlier studies demonstrate that visual features or contexts can enhance language learning [21-25].
2.2 Vision-language learning
Grounding language in vision has been of increasing interest in computational linguistics and machine learning. A common strategy is to fuse words with related visual information in terms of perceptual norms [21], bag-of-visual-worr [23, 26), or learnable visual features [27-30]. The models used for vision-language fusion evolve alongside those for NLP, such Latent Dirichlet Allocation (LDA) [23], log-bilinear model [31], Skip-gran gram model [25, 27], and recurrent neural network [28]. Generally, visual grounding may refine the distribution and interpretability of language representations [23, 25, 26, 28-32] and facilitate cross- mods- tasks [28-33]. More recent work has begun to use transformer [6, 34] for vision-language learning, showing strong performance in cross-moda tasks [35-38].
Contrastive learning [39, 40] is increasingly applied to not only unimodal data [41] but also multi- modal data [13, 42). is able to learn better representations than alternative prediction or classification objectives [43]. However, cross-modal contrastive learning is still under-explored for higher-level tasks, e.g., visual question answering [12), visual reasoning [44], scene graph generation [45]. Such tasks involve abstract reasoning about the relations between entities (e.g., visual objects). Prior

---

work approaches relational inference with multi-layer perceptron [46, 47] graph neural networks [48-50]. Arguably, more compelling idea [51] is to model entities as vectors in a continuous space and to model their relations as arithmetic operators (linear [52-54] or bilinear [55, 56]) applied to the vector representations of those entities.
2.3 Relation to prior work
In this work, we first build a two-stream model to jointly learn visual and language representation from image-caption pairs, similar to recent work [13, 14j. We then finetune the learned model by adding a cross-modal attention layer [35, 36] and bilinear operators [55] to represent the relations between visual objects, Both stages utilize eross-modal contrastive loss. Related to our work, Harwath al. match visual objects spoken words using triplet loss [42]. Early this year, Jia al. [13] and Radford and Kim et al. [14] use contrastive learning to pretrain a vision model using massive image-text dataset and demonstrate largely improved zero-shot transfer learning performance on visual and cross-modal tasks. Different from their perspectives, we focus on assessing the language encoders and word representations. Specifically, we perform a systematic evaluation of the semantic space grounded in vision vs. the ungrounded semantic space learned from texts only. This evaluation is possible since after training, the language and visual streams in our model are fully separable as stand-alone systems, unlike some vision-language models that require both visual and textual input to be usable [35, 36]. Our goal is to assess how visual grounding affects the distribution of textual representations by analyzing the distribution of word embeddings in the grounded semantic space, in line with related works [23, 25, 26, 28-32, 57].
Approach
3.1 Visual grounding of natural language
To build computational model for learning visually grounded language representations, we develop a model (Fig. I) that combines a stand-alone visual stream and a stand-alone language stream. The visual stream based on VGG16 [58] with an additional linear transformation as an embedder to match the feature dimension of the language stream and an additional multi-head self-attention layer [59] to enforce global information aggregation and learn long-range dependency. The language stream is based on Bert [6]. Using separate linear transformation heads [41], the output from both the visual stream and the language stream are projected to a common representational space. In this common space, the inner-product between the visual representation V at every location and the language representation of every word gives rise to 3D match-map, where each element indicates how word in the text matches each location in the image (See illustration in Fig. 1). The sum of the maximal match the similarity score S(V, L) between a pair image and text. See Eq. where i,j indicate the location in the 2D image feature map V and & indicates the k-th word in L.
MaJA = Vis LI, S(V, L)
Σ max M...k 2.3
(I)
Extending the unimodal normalized temperature-scaled cross- entsopy (NT-Xent) loss [13, 41, 60], we define the cross-modal contrastive loss using the anchor sample from one modality and the positive sample and negative samples from the other modality functions with the anchor sample from either images or texts and positive/negative samples from either texts or images, respectively.
[13, 14]. As such, we define and sum two loss
Lossi Σ log exp(S(V,, L,)/T)
B
exp(S(V, L.)r)) Loss, Σ log Σ SSV,
B
(2)
exp(S(V.,,
For Loss; Eq. 2, the anchor sample V. an input image and the positive sample L; is the corresponding image caption, whereas the negative samples Lj are unmatched textual descriptions included the same batch (B the batch size). Similarly, Loss, in Eq. 2 is defined to contrast the positive and negative image samples against anchor textual sample.
3

---

3.2 Visual grounding of object relations
We further finetune the model for visual relation prediction, as illustrated in Fig. 2. In this stage, we remove the linear transformation heads in Fig. I and add multi-heate cross-modal attention module [35, 36]. The attention module uses query based on the embedding of an object word from the language stream (Queryr.) and uses keys (Keyv) and values (Valuev) from every location in the feature map output from the visual stream. The attention score calculated as the inner-product of Queryz and Keyv followed by softmax. The attention-weightee sum of Valuey is concatenated across 8 attention heads to generate visually grounded object representation.
Multi-head Cross-modal Attention Module
Valuey
Visually Grounded Object
Visual
wrighted sm
Stream
concutoration
Keyy
safbres
Representation
Attention Score L V
imceepodtnct
Auxiliary Loss
clephant water pond ripples
Language Stream
Query's
Linar Sulspace
Bilinear Relation Module
object representation
relation score
Bilinear Relation Module
person
elephant
water pond
To
subject
ripples
relation embedding
S(re: roi Rp)
stand in
R2
Contrastive Learning
S(r Ry) R.y
Figure 2: Visual grounding of object relation. The language stream object description as input (e.g., large black elephant; we only show the object name "elephant" in this illustration for simplicity). The multi-head cross-attention module outputs representations (See detailed methods in Appendix A.3). The bilinear relation module (bottom left) further generates relation score given representations of (elephant, in, water pond)) for contrastive learning.
of visually grounded object
object) triplet (c.g.
Applied to the grounded object representation a bilinear relation module for predicting the visual relation between objects (linguistically a subject and object). For both the subject and the Rd (default object, their grounded representations are linearly transformed to a subspace learnable bilinear d 32), denoted as r, and ro, respectively. A predicate p is represented operator F, x + R. which represents the relation embedding RG- Applying this bilinear operator to the subject object representations measures their relation score S specific the given predicate [55] expressed as (Eq. 3).
S(ra, R,) =
For visual relation prediction, we also use contrastive learning with two loss functions by taking either relation embedding or subject/object representations as positive/ncgative samples.
exp(S(r.. Rp)/T)
Loss
Σ
log ΣkEK (xp(S(r,,,o)) Rt)/T)
(4)
[B]
(oi,)e)
exp(S(re, To: Rp)/T)
LosSobi
log
B
EEeK exp(S(rt, rē: NR: Rp)/T)
(Paito: Rp)CB

---

In Lossjei, Kee is the set that contains all relations available. The anchor sample is pair of subject and object in an image. The positive sample is the embedding of the ground truth relation. The negative samples are the embeddings of all other relations. In LoSSabj. the anchor sample is given relation. The positive sample is a subject-object pair that holds this relation. The negative samples are other subject-object pairs in a different relation. For both loss functions, the positive and negative samples are drawn from the same batch B.
In addition, we also add a classification head (two fully connected layers with ReLU in between) and apply it to the grounded object representation. We use object classification as an auxiliary objective (with a cross- cross-entropy loss) to constrain the grounded object representation to be separable across objects for classification.
3.3 Training and Testing
We train the model in three stages to progressively refine the model with increasingly demanding tasks. In the first stage, we pretrain the visual and language streams separately as image and text encoders. The language stream is the pretrained Bert' used as the baseline model for subsequent experiments. The visual stream is pretrained for object classification with ImageNet [61]. Relative to the baseline CNN, the inclusion of self-attention improves the top-1 classification accuracy from 71.6% to 74.3% on the ImageNet validation dataset. The attention module also renders the classification more robust when the input image is partially occluded (See details in Appendix A.1).
In the second stage, we refine the pretrained language and visual streams by matching texts to images, as illustrated in Fig. on the MS COCO dataset [11]. While freezing other layers, we refine the self-attention layer in the visual stream and the top k layers in Bert (by default k 8). Training with contrastive learning is based on the MS COCO dataset. As five captions are available for each image, we randomly sample one caption per image in each iteration. Earlier grounding (larger k) tends to support better image-text retrieval performance (see details in Appendix A.2).
In the third stage, we further finetune the model for visual relation prediction as illustrated in Fig. 2. We refine the visual self-attention layer and the higher 7 layers in Bert (by default = 2) based on the Visual Genome dataset [45] after cleaning the dataset to include 114 relations and 55 object classes in order to alleviate imbalanced data across different classes or relations (Appendix A.3). The training does not use any image annotation (e.g., bounding box). which otherwise requires other models (e.g., object detection). Instead, we use cross- attsenttttntoon to retrieve visual objects from raw images given textual queries and learn the representations of the retrieved objects and their relations altogether. After training, the model predicts the object class with 97. 97%71% top-1 I accuracy and predicts the relation label with 64.26 64.26% top-1 accuracy (See details and examples in Appendix A.3).
4 Experiments
4.1 Principal components of grounded semantic representations
To evaluate the visually grounded semantic space, we use the language stream as stand-alone model extract the output representations of commonly used English words the SemCat dataset (9, 197 words; 100 word categories) [62]. Details about how word representations are extracted from the language stream are explained in Appendix B. We apply the principal component analysis to the representations of all the words studied here and examine the top components as the principal dimensions of the grounded semantic space.
Table 1: Correlation between the 1st principal axis and human-rated word concreteness
Correlation (Pearson's 's
Relational Grounded
Group
Bert
Grounded
word-level
0.1040 0.3538
0.6615 0.8749
0.6948
category-
0.8001
'bert uncased: https://hugginge http://pww.....oeeeaaeaa/eapptaa html
5

---

Examples:
5.0
booking_togh animal
principal axis
human
country
house
word
category
rating
plants
cooking tool
30.86
4.97
car
zebra
animal
24.74
4.86
Human Rating
4.0
measurement
are
22.06
4.89
weather
furniture
house
20.83
4.89
people
mythical beasts
defense
country
4.65
4.19
math
chemistry
sciences
18.67
3.64
wood
plants
16.22
4.85
3.0
sciences
cartoon
11.84
big
humid
weather
11,58
3.48
angel
mythical beasts
emotions
lover
people
6.01
3.68
2.0
r 0.8749
thousand
3.65
huge
big
0.17
3.54
cheerful
emotions
-2554
2.34
0
10 Principal Axis
20
Figure The first principal component in the grounded semantic space captures the concrete-abstract axis of semantics. Left: Each dot represents a word category with the color indicative of the averaged human-rated concreteness (the y axis) and the size proportional to the standard deviation. The axis indicates the value projected onto the first principal axis. Right: Example words in labeled categories.
Interestingly, the first principal dimension is readily interpretable as an abstract-to-concrete axis (Fig. 3). For example, words with the highest values in this axis are ostrich, seagull, albatross, blender, pelican, broccoli, parakeet, lettuce, sailboat, vegetables, whereas words with the lowest values are displeasure, liking. to, outgoing, present, experienced, profitable. faithful, meaningful, multitude. The representations words along this axis is significantly correlated with human rating of their concreteness (ranging from 1 5) from prior study [63] (Fig. 3). The Pearson correlation coefficient reaches 0.8749 or 0.6615 across word categories or individual words, respectively; after grounding with object relations: 0.8001 for categories, r 0.6948 for words. In contrast, the principal axis of the ungrounded semantic space learned from the baseline Bert model is not straightforward to interpret and shows weak correlation with human ratings of concreteness (Table. 1). Other principal components are also intuitively interpretable. For example, PC 2 captures the human vs. non-human axis, PC captures the scene vs. object axis, PC 4 captures the natural vs. artificial axis, PC captures the indoor vs. outdoor axis, PC 6 highlights words related to food. See results about other principal components in Appendix B.1.
4.2 Relation to human-defined norms of semantic features
We further ask whether the visually grounded word embeddings are amenable to binary semantic features defined by humans [25, 64). We use the concept property norm dataset from the Centre for Speech, Language and the Brain (CSLB) [65]. The dataset includes binary semantic features (e.g., www.haasswwwaa_eee wheels) labeled for 638 concepts collected from 123 human participants. We keep 390 features that each contains at least samples. We hypothesize that the grounded word embeddings can be readout with linear and sparse projection readily support binary classification attainable by humans. To test this hypothesis, we train logistic regression model with LI regularization to predict each binary semantic feature from the grounded word embeddings and also repeat this for ungrounded semantics for comparison. See Appendix B.2 for details about this dataset and our evaluation method, Results suggest that the grounded word embeddings are significantly more predictive of visually relevant binary features than ungrounded counterparts obtained by Bert (Wilcoxon Signed Rank Test; p < 0.0001) (Fig. 4). This difference is less pronounced but still significant for other features related to other perceptual (e.g., has f_sv_flaorss. functional (e.g., does does_cut), encyclopaedic (e.g.. is i dangerous), and taxonomic features (e.g., is_ js_clothing). especially after visual grounding of object relations.

---

1.0
0.8
FI Score
0.6
Bert
Grounded
0.4
Relational
Grounded
visual
other perceptual
perceptual
functional
encyclopaedic
taxonomic
Figure 4: The F1 score of predicting semantic feature norms from word representations before and after visual grounding. Each box shows the lower (25%) percentile, the higher (75%) percentile, and the median of scores within a feature type. Whisker= 1.5. Significant level: n.s.: not significant; * 0.05; **, p 0.01; *** p 0.001; **** p 0.0001.
4.3 Clustering word representations
After visual grounding, the semantic representations tend to group themselves based perceptual similarity. We use the SemCat dataset (9, 197 English words from N = 100 categories) [62] and calculate the Silhouette coefficient (between -1 and I) to measure the degree to which these words are clustered by categories. The distance between word embeddings measured as the cosine distance (See details in Appendix B.3). The Silhouette coefficients across 100 categories higher for the visually grounded semantics than ungrounded ones (Wilcoxon Signed Rank Test; p 0.0001) (Fig. 5 left). The greatest gain in clustering are noticeable for categories that include concrete concepts (e.g. car, housing, mammal) with defining visual attributes (Fig. 5 right). For some abstract categories related to human emotion (e.g., happy), the grounded representations also better clustered than the ungrounded ones.
significantly
Word Categorization Performance
Most Improved Categories
0.6
hoosing
housing
hulld ngt
Conficient
0.4
body
vogetablez
lenders
0.3
metalo
Sumiture
0.2
heppieese
vegetables
0.1
postirs_words
coge
car
0.0
Relational
10
Silbonettc Coelficient
Grounded
Grunded
Relitional Gimenmied
Figure 5: Left: A boxplot showing Silhouette coefficients on word representations before and after visual grounding. Each box shows the lower (25%) percentile, the higher (75%) percentile, and the median of the Silhouette coefficients. Whisker= 1.5. Right: Top-15 word categories that are better clustered after visual grounding natural language and object relations.
4.4 Visually informed compositional reasoning
A drawback of distributional semantics is the inability to make visually informed compositional reasoning. We know that "zebra horse with black and white stripes", because have seen how zebra looks like, whereas ungrounded language model is never rarely exposed to such

---

Representation Similarity of Visually Inferred Concepts
Striped Horse: Zebra
Cosine
Word
Rank
Rank
Word
Cosine
0.61
tomcat
1
horse
0.80
0.59
seahorse
2
2
mule
0.74
0.58
squirrel
3
0.73
0.57
antelope
4
stallion
0.72
0.57
bobcat
5
5
seahorse
0.70
0.56
wildcat
donkey
0.70
Similarity Ranking
0.55
swordfish
7
7
camel
0.69
0.55
orangutan
8
8
antelope
0.66
rhinoceros
9
9
unicom
0.63
0.53
lion
10
10
leopard
0.63
0.53
monkey
11
chestnut
0.61
0.53
giraffe
12
12
zebra
0.60
0.49
leopard
30
20
rhinoceros
0.57
0.47
donkey
38
wildcat
0.44
mule
60
40
lion
0.51
0.41
horse
84
orangutan
0.51
0.40
came!
95
bobcat
0.51
0.36
mare
141
56
tomcat
0.47
0.34
squirrel
0.47
0.32
stallion
245
62
monkey
0.46
0.25
chestnut
682
179
giraffe
0.36
0.12
zebra
2914
swordfish
0.25
Before Visual Grounding
After Visual Grounding
Figure 6: Compositional reasoning (striped horse). The left part shows the cosine similarity and its ranking between each of the listed words and the query phrase (striped horse) before visual grounding. The right part shows the corresponding results after visual grounding of natural language. Orange lines indicate words with increased ranking after visual grounding and blue lines for the decreased cases. We highlight in red the target word "zebra" for this specific example, which shows significant increase in cosine similarity (from 0. 0212 to 0.60) and ranking (from 2914 to 12 out of 6238 unique words). Besides, the top words similar to "striped horse" all horse-like animals after visual grounding, but this is not the case for the ungrounded Bert model.
information [10]. We test whether the visually grounded semantics can perform compositional reasoning based on visual knowledge, without being explicit trained to do so. We choose some words (Table 2), for which the meaning can be intuitively inferred from the combination of other words.
Table 2: Examples of visually informed conceptual composition. Each shows the cosine similarity and its ranking in the vocabulary (unique words in the Semcat dataset) between the query phrase and the target word. Except (hot weather, summer), all others concepts supported by composition of visual knowledge the query phrase. For each case, the highest similarity are rank are in bold.
Similarity (cosine rank)
Query Phrase
Target Word
Bert
Grounded
Relational
striped horse
zebra
0.12
2914
0.60
12
0.63
8
black and white bear
panda
0.13
2478
0.69
2
0.81
2
flying
plane
0.36
167
0.66
4
0.61
11
round container
bowl
0.25
489
0.56
0.67
2
fruit
strawberry
0.39
239
0.75
0.85
young dog
puppy
0.40
94
0.92
2
0.93
2
iced mountain
glacier
0.44
20
0.86
1
0.73
5
clear sky
sunny
0.27
631
0.31
184
0.34
61
hot weather
summer
0.27
903
0.52
0.53
For example, we use phrase "striped horse" as compositional query to search for the matched words ranked in terms of cosine similarity. In Fig. 6, the left part shows the cosine similarity and

---

ranking between each of the listed words and the query phrase "striped horse" before visual grounding. The right part shows the corresponding results after visual grounding of natural language. With the grounded semantic representation, the phrase striped horse is highly similar to the word zebra (cosine similarity: 0.60), which is ranked as the 12-th the vocabulary. After further grounding the language model with visual object relations, the target word zebra has an even higher cosine similarity 0.63 ranked the 8-th in the vocabulary (Table 2). Other top-ranked words all refer to horse-like animals (i.e., horse, mule, mare, stallion, donkey, camel, antelope). This is in sharp contrast to the ungrounded semantic space, in which is impossible to relate striped horse to zebra based on the similarity of their representations (cosine similarity: 0.12; rank: 2,914). The ungrounded Bert model highlights the top-3 similar words as tomeat, seahorse, squirrel, which are animals sharing fewer visual features with horse-like animals. See other examples in (Table 2 and Appendix B.4).
4.5 Multimodal image search
Multimodal search query
Quaacil (1-a) ax) Q1 a Qw
image query
word query
horse
Q1
Q w
toward visual info
toward language info
vinsal info wirght ratin
bagguge weigh
sleep
milk
lego
Figure 7: Left: Illustration of multimodal image search with a "zebra" example. The image query Qi is the L2-normalized vector representation of zebra's 's skin pattern. The word query Qw is the 12-nommalized vector representation of word horse. As the weight ratio a in multimodal query Qseachh increases from ( to 1, the search results show progressive changes from stripped patterns, to a real zebra, and horse image. Right: Example multimodal image search results using other image-word pairs as the combined query. The query images (highlighted with a yellow boundary) are in the top row and the corresponding words (shown
purple background) are in the bottom row. The 1st to the 3rd columns correspond to combined queries, with a "cat" image and sleep word, "coffee" image and milk word, "car" image and a lego word. The results of multimodal image search are shown in the 2nd to 6th rows, corresponding increasing from 0 to 1.
In our model, the cross-attention module forms a joint representational space to combine both visual and textual input. We explore whether this joint space can be used support cross-modal tasks, e.g., image search based on image, text, or their combinations [13]. For this task, we add two additional heads (Fv and Fi). Each head includes two linear layers with ReLU between followed by average pooling (See details in Appendix B.5). It is applied to either visual or textual representations in the joint space and results a single vector representation for an image or text (Eq. 6, d = 768). While freezing our model described in Section 3.2, we train the two additional heads with contrastive loss to match the average-pooled representations paired images and texts in terms of their cosine similarity using the MS COCO dataset. To use the model for image search, we apply weighted sum to

---

the normalized representations of query image and a query text (with weights: 1 - and a, where S I 1). We use this multimodal query (Eq. to search a held-out database" for the matched images ranked in terms of cosine similarity.
Q = (Keyv) ER", Qw = Fu.ueryy R..
(6)
Qseach = (1-a)Qq +aQw.
As a controls the weighting between the textual and visual queries, we test how the image search returns different results as a increases from 0 (image only) to 1 (text only). For example, when combine a word (horse) and an image (a stripped pattern) into a query, the search finds images similar to the zebra's skin pattern when is close 0, or finds images of typical horses when is close to 1, but not necessarily a zebra for either case until when a is somewhere close to 0.5 (Fig. 7, left). This observation is generalizable to other examples. See similarly graded changes in (Fig. 7, right).
Discussion
In summary, we apply visual grounding not only words but also relations between words through cross-modal contrastive learning. The results suggest that grounding language learning in vision renders semantic representations more interpretable by human intuition. The grounded semantic space has its principal dimension encode the concrete-to-abstrad variation consistent with human ratings and neurobiological knowledge. The grounded semantic representations are better clustered by finer categories and capable of compositional reasoning (e.g., zebra = striped horse). In addition, our work also shows compelling evidence that both text and image-informed semantics represented in common, continuous, and grounded semantic space. Although this notion has been hypothesized in neuroscience and linguistics, has been rarely implemented and demonstrated with computational models. Uniquely, we demonstrate that continuously varying combination of text and an image into a multimodal query can be used to search images, showing results that make intuitive sense.
Several limitations of our work are noteworthy. The datasets used to train our model are orders of magnitude smaller than those used in recent studies [13, 14]. Scaling up the model training with increasingly larger datasets is expected to greatly improve the model's performance for cross-modal tasks, while generally preserving the interpretability of the grounded language representations as described herein. Some of our experiments and results are preliminary and primarily for illustrative purposes and await more comprehensive and quantitative evaluation in future studies, especially with more downstream vision- languae tasks. Whereas our evaluation focuses on the language model, grounding language to vision may also have refined the visual stream, awaiting further evaluation against visual tasks, as demonstrated in [13, 14].
The visually grounded language model may be usable as a computational model for studying the grounded cognition- theory in cognitive science [15, 16). Ungrounded linguistic models are explanatory about semantic processing in the brain's 's language network [66-70]. Combining the grounded language model with human behavioral and neural data may elucidate how the language network interacts with distributed sensory and motor areas for semantic processing [71].
natural to extend this study by incorporating other sensory input [72, 73] and further ground language learning in action [74-77] and emotion [78]. This study also leaves an open question as to whether grounding should occur an early or late stage of natural language processing. which awaits further exploration and evaluation. For comprehensive modeling of language grounding, it is desirable to expose an agent to naturalistic and multi-sensory environment and engage interactive actions to allow the agent to learn knowledge in the physical world, like how humans learn language.
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF IIS 21 12773.
241, 600 images from the validation dataset of Open Images Dataset V6.
10

---

References
[1] Ridley Stroop. Studies of interference in serial verbal reactions. Journal of experimental psychology. 18 (6):643, 1935.
[2] Friedemann Pulvermüiller. How neurons make meaning: brain mechanisms for embodied and abstract- symbolic semantics. Trends in cognitive sciences, 17(9)4558 470, 2013.
[3] Fritz Günther, Tri Nguyen, Lu Chen, Carolin Dudschig, Barbara Kaup, and Arthur M Glenberg. Immediate sensorimotor grounding of novel concepts learned from language alone. Journal of Memory and Language, 115:104172, 2020.
[4] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26: 3111-31 2013.
[5] Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems,
2014.
[6] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 4171 1186, 2019.
[7] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot leamners. 33: 187-1991, 2020.
[8] Guy Emerson. What are the goals of distributional semantics? Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1436-7453, 2020.
John R Searle et al. Minds, brains, and programs. The Turing Test: Verbal Behaviour as the Hallmark of Intelligence, pages 201-224, 1980.
[10] Stevan Harnad. The symbol grounding problem. Physica Nonlinear Phenomena, 42(1-3))335-344 1990.
[11] Tsung-Yo Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[12] Drew A Hudson and Christopher D Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700667007 0009,, 2019.
[13] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le. Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. arXiv preprint 20Xiv:2102.088
2021.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint Mxxiv:2113.0000: 2021.
[15] Alex Martin, properties and categorics are represented in the human brain. Psychonomic bulletin & review, 23(4): 979-990 2016.
representations action, perception, and emotion systems: How object
[16] Lawrence W Barsalou. Grounded cognition. Annu. Rev. Psychol., 59:617-645, 2008.
[17] Zellig Harris. Distributional structure. Word, 10(2-3):146 162, 1954.
[18] Jeffrey Pennington, Richard Socher. and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[19] Yonatan Bisk. Ari Holtzman. Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 8718 8735, 2020.
11

---

[20] Arthur M Glenberg and David A Robertson. Symbol grounding and meaning: A comparison of high- dimensional and embodied theories of meaning. Journal of memory and language, 43(3):379-401, 2000.
[21] Carina Silberer and Mirella Lapata. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423-1433, 2012.
[22] Stephen Roller and Sabine Schulte Im Walde. A multimodal Ida model integrating textual, cognitive and visual modalities. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1146-1157, 2013.
[23] Elia Bruni, Nam- Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of artificial intelligence research, 19,1-77,200 2014.
[24] Felix Hill, Roi Reichart. and Anna Korhonen. Multi-modal models for concrete and abstract concept meaning. Transactions of the Association for Computational Linguistics, 2:285-296, 2014.
[25] Eloi Zablocki, Benjamin Piwowarski, Laure Soulier, and Patrick Gallinari. Learning multi-modal word representation grounded in visual context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[26] Angeliki Lazaridou, Elia Bruni, and Marco Baroni. this a wampimuk? cross-modal mapping between distributional semantics and the visual world. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1403-1414, 2014.
[27] Angeliki Lazaridou, Marco Baroni, et al. Combining language and vision with multimodal skip-gram model. In Proceedings of the 2015 Conference the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 153-163, 2015.
[28] Grzegorz Chrupata, Akos Kádár, and Afra Alishahi, Learning language through pictures. arXiv preprint ur0iv 2015.
[29] Jamie Kiros, William Chan, and Geoffrey Hinton. Illustrative language understanding: Large-scale visual grounding with image search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)) pages 922-933, 2018
[30] Melissa Ailem, Bowen Zhang, Aurelien Bellet, Pascal Denis, and Fei Sha. probabilistic model for joint learning of word embeddings from texts and images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1478-1487, 2018.
[31] Tanmay Gupta, Alexander Schwing, and Derek Hoiem. Vico: Word embeddings from visual co- oc-occurrenoss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[32] Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. Incorporating visual semantics into sentence representations within a grounded space. arXiv preprint arXiv:2002.02773 022344 2020.
[33] Marco Baroni. Grounding distributional semantics in the visual world. Language and Linguistics Compass, 10(1):3-13 13, 2016.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998 00088866088222200 2017.
[35] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pages 5103-5114, 2019.
[36] Jiasen La, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lanm and--aguaage tasks. arXiv preprint M.XXv:1908.02200 2019.
[37] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. In International Conference on Learning Representations, 2019.
[38] Yen Ch-C Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. 2019.
12

---

[39] Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random- dot stereograms. Nature, 355(6356) 1992.
[40] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances neural information processing systems, 6: 737-744, 1993.
[41] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffryy Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[42] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass, Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European conference on compuder vision (ECCV), pages 649-665, 2018.
[43] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pages 776-794. Springer, 2020.
[44] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
[45] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
[46] Cewu Lu, Ranjay Krishna, Michael Bemstein, and Li Fei-Fei. Visual relationship detection with language priors. European conference on computer vision, pages 852-869. Springer, 2016.
[47] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in Neural Information Processing Systems, 30:4967-4977, 2017.
[48] Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinay Gupta. Iterative visual reasoning beyond convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7239-7248, 2018.
[49] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10313- 10022,, 2019.
[50] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image- text matching. In Proceedings of the IEEEJCVF International Conference on Computer Vision, pages 4654-4662, 2019.
[51] Huda Hakami, Danushka Bollegala, and Hayashi Kohei. Why pairdiff works?-a mathematical analysis of bilinear relational compositional operators for analogy detection. arXiv preprint arXiv: :tfi::/700066677 2017.
[52] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-7511 2013.
[53] Antoine Bordes, Nicolas Usunicr, Alberto Garcia-Durai Jason Weston, and Oksana Yakhnenko. Translat- ing embeddings for modeling multi-rclational data. In Neural Information Processing Systems (NIPS), pages 1-9, 2013.
[54] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[55] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint 11165:14 2014.
[56] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. Holographic embeddings of knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. 2016.
[57] Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing text models for common ground with visual representations. arXiv preprint arXiv:2005.0060 00619, 2020.
13

---

[58] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409.1553 2014.
[59] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[60] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1.Xi::88770037455 2018.
[61] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-222. 2015.
[62] Lütfi Kerem Senel, Ihsan Utlu, Veysel Yücesoy, Aykut Koc, and Tolga Cukur. Semantic structure and interpretability of word emheddings. IEEF/ACM Transactions on Audio, Speech, and Language Processing, (6(10):1769-177 1779, 2018
[63] Mare Bryshaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911, 2014.
[64] Lucy Li and Jon Gauthier. Are distributional representations ready for the real world?? evaluating word vectors for grounded perceptual meaning. In Proceedings of the First Workshop on Language Grounding for Robotics, pages 76-85, 2017.
[65] Barry J Devereux, Lorraine K Tyler, Jeroen Geertzen, and Billi Randall. The centre for speech, language and the brain (csIb) concept property norms. Behavior research methods, 46(4): 19-1 127, 2014.
[66] Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang. Vicente L Malave, Robert A Mason, and Mareel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 1995, 2008.
[67] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):443- 458, 2016.
[68] Luana Bulat, Stephen Clark, and Ekaterina Shutova. Speaking, seeing, understanding: Correlating semantic models with conceptual representation in the brain. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1091,1091, 2017.
[69] Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):1133, 2018.
[70] Yizhen Zhang, Kuan Han, Robert Worth, and Zhongming Liu. Connecting concepts in the brain by mapping cortical representations of semantic relations. Nature communications, 11(1):1 11, 2020.
[71] Stefan Heinrich, Yuan Yao, Tobias Hinz, Zhiyuan Liu, Thomas Hummel, Matthias Kerzel, Comellus Weber, and Stefan Wermter, Crossmodal language grounding in an embodied neurocognitive model. Frontiers in Neurorobotics, 14, 2020.
[72] Douwe Kiela, Luana Bulat, and Stephen Clark. Grounding semanties olfactory perception. Proceed- ings of the 53yd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (V (Volume 2: Short Papers), pages 231 231-236,77 2015.
[73] Douwe Kiela and Stephen Clark. Multi-and cross-modal semantics beyond vision: Grounding in auditory perception. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 2461 12401222022222 2015.
[74] Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and generalization in 2d world. arXiv preprini arXiv: 802.01433. 2018
[75] Joyce Y Chai, Qiaozi Gao, Lanbo She. Shaohua Yang, Sari Saba-Sadiya, and Guangyue Xu. Language to action: Towards interactive task learning with physical agents. In IJCAI, pages 2-9, 2018.
[76] Rohan Paul, Jacob Arkin, Nicholas Roy, and Thomas M Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. 2016.
[77] Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
[78] Armand S Rotaru and Gabriella Vigliocco. Constructing semantic models from words, images, and emojis. Cognitive science, 44(4)=12230, 2020.
14


Reviews:
[{"review_id": "l1e9NOFPGSL", "reviewer": "Reviewer_1xyN", "summary": "This paper analyzes the linguistic embedding in the joint image-language space learned by cross-modal contrastive learning. Comparing with the standard word embedding, the grounded language embedding is human explainable. Specifically, the PCA of the grounded langauge embedding is aligned with human rating. When clustering the language embedding, the cluster of grounded language embedding is more semantically coherent than the un-grounded language embedding. Furthermore, the paper showed that the grounded langauge embedding could benefit visual-linguistic downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\n\n1. Method: The main focus of this paper is to show that the grounded linguistic embedding is more aligned with human perception. Therefore, in terms of the method, the paper uses a pretty standard two stream vision and language model to learn the embedding. The description of the approach is clean and easy to follow.\n\n2. Experiment:the analysis conducted in this paper is pretty interesting. However, this paper is not the first one to conduct similar experiments. \n\n  2.a In terms of 'Clustering of word representations', ViCo (Gupta et, al., 2019) conducted similar experiments (Sect. 4.1) and showed the grounded embedding is better than the un-grounded. \n\n  2.b In terms of the Multimodal image search, Ailem et. al, 2018 conducted similar experiments. They showed that the grounded embedding can help the image retrieval model to achieve better performance than the un-grounded.\n  \nTherefore, the experiments and analysis are very interesting. However, they are not surprising given the previous research.\n\n---\nExperiment:\n\n1. Word coverage rate: The experiments in Sect. 4.1 and 4.4 are very interesting. However, I wonder whether all the tokens in those two datasets are covered by MS-COCO? Specifically, MS-COCO is annotated with image and text. The text is descriptive sentence which describe the object/event in the image. In Sect. 4.1 the dataset contains word like 'emotions' and 'mythical_beasts' (Fig. 3). I wonder whether those words are covered by the MS-COCO or Visual Genome?\n\n2. In Sect. 4.1, does the ungrounded word embedding show similar behavior? Specifically, could the PCA on the ungrounded word embedding provide any useful interpretation?\n\n3. Sect. 4.5. I am not quite sure how did the image-text retrieval conduct? I could understand everything till L236. But I don't know hwy a query image and a query text need to be combined. How to use the combined representation to conduct image retrieval.\n\n4. Sect. 4.5. As the MS-COCO is a dataset defined for image retrieval, I wonder why the standard image retrieval task and metrics (Recall) is not used/reported?\n\n---\nThe rating is based on the novelty and the questions in experiment. I would happy to raise the score if the concerns are resolved.\n\nGupta T. et. al., ViCo: Word Embeddings from Visual Co-occurrences, ICCV 2019.\nAilem, M. et. al., A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images. EMNLP 2018. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "_s1-Y6EXV51", "reviewer": "Reviewer_Zh5u", "summary": "This paper proposes a contrastive learning framework to learn visually grounded, and as a result, semantically interpretable language representations. It proposes two contrastive losses which make use of multiheaded cross-modal attention and bilinear interaction to compute similarity scores between subject and object representations, which can then we used in the contrastive loss.\n\nThey perform thorough experiments with the resulting representation to illustrate how well they correlate to human ratings along different language axes such as concrete-abstract and compositionality, as well as qualitative multimodal image search examples.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a meaningful contribution that naturally follows recent image-text representations learned with a contrastive loss (e.g. CLIP/ALIGN). The writing and structure of the paper is strong and clear, it was easy to read. The authors also clearly motivate the work, as having more semantically meaningful language representations is something very desirable, especially from an ethical and interpretability perspective. \n\nThe overall approach and results are well illustrated, I think they could be reproduced fairly easily (although time consuming if associated code is not released). There are a comprehensive and interesting set of experiments on the language representations, which include analyzing representations with PCA and their relationship to SemCat words, use for predicting binary semantic features, how well they cluster according to SemCat categories, how they capture compositional relationships, and how search results change along an image to text feature continuum.\n\nThere are some remaining questions I hope can be answered:\n1. In Section 4.1, how many principal components are there?\n2. L173-175, Bert is discussed as comparison. What about vision-language transformers or other vision-language language models? It seems like the comparison is a bit incomplete/unfair given that there are vl language representations out there which may have better captured the semantic space than Bert (e.g., ViLBERT, LXMERT, VisualBERT, Oscar, MT-GrOVLE etc)\n3. The illustration of the significance levels in Figure 4 are pretty hard to follow - why are there overlapping brackets with different *? Shouldn't each box plot be associated with only one of the significance values?\n4. Typo at L225, I believe \"shape\" should be \"sharp\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not fully addressed limitations of their work. The method, motivation, and qualitative results are important alone, but the goal in having visually grounded and interpretable language representations is to then also use them in multimodal tasks (and replace current representations which are not grounded nor interpretable). The hope is that the representations proposed in this work perform as well as current state of the art or better, but this isn't demonstrated with any downstream experiments. E.g., the multimodal image search section provides a very cool illustration (Figure 7), but then no quantitative experiments on retrieval are provided. Of course the main contribution of this paper concerns interpretability, but connecting the work to real world applications would be an important point to include in the discussion.\n\nAs far as societal impact, this paper only provides positive impact, which is great to see!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "DtS2uVFE0Pl", "reviewer": "Reviewer_zUxZ", "summary": "The paper implements a model which learns word representations from\ntext grounded in images. The model is composed of a BERT-based text\nstream, a VGG-based image stream, and visual relation classification\nmodule. The text and image components are pre-trained and fine tuned\non the COCO dataset of captioned images as well as on Visual Genome\nrelations. The main contribution of the paper is the large number of\nanalyses carried out on the induced grounded word embeddings, compared\nto plain BERT embedddings. The paper shows some evidence that the\nprincipal components of the grounded embedding space correspond to\ninterpretable semantic dimensions (i.e. concrete-abstract), that\nwords' semantic features can be predicted based on embeddings. The\npaper also claims that the model is capable of compositional\nunderstanding.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThe main idea of visual grounding  has been of interest in\ncomputational linguistics for a long time, and has been investigated\ncomputationally many times. This paper main contributions are the use\nof state-of-the-art neural architectures and pretrained models, and an\nin-depth study of the nature of the learned embeddings. The results\nrange from compelling (the interpretable principal components) to\nsuggestive but preliminary (compositionality), but overall show\nconvincingly that in this particular incarnation of grounding does\nmake a major contribution to make the word representations more\nhuman-like.\n\n## Weaknesses\n\nThe discussion of and comparison to earlier related work is very\nincomplete. Given the analytical focus of the paper quantitative\ncomparisons to alternative models are not absolutely necessary but it\nwould be good to at least see a more in-depth related work\nsection. There are a lot of papers that could be mentioned here; \na small sample follows:\n\n\n- Bordes, P., Zablocki, E., Soulier, L., & Piwowarski, B. (2019,\n  November). Incorporating Visual Semantics into Sentence\n  Representations within a Grounded Space. In Proceedings of the 2019\n  Conference on Empirical Methods in Natural Language Processing and\n  the 9th International Joint Conference on Natural Language\n  Processing (EMNLP-IJCNLP) (pp. 696-707).\n\n- Kiros, J., Chan, W., & Hinton, G. (2018, July). Illustrative\n  language understanding: Large-scale visual grounding with image\n  search. In Proceedings of the 56th Annual Meeting of the Association\n  for Computational Linguistics (Volume 1: Long Papers) (pp. 922-933).\n\n- Chrupała, G., Kádár, Á., & Alishahi, A. (2015, July). Learning\n  language through pictures. In Proceedings of the 53rd Annual Meeting\n  of the Association for Computational Linguistics and the 7th\n  International Joint Conference on Natural Language Processing\n  (Volume 2: Short Papers) (pp. 112-118).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not explicitly discussed. The preliminary nature and\nsmall scale of some of the experiments (e.g. section 4.4) should be\nacknowledged. The word \"explainable\" suggests that the architecture is\ndesigned to be explainable, but in fact it doesn't seem to be the case: rather, \nanalyses are carried out which provide some insight into the learned \nrepresentations.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AUiSIezIrX-", "reviewer": "Reviewer_BB9g", "summary": "The paper aims to ground language to vision via cross-modal contrastive learning to build an explainable shared semantic space. This is achieved in two parts (i) align visual and language representations with MS COCO dataset (ii) Retrieve visual objects with language queries through a cross-modal attention module and infer the visual relations between the retrieved objects through a bilinear operator on Visual Genome dataset. This allows model's language encoder to become a stand-alone language model that can encoder embedding concepts in a visually grounded semantic space. The model enables compositional language understanding based on visual knowledge and multimodal image search with queries. The authors show experiments on clustering, compositional reasoning, relationship with human-defined norms, and image search to showcase the usefulness of the model.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper targets a less-studied problem of grounding language in visual semantic space and uses a two-step process to achieve that. Specifically, for the first step the paper uses a CLIP-style [1] contrastive loss to align visual and language representations using MS COCO dataset. For the second step, which seems useful in helping ground the object relations to generate visually ground object representation for the text, the model uses a cross-attention module similar to VILBERT [2] model, by passing language stream as a query and visual output as a key and value to generate a language conditioned visual representation which is basically visually grounded object representation for the object present in the text. A bilinear relation module is applied to the ground object representation for predicting the relationship between the two objects which is trained using a contrastive learning approach. The idea of aligning using MS COCO dataset uses CLIP-style contrastive loss and is not new while the idea of improving the object semantic grounding using Visual Genome dataset and contrastive loss over relations is new and insightful. Comments on the paper are as follows:\n\n1) The paper doesn't provide a concrete way of evaluating the usefulness of these embeddings on the downstream vision-and-language tasks in a quantitative way. I was expecting to see some experiments similar to CLIP paper on how exactly the better grounding can help achieve a better general model for downstream tasks. The paper does cover a qualitative study on multimodal image search but didn't do extensive experiments on image or text retrieval which this model would have empowered.The supplementary material provides some results on retrieval on MS-COCO dataset but the number seem to very low compared to state-of-the-art model which seems surprising (the appendix only have qualitative figure for comparison and no table). This leaves the reader with incomplete knowledge on the actual practical usefulness of the model.\n2) The section 4.3 is complicated and complex to understand in the way it is written. I would suggest that authors to either improving the writing or remove that section for the sake of the clarity of the paper. It took me some time to understand how figure 6 was working and what it was supposed to show after reading section 4.3.\n3) The paper does evaluate self-attention on top of VGG16 feature extractor they use for imagenet classification accuracy (in appendix) but never evaluate their final visual stream on imagenet to check how grounding helps.\n4) The paper also evaluates on GLUE tasks in appendix but it looks like the performance is decreasing on most tasks which is counter-intuitive as learning about world through images should maintain or improve original BERT model. I would suggest that authors take a look at [3] which aims to learn multimodal and unimodal tasks using a single transformer architecture.\n5) The paper doesn't provide any ablations on not using \"visual grounding of object relations\" stage or only using that stage. These ablations are necessary and insightful to get a sense to how much visual grounding of object relations is helping the model in comparison to just using CLIP-style loss. Ideally, if it helps, it could also help improve the already strong CLIP model as well. Furthermore, the ablation study on only using \"visual grounding of object relations\" could help understand how much CLIP-style loss helps in overall grounding of the language to vision and vice versa.\n6) Similarly, the ablations of Loss_l and Loss_v alone are missing to understand impact of two modalities on contrastive loss. The paper also doesn't provide ablations on how much data helps in how much performance. Ideally, a study could be conducted on using percentage of coco dataset to showcase if more data can help in the grounding process. The paper also doesn't mention any hyperparameter tuning or otherwise.\n7) There is a lot going on in the appendix and some of the things there can make it into the paper (such as ablations on visual grounding loss's components relations and object prediction loss) while some the things like I mentioned can be moved to supplementary.\n\nThe paper is a good attempt in a less-explored direction but fails to provide reader with useful insights on actual practical help of doing the visual grounding as GLUE results drop, multimodal retrieval results are not impressive among others. Though qualitative results make sense around clustering and probing, the model at large doesn't look useful from the provided information in the manuscript for the downstream tasks.\n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" arXiv preprint arXiv:2103.00020 (2021).\n\n[2] Lu, Jiasen, et al. \"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.\" arXiv preprint arXiv:1908.02265 (2019).\n\n[3] Hu, Ronghang, and Amanpreet Singh. \"UniT: Multimodal Multitask Learning with a Unified Transformer.\" arXiv preprint arXiv:2102.10772 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper doesn't have a broader impact statement included.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates cross-modal contrastive learning for semantic representations and finds that grounded language embeddings are more semantically coherent than un-grounded ones. The reviewers generally like this direction and the paper is well-written. The authors should really heed the feedback from the reviewers, however, and in particular: the authors should make *very* clear that this is not a methods paper, but an analysis paper. Occasionally the paper reads too much like it is claiming a new method, which is problematic because a) the comparison to other models is lacking; and b) the paper is not adequately positioned in the existing literature. There are many papers that have explored visually-grounded semantic embeddings in the past, several of them already quite old, and this work should be positioned accordingly.
Decision: Accept (Poster)