Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning
Year: 2021

Abstract:
In natural language processing, most models try to learn semantic representations merely from texts. The learned representations encode the “distributional semantics” but fail to connect to any knowledge about the physical world. In contrast, humans learn language by grounding concepts in perception and action and the brain encodes “grounded semantics” for cognition. Inspired by this notion and recent work in vision-language learning, we design a two-stream model for grounding language learning in vision. The model includes a VGG-based visual stream and a Bert-based language stream. The two streams merge into a joint representational space. Through cross-modal contrastive learning, the model first learns to align visual and language representations with the MS COCO dataset. The model further learns to retrieve visual objects with language queries through a cross-modal attention module and to infer the visual relations between the retrieved objects through a bilinear operator with the Visual Genome dataset. After training, the model’s language stream is a stand-alone language model capable of embedding concepts in a visually grounded semantic space. This semantic space manifests principal dimensions explainable with human intuition and neurobiological knowledge. Word embeddings in this semantic space are predictive of human-defined norms of semantic features and are segregated into perceptually distinctive clusters. Furthermore, the visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based on images, texts, or their combinations.

heading: 1 Introduction
text: Humans take much longer time to name a colored word when the color and the word mismatch (e.g., “red” shown in green) than when they match (e.g., “red” shown in red) [1]. This effect is an example of rich psychological evidence suggesting that humans learn language by grounding meanings to knowledge about the world [2, 3]. In contrast, most models in natural language processing (NLP) [4–7] encode “distributional semantics” [8] learned from texts only. Put yourself as machines in a thought experiment for the “Chinese Room Argument” [9]. Imagine that you have to learn Chinese from scratch as your first language. All that you have is a Chinese-to-Chinese dictionary. You might be able to relate a word to other words based on textual distributions. It is, however, impossible to learn word meanings without any additional explanation in reference to the physical world [10].
A language model may learn concepts from texts paired with sensory data, such as images. Joint vision-language learning has been explored for image captioning [11], visual question answering [12], and pre-training vision models with weak supervision [13, 14]. In line with these studies, we
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
train a language model and a vision model jointly to match images and texts. We further analyze the semantic space obtained with the visually grounded language model. In this space, semantic embeddings are found to be organized and clustered by visual attributes, predictive of human-defined norms of semantic features, useful for compositional language understanding and cross-modal image search. We expect this visually grounded language model to also be useful for understanding the computational basis of grounded cognition [15, 16].

heading: 2 Background and Related Work

heading: 2.1 Distributional vs. grounding hypothesis
text: In the distributional hypothesis [17], words that occur in similar contexts carry similar meanings. This hypothesis has motivated influential machine learning models to learn word embeddings from large text corpora [4, 18]. However, the learned word embeddings are not straightforward to interpret [8, 19]. Alternatively, the symbol grounding hypothesis suggests that a word is connected to its meaning by relating to its referent in the physical world [10, 20]. In line with this hypothesis, earlier studies demonstrate that visual features or contexts can enhance language learning [21–25].

heading: 2.2 Vision-language learning
text: Grounding language in vision has been of increasing interest in computational linguistics and machine learning. A common strategy is to fuse words with related visual information in terms of perceptual norms [21], bag-of-visual-word [23, 26], or learnable visual features [27–30]. The models used for vision-language fusion evolve alongside those for NLP, such as Latent Dirichlet Allocation (LDA) [23], log-bilinear model [31], Skip-gram model [25, 27], and recurrent neural network [28]. Generally, visual grounding may refine the distribution and interpretability of language representations [23, 25, 26, 28–32] and facilitate cross-modal tasks [28–33]. More recent work has begun to use transformer [6, 34] for vision-language learning, showing strong performance in cross-modal tasks [35–38] .
Contrastive learning [39, 40] is increasingly applied to not only unimodal data [41] but also multimodal data [13, 42]. It is able to learn better representations than alternative prediction or classification objectives [43]. However, cross-modal contrastive learning is still under-explored for higher-level tasks, e.g., visual question answering [12], visual reasoning [44], scene graph generation [45]. Such tasks involve abstract reasoning about the relations between entities (e.g., visual objects). Prior
work approaches relational inference with multi-layer perceptron [46, 47] or graph neural networks [48–50]. Arguably, a more compelling idea [51] is to model entities as vectors in a continuous space and to model their relations as arithmetic operators (linear [52–54] or bilinear [55, 56]) applied to the vector representations of those entities.

heading: 2.3 Relation to prior work
text: In this work, we first build a two-stream model to jointly learn visual and language representation from image-caption pairs, similar to recent work [13, 14]. We then finetune the learned model by adding a cross-modal attention layer [35, 36] and bilinear operators [55] to represent the relations between visual objects. Both stages utilize cross-modal contrastive loss. Related to our work, Harwath et al. match visual objects to spoken words using triplet loss [42]. Early this year, Jia et al. [13] and Radford and Kim et al. [14] use contrastive learning to pretrain a vision model using a massive image-text dataset and demonstrate largely improved zero-shot transfer learning performance on visual and cross-modal tasks. Different from their perspectives, we focus on assessing the language encoders and word representations. Specifically, we perform a systematic evaluation of the semantic space grounded in vision vs. the ungrounded semantic space learned from texts only. This evaluation is possible since after training, the language and visual streams in our model are fully separable as stand-alone systems, unlike some vision-language models that require both visual and textual input to be usable [35, 36]. Our goal is to assess how visual grounding affects the distribution of textual representations by analyzing the distribution of word embeddings in the grounded semantic space, in line with related works [23, 25, 26, 28–32, 57] .

heading: 3 Approach

heading: 3.1 Visual grounding of natural language
text: To build a computational model for learning visually grounded language representations, we develop a model (Fig. 1) that combines a stand-alone visual stream and a stand-alone language stream. The visual stream is based on VGG16 [58] with an additional linear transformation as an embedder to match the feature dimension of the language stream and an additional multi-head self-attention layer [59] to enforce global information aggregation and learn long-range dependency. The language stream is based on Bert [6]. Using separate linear transformation heads [41], the output from both the visual stream and the language stream are projected to a common representational space. In this common space, the inner-product between the visual representation V at every location and the language representation L of every word gives rise to a 3D match-map, where each element indicates how a word in the text matches each location in the image (See illustration in Fig. 1). The sum of the maximal match is the similarity score S(V ,L) between a pair of image and text. See Eq. 1, where i, j indicate the location in the 2D image feature map V and k indicates the k-th word in L.
M i,j,k = V i,j ·LTk , S(V ,L) = K∑
k=1
max i,j M i,j,k (1)
Extending the unimodal normalized temperature-scaled cross-entropy (NT-Xent) loss [13, 41, 60], we define the cross-modal contrastive loss using the anchor sample from one modality and the positive sample and negative samples from the other modality [13, 14]. As such, we define and sum two loss functions with the anchor sample from either images or texts and positive/negative samples from either texts or images, respectively.
Lossl = − 1
B B∑ i=1 log exp(S(V i,Li)/τ)∑B j=1 exp(S(V i,Lj)/τ) , Lossv = − 1 B B∑ i=1 log exp(S(V i,Li)/τ)∑B j=1 exp(S(V j ,Li)/τ) (2)
For Lossl in Eq. 2, the anchor sample V i is an input image and the positive sample Li is the corresponding image caption, whereas the negative samples Lj are unmatched textual descriptions included in the same batch (B is the batch size). Similarly, Lossv in Eq. 2 is defined to contrast the positive and negative image samples against an anchor textual sample.

heading: 3.2 Visual grounding of object relations
text: We further finetune the model for visual relation prediction, as illustrated in Fig. 2. In this stage, we remove the linear transformation heads in Fig. 1 and add a multi-head cross-modal attention module [35, 36]. The attention module uses a query based on the embedding of an object word from the language stream (QueryL) and uses keys (KeyV ) and values (ValueV ) from every location in the feature map output from the visual stream. The attention score is calculated as the inner-product of QueryL and KeyV followed by softmax. The attention-weighted sum of ValueV is concatenated across 8 attention heads to generate a visually grounded object representation.
Applied to the grounded object representation is a bilinear relation module for predicting the visual relation between two objects (linguistically a subject and an object). For both the subject and the object, their grounded representations are linearly transformed to a subspace D = Rd (default d = 32), denoted as rs and ro, respectively. A predicate p is represented as a learnable bilinear operator Fp : D × D → R, which represents the relation embedding Rp. Applying this bilinear operator to the subject vs. object representations measures their relation score S specific to the given predicate [55] expressed as (Eq. 3).
S(rs, ro;Rp) = Fp(rs, ro) = rsRpr T o (3)
For visual relation prediction, we also use contrastive learning with two loss functions by taking either relation embedding or subject/object representations as positive/negative samples.
Lossrel = − 1 |B| ∑
(rs,ro;Rp)∈B
log exp(S(rs, ro;Rp)/τ)∑ k∈Krel exp ( S(rs, ro;Rkp)/τ ) (4) Lossobj = − 1 |B| ∑
(rs,ro;Rp)∈B
log exp(S(rs, ro;Rp)/τ)∑
k∈Kobj exp(S(r k s , rko ;Rp)/τ)
(5)
In Lossrel, Krel is the set that contains all relations available. The anchor sample is a pair of subject and object in an image. The positive sample is the embedding of the ground truth relation. The negative samples are the embeddings of all other relations. In Lossobj, the anchor sample is a given relation. The positive sample is a subject-object pair that holds this relation. The negative samples are other subject-object pairs in a different relation. For both loss functions, the positive and negative samples are drawn from the same batch B. In addition, we also add a classification head (two fully connected layers with ReLU in between) and apply it to the grounded object representation. We use object classification as an auxiliary objective (with a cross-entropy loss) to constrain the grounded object representation to be separable across objects for classification.

heading: 3.3 Training and Testing
text: We train the model in three stages to progressively refine the model with increasingly demanding tasks. In the first stage, we pretrain the visual and language streams separately as image and text encoders. The language stream is the pretrained Bert1 used as the baseline model for subsequent experiments. The visual stream is pretrained for object classification with ImageNet [61]. Relative to the baseline CNN, the inclusion of self-attention improves the top-1 classification accuracy from 71.6% to 74.3% on the ImageNet validation dataset. The attention module also renders the classification more robust when the input image is partially occluded (See details in Appendix A.1).
In the second stage, we refine the pretrained language and visual streams by matching texts to images, as illustrated in Fig. 1 on the MS COCO dataset [11]. While freezing other layers, we refine the self-attention layer in the visual stream and the top k layers in Bert (by default k = 8). Training with contrastive learning is based on the MS COCO dataset. As five captions are available for each image, we randomly sample one caption per image in each iteration. Earlier grounding (larger k) tends to support better image-text retrieval performance (see details in Appendix A.2).
In the third stage, we further finetune the model for visual relation prediction as illustrated in Fig. 2. We refine the visual self-attention layer and the higher l layers in Bert (by default l = 2) based on the Visual Genome dataset [45] after cleaning the dataset to include 114 relations and 55 object classes in order to alleviate imbalanced data across different classes or relations (Appendix A.3). The training does not use any image annotation (e.g., bounding box) , which otherwise requires other models (e.g., object detection). Instead, we use cross-attention to retrieve visual objects from raw images given textual queries and learn the representations of the retrieved objects and their relations altogether. After training, the model predicts the object class with 97.71% top-1 accuracy and predicts the relation label with 64.26% top-1 accuracy (See details and examples in Appendix A.3).

heading: 4 Experiments

heading: 4.1 Principal components of grounded semantic representations
text: To evaluate the visually grounded semantic space, we use the language stream as a stand-alone model to extract the output representations of commonly used English words in the SemCat dataset (9, 197 words; 100 word categories) [62]. Details about how word representations are extracted from the language stream are explained in Appendix B. We apply the principal component analysis to the representations of all the words studied here and examine the top components as the principal dimensions of the grounded semantic space.
Interestingly, the first principal dimension is readily interpretable as an abstract-to-concrete axis (Fig. 3). For example, words with the highest values in this axis are ostrich, seagull, albatross, blender, pelican, broccoli, parakeet, lettuce, sailboat, vegetables, whereas words with the lowest values are displeasure, liking, to, outgoing, present, experienced, profitable, faithful, meaningful, multitude. The representations of words along this axis is significantly correlated with human rating of their concreteness (ranging from 1 to 5) from prior study [63] (Fig. 3). The Pearson correlation coefficient reaches 0.8749 or 0.6615 across word categories or individual words, respectively; after grounding with object relations: r = 0.8001 for categories, r = 0.6948 for words. In contrast, the principal axis of the ungrounded semantic space learned from the baseline Bert model is not straightforward to interpret and shows a weak correlation with human ratings of concreteness (Table. 1). Other principal components are also intuitively interpretable. For example, PC 2 captures the human vs. non-human axis, PC 3 captures the scene vs. object axis, PC 4 captures the natural vs. artificial axis, PC 5 captures the indoor vs. outdoor axis, PC 6 highlights words related to food. See results about other principal components in Appendix B.1.

heading: 4.2 Relation to human-defined norms of semantic features
text: We further ask whether the visually grounded word embeddings are amenable to binary semantic features defined by humans [25, 64]. We use the concept property norm dataset from the Centre for Speech, Language and the Brain (CSLB) [65]. The dataset includes binary semantic features (e.g., has_wheels) labeled for 638 concepts collected from 123 human participants. We keep 390 features that each contains at least 5 samples. We hypothesize that the grounded word embeddings can be readout with a linear and sparse projection to readily support binary classification attainable by humans. To test this hypothesis, we train a logistic regression model with L1 regularization to predict each binary semantic feature from the grounded word embeddings and also repeat this for ungrounded semantics for comparison. See Appendix B.2 for details about this dataset and our evaluation method. Results suggest that the grounded word embeddings are significantly more predictive of visually relevant binary features than ungrounded counterparts obtained by Bert (Wilcoxon Signed Rank Test; p < 0.0001) (Fig. 4). This difference is less pronounced but still significant for other features related to other perceptual (e.g., has_flavors), functional (e.g., does_cut), encyclopaedic (e.g., is_dangerous), and taxonomic features (e.g., is_clothing), especially after visual grounding of object relations.

heading: 4.3 Clustering of word representations
text: After visual grounding, the semantic representations tend to group themselves based on perceptual similarity. We use the SemCat dataset (9, 197 English words from N = 100 categories) [62] and calculate the Silhouette coefficient (between−1 and 1) to measure the degree to which these words are clustered by categories. The distance between word embeddings is measured as the cosine distance (See details in Appendix B.3). The Silhouette coefficients across 100 categories are significantly higher for the visually grounded semantics than ungrounded ones (Wilcoxon Signed Rank Test; p < 0.0001) (Fig. 5 left). The greatest gain in clustering are noticeable for categories that include concrete concepts (e.g. car, housing, mammal) with defining visual attributes (Fig. 5 right). For some abstract categories related to human emotion (e.g., happy), the grounded representations are also better clustered than the ungrounded ones.

heading: 4.4 Visually informed compositional reasoning
text: A drawback of distributional semantics is the inability to make visually informed compositional reasoning. We know that "zebra is a horse with black and white stripes", because we have seen how zebra looks like, whereas an ungrounded language model is never or rarely exposed to such
information [10]. We test whether the visually grounded semantics can perform compositional reasoning based on visual knowledge, without being explicit trained to do so. We choose some words (Table 2), for which the meaning can be intuitively inferred from the combination of other words.
For example, we use a phrase "striped horse" as a compositional query to search for the matched words ranked in terms of cosine similarity. In Fig. 6, the left part shows the cosine similarity and
ranking between each of the listed words and the query phrase "striped horse" before visual grounding. The right part shows the corresponding results after visual grounding of natural language. With the grounded semantic representation, the phrase striped horse is highly similar to the word zebra (cosine similarity: 0.60), which is ranked as the 12-th in the vocabulary. After further grounding the language model with visual object relations, the target word zebra has an even higher cosine similarity of 0.63 ranked the 8-th in the vocabulary (Table 2). Other top-ranked words all refer to horse-like animals (i.e., horse, mule, mare, stallion, donkey, camel, antelope). This is in sharp contrast to the ungrounded semantic space, in which it is impossible to relate striped horse to zebra based on the similarity of their representations (cosine similarity: 0.12; rank: 2,914). The ungrounded Bert model highlights the top-3 similar words as tomcat, seahorse, squirrel, which are animals sharing fewer visual features with horse-like animals. See other examples in (Table 2 and Appendix B.4).

heading: 4.5 Multimodal image search
text: In our model, the cross-attention module forms a joint representational space to combine both visual and textual input. We explore whether this joint space can be used to support cross-modal tasks, e.g., image search based on image, text, or their combinations [13]. For this task, we add two additional heads (FV and FL). Each head includes two linear layers with ReLU in between followed by average pooling (See details in Appendix B.5). It is applied to either visual or textual representations in the joint space and results in a single vector representation for an image or a text (Eq. 6, d = 768). While freezing our model described in Section 3.2, we train the two additional heads with contrastive loss to match the average-pooled representations of paired images and texts in terms of their cosine similarity using the MS COCO dataset. To use the model for image search, we apply weighted sum to
the normalized representations of a query image and a query text (with weights: 1− α and α, where 0 ≤ α ≤ 1). We use this multimodal query (Eq. 7) to search a held-out database2 for the matched images ranked in terms of cosine similarity.
QI = FV (KeyV ) ∈ R d, QW = FL(QueryL) ∈ R d. (6)
Qsearch = (1− α)QI + αQW . (7)
As α controls the weighting between the textual and visual queries, we test how the image search returns different results as α increases from 0 (image only) to 1 (text only). For example, when we combine a word (horse) and an image (a stripped pattern) into a query, the search finds images similar to the zebra’s skin pattern when α is close to 0, or finds images of typical horses when α is close to 1, but not necessarily a zebra for either case until when α is somewhere close to 0.5 (Fig. 7, left). This observation is generalizable to other examples. See similarly graded changes in (Fig. 7, right).

heading: 5 Discussion
text: In summary, we apply visual grounding to not only words but also relations between words through cross-modal contrastive learning. The results suggest that grounding language learning in vision renders semantic representations more interpretable by human intuition. The grounded semantic space has its principal dimension encode the concrete-to-abstract variation consistent with human ratings and neurobiological knowledge. The grounded semantic representations are better clustered by finer categories and capable of compositional reasoning (e.g., zebra = striped horse). In addition, our work also shows compelling evidence that both text and image-informed semantics are represented in a common, continuous, and grounded semantic space. Although this notion has been hypothesized in neuroscience and linguistics, it has been rarely implemented and demonstrated with computational models. Uniquely, we demonstrate that a continuously varying combination of a text and an image into a multimodal query can be used to search images, showing results that make intuitive sense.
Several limitations of our work are noteworthy. The datasets used to train our model are orders of magnitude smaller than those used in recent studies [13, 14]. Scaling up the model training with increasingly larger datasets is expected to greatly improve the model’s performance for cross-modal tasks, while generally preserving the interpretability of the grounded language representations as described herein. Some of our experiments and results are preliminary and primarily for illustrative purposes and await more comprehensive and quantitative evaluation in future studies, especially with more downstream vision-language tasks. Whereas our evaluation focuses on the language model, grounding language to vision may also have refined the visual stream, awaiting further evaluation against visual tasks, as demonstrated in [13, 14].
The visually grounded language model may be usable as a computational model for studying the grounded cognition - a theory in cognitive science [15, 16]. Ungrounded linguistic models are explanatory about semantic processing in the brain’s language network [66–70]. Combining the grounded language model with human behavioral and neural data may elucidate how the language network interacts with distributed sensory and motor areas for semantic processing [71].
It is natural to extend this study by incorporating other sensory input [72, 73] and further ground language learning in action [74–77] and emotion [78]. This study also leaves an open question as to whether grounding should occur at an early or late stage of natural language processing, which awaits further exploration and evaluation. For comprehensive modeling of language grounding, it is desirable to expose an agent to a naturalistic and multi-sensory environment and to engage interactive actions to allow the agent to learn knowledge in the physical world, like how humans learn language.
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF IIS 2112773.
241, 600 images from the validation dataset of Open Images Dataset V 6.

Review summaries:
This paper analyzes the linguistic embedding in the joint image-language space learned by cross-modal contrastive learning. Comparing with the standard word embedding, the grounded language embedding is human explainable. Specifically, the PCA of the grounded langauge embedding is aligned with human rating. When clustering the language embedding, the cluster of grounded language embedding is more semantically coherent than the un-grounded language embedding. Furthermore, the paper showed that the grounded langauge embedding could benefit visual-linguistic downstream tasks.

The paper implements a model which learns word representations from text grounded in images. The model is composed of a BERT-based text stream, a VGG-based image stream, and visual relation classification module. The text and image components are pre-trained and fine tuned on the COCO dataset of captioned images as well as on Visual Genome relations. The main contribution of the paper is the large number of analyses carried out on the induced grounded word embeddings, compared to plain BERT embedddings. The paper shows some evidence that the principal components of the grounded embedding space correspond to interpretable semantic dimensions (i.e. concrete-abstract), that words' semantic features can be predicted based on embeddings. The paper also claims that the model is capable of compositional understanding. 

This paper proposes a contrastive learning framework to learn visually grounded, and as a result, semantically interpretable language representations. It proposes two contrastive losses which make use of multiheaded cross-modal attention and bilinear interaction to compute similarity scores between subject and object representations, which can then we used in the contrastive loss.  They perform thorough experiments with the resulting representation to illustrate how well they correlate to human ratings along different language axes such as concrete-abstract and compositionality, as well as qualitative multimodal image search examples.

The paper aims to ground language to vision via cross-modal contrastive learning to build an explainable shared semantic space. This is achieved in two parts (i) align visual and language representations with MS COCO dataset (ii) Retrieve visual objects with language queries through a cross-modal attention module and infer the visual relations between the retrieved objects through a bilinear operator on Visual Genome dataset. This allows model's language encoder to become a stand-alone language model that can encoder embedding concepts in a visually grounded semantic space. The model enables compositional language understanding based on visual knowledge and multimodal image search with queries. The authors show experiments on clustering, compositional reasoning, relationship with human-defined norms, and image search to showcase the usefulness of the model.