Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning

1 Introduction . Humans take much longer time to name a colored word when the color and the word mismatch ( e.g. , “ red ” shown in green ) than when they match ( e.g. , “ red ” shown in red ) [ 1 ] . This effect is an example of rich psychological evidence suggesting that humans learn language by grounding meanings to knowledge about the world [ 2 , 3 ] . In contrast , most models in natural language processing ( NLP ) [ 4–7 ] encode “ distributional semantics ” [ 8 ] learned from texts only . Put yourself as machines in a thought experiment for the “ Chinese Room Argument ” [ 9 ] . Imagine that you have to learn Chinese from scratch as your first language . All that you have is a Chinese-to-Chinese dictionary . You might be able to relate a word to other words based on textual distributions . It is , however , impossible to learn word meanings without any additional explanation in reference to the physical world [ 10 ] . A language model may learn concepts from texts paired with sensory data , such as images . Joint vision-language learning has been explored for image captioning [ 11 ] , visual question answering [ 12 ] , and pre-training vision models with weak supervision [ 13 , 14 ] . In line with these studies , we 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . train a language model and a vision model jointly to match images and texts . We further analyze the semantic space obtained with the visually grounded language model . In this space , semantic embeddings are found to be organized and clustered by visual attributes , predictive of human-defined norms of semantic features , useful for compositional language understanding and cross-modal image search . We expect this visually grounded language model to also be useful for understanding the computational basis of grounded cognition [ 15 , 16 ] . 2 Background and Related Work . 2.1 Distributional vs. grounding hypothesis . In the distributional hypothesis [ 17 ] , words that occur in similar contexts carry similar meanings . This hypothesis has motivated influential machine learning models to learn word embeddings from large text corpora [ 4 , 18 ] . However , the learned word embeddings are not straightforward to interpret [ 8 , 19 ] . Alternatively , the symbol grounding hypothesis suggests that a word is connected to its meaning by relating to its referent in the physical world [ 10 , 20 ] . In line with this hypothesis , earlier studies demonstrate that visual features or contexts can enhance language learning [ 21–25 ] . 2.2 Vision-language learning . Grounding language in vision has been of increasing interest in computational linguistics and machine learning . A common strategy is to fuse words with related visual information in terms of perceptual norms [ 21 ] , bag-of-visual-word [ 23 , 26 ] , or learnable visual features [ 27–30 ] . The models used for vision-language fusion evolve alongside those for NLP , such as Latent Dirichlet Allocation ( LDA ) [ 23 ] , log-bilinear model [ 31 ] , Skip-gram model [ 25 , 27 ] , and recurrent neural network [ 28 ] . Generally , visual grounding may refine the distribution and interpretability of language representations [ 23 , 25 , 26 , 28–32 ] and facilitate cross-modal tasks [ 28–33 ] . More recent work has begun to use transformer [ 6 , 34 ] for vision-language learning , showing strong performance in cross-modal tasks [ 35–38 ] . Contrastive learning [ 39 , 40 ] is increasingly applied to not only unimodal data [ 41 ] but also multimodal data [ 13 , 42 ] . It is able to learn better representations than alternative prediction or classification objectives [ 43 ] . However , cross-modal contrastive learning is still under-explored for higher-level tasks , e.g. , visual question answering [ 12 ] , visual reasoning [ 44 ] , scene graph generation [ 45 ] . Such tasks involve abstract reasoning about the relations between entities ( e.g. , visual objects ) . Prior work approaches relational inference with multi-layer perceptron [ 46 , 47 ] or graph neural networks [ 48–50 ] . Arguably , a more compelling idea [ 51 ] is to model entities as vectors in a continuous space and to model their relations as arithmetic operators ( linear [ 52–54 ] or bilinear [ 55 , 56 ] ) applied to the vector representations of those entities . 2.3 Relation to prior work . In this work , we first build a two-stream model to jointly learn visual and language representation from image-caption pairs , similar to recent work [ 13 , 14 ] . We then finetune the learned model by adding a cross-modal attention layer [ 35 , 36 ] and bilinear operators [ 55 ] to represent the relations between visual objects . Both stages utilize cross-modal contrastive loss . Related to our work , Harwath et al . match visual objects to spoken words using triplet loss [ 42 ] . Early this year , Jia et al . [ 13 ] and Radford and Kim et al . [ 14 ] use contrastive learning to pretrain a vision model using a massive image-text dataset and demonstrate largely improved zero-shot transfer learning performance on visual and cross-modal tasks . Different from their perspectives , we focus on assessing the language encoders and word representations . Specifically , we perform a systematic evaluation of the semantic space grounded in vision vs. the ungrounded semantic space learned from texts only . This evaluation is possible since after training , the language and visual streams in our model are fully separable as stand-alone systems , unlike some vision-language models that require both visual and textual input to be usable [ 35 , 36 ] . Our goal is to assess how visual grounding affects the distribution of textual representations by analyzing the distribution of word embeddings in the grounded semantic space , in line with related works [ 23 , 25 , 26 , 28–32 , 57 ] . 3 Approach . 3.1 Visual grounding of natural language . To build a computational model for learning visually grounded language representations , we develop a model ( Fig . 1 ) that combines a stand-alone visual stream and a stand-alone language stream . The visual stream is based on VGG16 [ 58 ] with an additional linear transformation as an embedder to match the feature dimension of the language stream and an additional multi-head self-attention layer [ 59 ] to enforce global information aggregation and learn long-range dependency . The language stream is based on Bert [ 6 ] . Using separate linear transformation heads [ 41 ] , the output from both the visual stream and the language stream are projected to a common representational space . In this common space , the inner-product between the visual representation V at every location and the language representation L of every word gives rise to a 3D match-map , where each element indicates how a word in the text matches each location in the image ( See illustration in Fig . 1 ) . The sum of the maximal match is the similarity score S ( V , L ) between a pair of image and text . See Eq . 1 , where i , j indicate the location in the 2D image feature map V and k indicates the k-th word in L. M i , j , k = V i , j ·LTk , S ( V , L ) = K∑ k=1 max i , j M i , j , k ( 1 ) Extending the unimodal normalized temperature-scaled cross-entropy ( NT-Xent ) loss [ 13 , 41 , 60 ] , we define the cross-modal contrastive loss using the anchor sample from one modality and the positive sample and negative samples from the other modality [ 13 , 14 ] . As such , we define and sum two loss functions with the anchor sample from either images or texts and positive/negative samples from either texts or images , respectively . Lossl = − 1 B B∑ i=1 log exp ( S ( V i , Li ) /τ ) ∑B j=1 exp ( S ( V i , Lj ) /τ ) , Lossv = − 1 B B∑ i=1 log exp ( S ( V i , Li ) /τ ) ∑B j=1 exp ( S ( V j , Li ) /τ ) ( 2 ) For Lossl in Eq . 2 , the anchor sample V i is an input image and the positive sample Li is the corresponding image caption , whereas the negative samples Lj are unmatched textual descriptions included in the same batch ( B is the batch size ) . Similarly , Lossv in Eq . 2 is defined to contrast the positive and negative image samples against an anchor textual sample . 3.2 Visual grounding of object relations . We further finetune the model for visual relation prediction , as illustrated in Fig . 2 . In this stage , we remove the linear transformation heads in Fig . 1 and add a multi-head cross-modal attention module [ 35 , 36 ] . The attention module uses a query based on the embedding of an object word from the language stream ( QueryL ) and uses keys ( KeyV ) and values ( ValueV ) from every location in the feature map output from the visual stream . The attention score is calculated as the inner-product of QueryL and KeyV followed by softmax . The attention-weighted sum of ValueV is concatenated across 8 attention heads to generate a visually grounded object representation . Applied to the grounded object representation is a bilinear relation module for predicting the visual relation between two objects ( linguistically a subject and an object ) . For both the subject and the object , their grounded representations are linearly transformed to a subspace D = Rd ( default d = 32 ) , denoted as rs and ro , respectively . A predicate p is represented as a learnable bilinear operator Fp : D × D → R , which represents the relation embedding Rp . Applying this bilinear operator to the subject vs. object representations measures their relation score S specific to the given predicate [ 55 ] expressed as ( Eq . 3 ) . S ( rs , ro ; Rp ) = Fp ( rs , ro ) = rsRpr T o ( 3 ) For visual relation prediction , we also use contrastive learning with two loss functions by taking either relation embedding or subject/object representations as positive/negative samples . Lossrel = − 1 |B| ∑ ( rs , ro ; Rp ) ∈B log exp ( S ( rs , ro ; Rp ) /τ ) ∑ k∈Krel exp ( S ( rs , ro ; Rkp ) /τ ) ( 4 ) Lossobj = − 1 |B| ∑ ( rs , ro ; Rp ) ∈B log exp ( S ( rs , ro ; Rp ) /τ ) ∑ k∈Kobj exp ( S ( r k s , rko ; Rp ) /τ ) ( 5 ) In Lossrel , Krel is the set that contains all relations available . The anchor sample is a pair of subject and object in an image . The positive sample is the embedding of the ground truth relation . The negative samples are the embeddings of all other relations . In Lossobj , the anchor sample is a given relation . The positive sample is a subject-object pair that holds this relation . The negative samples are other subject-object pairs in a different relation . For both loss functions , the positive and negative samples are drawn from the same batch B . In addition , we also add a classification head ( two fully connected layers with ReLU in between ) and apply it to the grounded object representation . We use object classification as an auxiliary objective ( with a cross-entropy loss ) to constrain the grounded object representation to be separable across objects for classification .

Summary: This paper analyzes the linguistic embedding in the joint image-language space learned by cross-modal contrastive learning. Comparing with the standard word embedding, the grounded language embedding is human explainable. Specifically, the PCA of the grounded langauge embedding is aligned with human rating. When clustering the language embedding, the cluster of grounded language embedding is more semantically coherent than the un-grounded language embedding. Furthermore, the paper showed that the grounded langauge embedding could benefit visual-linguistic downstream tasks.