Contrastive Language-Image Pre-Training with Knowledge Graphs
Authors: Xuran Pan
Tianzhu Ye
Dongchen Han
Shiji Song
Gao Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: vision-language pre-training
knowledge graph

Abstract: Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

Contrastive Language-Image Pre-Training with Knowledge Graphs
Xuran Pan Tianzhu Ye Dongchen Han Shiji Song Gao Huang"
Department of Automation, BNRist, Tsinghua University, Beijing, China (pxr18, ytz20, hdc19) tsinghua. braannhua.edu.com edu.cn {shijis, gaohuang)Otsinghuss edu.
Abstract
Recent years have witnessed the fast development of large-scale pre-training frame- works that extract multi-modal representations in unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, ex- isting approaches mainly focus pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose knowledge-basee pre-training framework, dubbed Knowledge-CL.n CLI,, which injects semantic information into the widely used CLIP model [38]. Through introducing knowledge- objectives the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Noowedgee-CIII compared with the original CLIP and com- petitive baselines.
Introduction
Large-scale vision-language pre-training has attracted wide research interests in recent years [9, 26, 38, 72]. Different from training independent models for each specific task, pre-trained models take the analogy of human biological intelligence system, trying to perceive the world from various data modalities and handle comprehensive tasks. Specifically, it aims to provide a unified inference paradigm that simultaneously learns representations for multi-modal data and can easily transfer to variety of downstream tasks. Benefiting from the accessibility massive image-text pairs from the web, the vision-lanee languaee pre-training can leverage broader source of supervision, and effectively improves the model's ss generalization power.
Early attempts vision-language pre-training mainly focus on detecting objects in the images and aligning the corresponding word tokens with object regions [9, 28, 50]. Though effective, the entanglement with the concept of objects, and the additional resources for pre-trained object detectors impose restrictions on real-world applications. One of the pioneer works, CLIP [38], extends the scale of the pre-training dataset to 400 million image-text pairs, and learns representations by directly matching raw text with the corresponding image. Through contrastive-based training scheme, CLIP learns visual concepts under large vocabulary which significantly improves the model performances on various downstream tasks. Taking inspiration from CLIP, the following researches further extend the work from several perspectives, including data modality [72], downstream tasks [57], and training data efficiency [19, 44].
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

correct label: airliner
template: a photo of 0
template: not photo of
0.95
airtiner
0.80
airliner
0.02
military
0.02
airplane wing
0.01
albatross
0.02
military aireraft
0.01
taxicab
oxygen mask
high-speed train
0.01
space shuttie
(a) Results templates with opposite semantic description
correct label: sports car
template: photo of yellow 0
template: a photo of green 0
0.72
sports car
0.72
sports
0.07
race
0.10
race car
0.04
wheel
0.05
wheel
0.02
motorboat
0.02
spaghetti squash
0.02
amphibious vehicle
0.01
convertible
(b) Results on templates with wrong semantic description
Figure 1: CLIP fails to accurately capture some fine-grained semantic information. When given opposite semantic descriptions, e,g., adding 'not' in the template or describing an image with wrong color, CLIP tends to give similar distribution as the correct counterpart. Best view in color.
Although showing promising results, the current pre-training frameworks also suffer from limitations. Specifically, the data pairs for pre-training are organized in the simplest manner, where only the descriptions of matched and unmatched are used to represent the relation between given image and text pair. This usually leads to a degenerated scenario, where the model tends to rely on the co-occurrence of inputs instead of their semantic meanings. We give toy example in Fig. by evaluating the zero-shot transfer performance of CLIP on the ImageNet dataset [10] with the templates a photo of a and 'oot a photo of a ()'. It is shown that the distributions of CLIP outputs under two templates are quite similar, suggesting that the current model fails to understand the semantic meaning of word tokens. As result, the transferability of the model is restricted, and tends to show worse performances on tasks that require reasoning ability, egg., visual question answering.
To address the limitation of pre-traue pre-trained models on semantic perceiving, we resort to the technique of knowledge graph, which has been widely studied in the field of natural language processing [7, 58]. Knowledge graph (KG) is large-scale semantic network that comprises entities as nodes and semantic relations as edges. Through organizing data in graph structure, knowledge graphs provide rich information describing the relations between entities and enable reasoning process through data are favorable on various tasks including guestion-answering [18, 70], relation prediction [29, 43] and knowledge reasoning [6, 59]. In recent years, knowledge graph has also been investigated in the field of computer vision, e.g., scene graph [65], and the integration of both language and image [2]. This bridges the gap between different modalities in the knowledge graph, which inspires us to explore a new knowledge-bart based pre-training framework, and inject semantic information into simple image-text pairs.
the whole graph. These advantages over eggula-structuur
In this paper, we propose novel vision-language pre-training approach, dubbed Knowledge-CC CLIII, by constructing knowledge augmented pre-training framework based on the widely used CLIP models. As illustrated in Fig. 2, we follow the structure of CLIP, and use two Transformer- based models image and text encoders respectively. These two encoders take entities and relations in the knowledge graph as input and extract raw features for both entities and relations. Notably, entities can be in the form of image/text, while the relations are constantly described by language tokens. Then, multi-modal Transformer encoder is adopted to fuse the entity features conditioned on their relations. In this way, the pre-traines trained model is pushed to concentrate on understanding semantic relations between visual and word concepts, thereby establishing strong semantic connections between vision and language modalities.
To additionally improve the training efficiency and avoid the massive computation cost in the pre- training procedure, adopt simple continuous learning strategy by training model based

---

on the pre-trained weights of CLIP. This provides a possibility of efficiently promoting the model performance of CLIP with low training resources.
We train our model on three knowledge graph datasets, namely Visual-Genome [24] (scene graph), ConceptNet [46] (language-bard tanguage--baedd graph), and VisualSem [2] (multi-modal graph), and also adopt part of datasets from CLIP to avoid the model forgetting problem. With the knowledge-enhanee pre-training, Knowledge-CLIF achieves consistent improvements over the original CLIP models on various vision and language downstream tasks.
2 Related works
Large-scale pre-training. Benefited from the development of Transformer in both vision [35, 63, 36] and language [54] tasks, large-scale pre-training framework has received wide concerns in recent years and shown promising results in the field of computer vision and natural language processing. GPT [39] is one of the pioneer works for language pre-training which optimizes the probability of output based on previous words in the sequence. BERT [11] adopts the masked language modeling technique and predicts the masked tokens conditioned on the unmasked ones.
Similarly, computer vision society also witnesses the development of pre-training models thanks to the emergence of large-scale image datasets. IGPT [5] proposes generative pre-training technique and shows promising results on classification task. MAE [17] adopts a similar pre-training scheme as BERT and predicts the masked regions of an image with unmasked ones.
Multi-modal pre-training bears differences from the aforementioned frameworks and requires the alignment between various data modalities. Using enormous image-text pairs collected from Internet, vision-language models show significant improvements on various downstream tasks. Among these approaches, various pre-training scheme is adopted, including contrastive learning [1, 27, 31], masked language modeling [47, 51], and masked region modeling [9].
The problem of semantic misunderstanding has also been investigated by previous works. EI- CLIP [33] considers the problem of cross-modal retrieval in the field of E-commerce. Sharing similar insight with our work, the authors notice the model bias towards some specific word tokens in CLIP, and introduce causal inference to align the text encoder with e-commerce domain knowledge. K3M [73] focuses on the modality-missing and modality-noiss problem and introduces knowledge modality into E-commerce tasks. De [69] studies the spurious correlations between different modalities and adjusts the conditional probability of image tokens and word tokens. Kaleido-
BERT [74] focuses on image-text coherence by introducing several novel self-supervised tasks.
Compared to previous approaches, we are the first to incorporate multi-modal knowledge graphs into the pre-training process, and effectively enhance the model perception semantic relations between visual and language concepts.
Knowledge graph is first introduced in the field of natural language processing, and the knowledge graph embedding approaches have been successful on capturing the semantics of symbols (entities and relations) and achieving impressive results on wide range of real-world applications including text understanding [13, 66], recommendation system [16, 56] and natural language question answering [18, 70]. On the other hand, scene graphs represent type of graph- structured data in computer vision, where the visual concepts in the image are connected with semantic relations. Scene graphs emphasize the fine-grained semantic features for images and widely adopted in various downstream tasks, including scene graph generation [65], and Scene Graph Parsing [68]. Besides scene graph, knowledge graph is also adopted in other computer vision tasks, including image classification [22], panoptic segmentation [62]. and image captioning [71]. On this basis, multi-modal knowledge graph earns wide concers in recent years. Considering the natural alignment between different data modalities, multi-modal knowledge graphs have been widely adopted in various graph-based tasks including link prediction [3, 30), entity classification [61], while also showing great potential on out of graph applications like visual question answering [20, 41] and recommendation systems [49, 52].
Knowledge Graph.
3 Contrastive Language-Image Pre-training (CLIP)
We first provide brief review of model architectures and training settings in CLIP.
3

---

A. Model Sirucutre of Knowlelg=CLIIP (sup.))) Tripkt reprosunaaiion
B. Training Objectives
R(s,p,o) Redation reproseatation
Oktrr
CLLP
Karnlote-CLIP
Malti-madal Encader (Randoom Initialization)
tripltt
triplct sequince
De
Imeage Reendde (CI Inliakndtion
Ieat Eocader (C' IP Initatioaiin)
lange per anhell
Eubt
lust rabted
Knowlelgg ClaP
Knowledge CLPP
text tokens
Example Input Triplet (h,ss)):
bas
Castle in Europe
Clasitistinn Lass
Tea
Gropth Lorn
An overview of our framework. (A) Given a data triplet h, r, r with entities h, and their relation r, image and text encoders first extract raw features, then multi-modal encoder consumes the concatenated triplet sequence and outputs triplet and relation representations. (B) Three types of training objectives adopted in our framework.
Figure 2:
CLIP uses two separate models for image encoder and text encoder respectively. For text inputs, a 12-layer Transformer is adopted with 512 width and 8 attention heads. Raw texts are first converted using byte pair encoding [40] technique under a vocabulary size of 49,152. The text sequence length capped at 76 and added by positional encoding before being sent into the text encoder. On the other hand, CLIP has different versions of image encoder with ResNet-based and Vision Transformer- architectures. As the following researches have demonstrated the better performances of Vision Transformer models, we only consider Transformer-base the text input, images are first converted to patches, and added by positional encoding. At the last stage of both encoders, a global pooling function is adopted to compress the feature map into a single feature, which serves the representation of the whole image/text sequence. The cosine distance of the image and text features is computed as the similarity of the data pair. For training supervision, a contrastive loss adopted to maximize the similarity of matched pairs while minimizing the similarity of unmatched pairs. Given a batch of N data pairs (T. T where T and represents the ith image and text respectively, the loss function can be parameterized as:
image encoders in this paper. Similar to
log exp(cos((f(44)::1( fr()) llog exp(cos((f(L))
L
j=1 exp(cos((f1()).. (rr()))))))(()))) (T(T))/T))
(1)
where fi and fr correspond to image and text encoders respectively, cos(-) denotes the cosine similarity between the inputs, and r is learnable temperature initialized at 0.07.
This simple training framework actually brings several concerns that need to be addressed. First, the pre-training framework fails to model the semantic information of inputs due to the simplicity of the data structure. This results in inferior performances tasks that require reasoning ability, e.g., visual question answering and visual commonsense reasoning. Second, the image and text features reside in separate spaces, which makes it difficult to model the interactions between different modalities. Third, the massive time and resource consumption in the training procedure set restrictions on performing a full pre-trate mre-training schedule from scratch.
4 KnOW
As we have summarized above, there are several concerns that hinder the transferability of CLIP and potential improvements on model performances. In this paper, we propose a novel pre-training framework based on knowledge graphs, that addresses the limitation of the original CLIP model from several perspectives: (1) we introduce knowledge graphs into the training dataset where the graph-structured data and semantic relations between concepts enable the model to extract semantic features and establish semantic connection across inputs; (2) multi-modal encoder is added on top of the current image and text encoders to fuse the features from different modalities, and model the

---

Pollution
Water conservation
Atmospharic optrs
Teacher
Destre
Aatroyuu
Student
Hulding
Clane
Rolatate
Hos4
RelatedTo
Pelitedio
A Attation
Panor
Sunnlnnon
Study
Air quality Index
(a) VisualSem
(b) Visual G enome
(c) ConceptNet
Illustrations of the pre-training knowledge graph datasets, including ViusalSem [2] (multi- modal graph), Visual Genome [24] (scene graph), and ConceptNet [46] (language-basser graph).
Figure 3:
joint distribution between inputs; (3) continuous learning strategy based on the pre-trained model of CLIP is adopted which avoids the massive computation cost in the pre-training procedure, and enhance the generalization power of the model efficiently. We introduce our framework in detail in the following sections, and show the overview in Fig. 2.
4.1 Data Preparation
Different from raw image-text pairs adopted in the original CLIP, our model takes knowledge graphs as input. A knowledge graph can be defined as directed graph G {E, R. TR]. where 6, R correspond to sets of entities and relations, and TR represent the set of relation triplets. A triplet rh,r,t) t) e TR denotes that entity h E has relation R with entity &. As illustrated in Fig. we pre-train our model on three types of knowledge graphs, including multi-modal knowledge graph, scene graph, and language- based knowledge graph. Among these, relations are constantly described in language tokens, where the entities are from different modalities in different forms.
For multi-modal knowledge graph, the entities contain both illustrative images and language descrip- tions. Through representing the same entity under various modalities and connecting entities with relations, it helps build semantic connections between vision and language concepts. In practice, language and vision descriptions are randomly chosen for each entity, In this way. the triplet set TR contains different forms including (Img, Rel, Img), (Img, Rel, Text), and (Text, Rel, Text). providing rich information across modalities while also enhancing perceptions within modalities.
Different from multi-modal knowledge graph, scene graph extracts visual concepts (mainly objects) for each image, and connects them with predefined semantic relations describing relative locations, actions, etc. Therefore, the entities in the scene graph correspond the triplet form of (Img, Rel, Img). We practically use the selected regions as the input and discard the irrelevant parts. As two entities the same triplet denote different regions in the same image, forces the model to extract more fine-grained features.
certain region in image, with
Lastly, language-based knowledge graph connects words and phrases of natural language with labeled edges. is built on only language modality with the triplet form (Text, Rel, Text), while helping to build semantic alignment within word tokens.
4.2 Model Architecture
The model architecture and the training framework are illustrated in Fig. 2(A). Specifically, first process the inputs into token sequences with modality -specffc tokenizers. The BPE tokenzier [40] is adopted for language inputs, while image inputs into a sequence of patches following ViT [12]. For convenient processing, we set the length of the image sequence and text sequence 4 and by respectively for all inputs. To preserve the relative position information in the input, learnable positional encodings are added to the corresponding sequences before being sent the model.
sliced into non-overlapped patches and converted
Two separate image encoder fi(.) and text encoder fr(-) are then adopted to extract features from raw inputs. For given triplet (h, r, t), the entities f and t sent the encoders with respect to
5

---

their modalities (image or text). The relation which is represented by language tokens, is sent to text encoder similar to text entity.
Compared to the model structure in CLIP, we introduce a modification to better adapt our framework. Specifically, vanilla CLIP models pooling function the last layer of two encoders compress the feature map into a global representation. Namely, for an input E R RLxd; where L and d, denote the sequence length and feature dimension, the output of the encoder can be formulated as:
Fu = f(u) E Rlxd, Fy = (o) R
where f represents the feature extraction module, Pool(-) denotes the pooling function, and d, the output dimension. Though efficient, it also leads to inevitable information loss in the local region, especially for the image inputs. Therefore. we remove the pooling functions for image and text entities to preserve the local information, and use Rlxdd as the extracted feature. The relation, on the other hand, normally under a limited sequence length, e.g., one or two word tokens, where the information density is smaller than entities. Therefore, we retain the pooling function for relation input and use T. E Rd as the extracted features.
In this way, we have extracted the features defined as (25, TT), which correspond to the elements in the input triplet rh,r,t), To model the joint distribution of different elements in the triplet, we consider a multi- Specifically, we first concatenate all the features in the triplet into a single sequence and use head token < head > at the beginning of the sequence. To emphasize the status of the tokens in the sequence, we consider additional learnable encodings for each element h, r, & in the triplet:
encoder TransEncoder(() to fuse the features from different sources.
X (h, r) [<head>, 2h + PEA, T, PEE, It PP,]
(3)
After processing by the multi-modal encoder, the feature of the head token <head> finally serves as the representation of the whole sequence:
(h, r, 6) = (aansnncdee
(4)
Also, representation for relation is extracted from the corresponding token:
R(h, r,t) t) (ransEnccdder((((h..r1 (en())))))))))))))())))))))))..
(5)
4.3
Training Targets
Considering the unique data structure of knowledge graphs, we mainly adopt two types of training targets in our framework, including triplet- ppsset-based loss and graph-based based loss illustrated in Fig. 2(B). Besides, a knowledge distillation loss is also considered due to the continuous learning strategy adopted in our framework.
Triplet-based loss considers a batch of triplets as the input and supervises the training of our model by estimating the joint distribution of elements in the triplets. Inspired by the mask prediction technique that models the distribution of masked tokens conditioned on the unmasked regions, we similarly mask the elements in the triplets and predict the distribution with the help of a multi-modal encoder. Specifically, for incomplete triplcts where certain elements are missing in the input, the concatenated sequence can be similarly derived as in Eq. 3 by masking the corresponding feature. For example, the concatenated sequence for an input (h, r.- can be represented as:
X(h, r, -) [<head>, on + PE,, Fr PE,, 0).
On this basis, given set of input D = ((h,, (.) we first model the distribution when one of the entities, i.e., tú, is masked, and derive the Entity-Entity (E2E) Loss by minimizing the negative log-likelihood:
-logg P (arel T,)))
(7)
We practically approximate the distribution P T,) the cosine similarity of P(xx) and P(r,, Z,), and defined the loss function as:
N
exp(cos( (h,, rh:)))
Le28 = log(
(8)
Σ, ( (hy,, -))=))

---

We also model the distribution when the relation in the triplet is masked, and similarly derive the Entity-Relation (E2R) Loss:
P(h..)-Doogl (,,, [.))-
Different from E2E loss, the relations in the triplets are defined in limited set of relation groups. Therefore, we instead extract the representation of relation through an auxiliary two-layer ML.P network, and model the objective as classification problem from predefined set of relation labels R. In this way, the loss function can be defined as:
N
LE2R = Σ Σ
(0=r))oo((((r.)) where y(Tr,) MLP(R((,,,4)))
(10)
im
PER
is extracted from an MLP model followed by the output of multi-modal encoder defined in Eq. (5).
Graph-based loss. We also take advantage of the graph structure in knowledge graph datasets, and adopt a graph neural network to extract deeper structural information among entities. We propagate information through connected edges in the graph, and update entity representations with aggregated feature. Specifically, for graph neural network with L layers, the update function for the 4h layer can be formulated as:
(t) ElhordeTTT
- Y(--t)((-))))
(11)
where gd (R(h,, ()) = W R(s:-,,))
(12)
calculates the aggregation weights by relation representation R[h,,- with learnable matrix W(
Finally, we define the before and after the propagation procedure in the graph:
Loss by computing the cosine similarity of entity features
1 log( Ne
(xp(cos(Y((,,))a)) G (L) KEE Σn, exp(cos(V((.-.)) (L))T)))
LG2F
(13)
Continuous Learning. Large-scale pre-training usually requires massive computation resources which makes it highly inefficient when training from scratch. Therefore, to inject the semantic information an efficient manner, we consider training our model based on the pre-trained weights from the original CLIP. This powerful initialization promotes the convergence of our model and greatly enhances the training efficiency. However, naively extending the training process with new data leads to severe forgetting problem that hampers the performance of the original models.
To address this limitation, we adopt simple solutions to maintain CLIP performances while improving its ability to extract semantic features from knowledge graphs. (1) Besides the knowledge graph datasets, we also train our model on several widely adopted image-text datasets that share a similar data distribution with the training data in CLIP. To better fit our pre-training framework, we convert the original image-text pair into the form of triplets, with specifically designed relations 'image of" and cappton of". (2) We also use the original CLIP model as the teacher, and use auxiliary loss LKD to measure KI. distance between the output of CLIP and our model.
Overall, the final pre-training objective of Knowledge-CLIP is formulated as:
L = LEZE + LEZR + LGzE + LKD-
(14)
Experiments
5.1 Implementation Details
Experimental Setup. In all the experiments, we use the same model structure as CLIP [38]. A 12-layer Transformer model with 512 width adopted for text encoder, and ViT-L/14 is adopted for image encoder. For text and image encoder, we use the pre- pre-trained weights in the original CLIP as the initialization. For the multi-modal encoder, we consider 4 layer Transformer model with 1024 width. The rate for drop path is set as 0.1 during training. As the added multi-modal encoder is trained from random initialization, we decrease the learning rate for the pre-trained weights from CLIP to achieve more balanced step in the optimization. We train Knowledge- CLII with initial
7

---

Table 1:
Fine-tuned image-text retrieval results on Flockr30K and COCO datasets. The best result is shown in blue and the better result between CLIP and our approach is shown in bold.
Flickr30K (IK test set) Text Retrieval
MSCOCO(5K test set)
Method
Image Retrieval R@1 R@5 @@ 10 R @@l R @@5 R@ 10
Text Retrieval
Image Retrieval R@5 R @ 10 R@ R R@5 R@10
R @@
UNITER [9]
87.3 98.0
99.2
75.6
94.1
96.8
65.7
88.6 93.8
52.9 79.9 88.0
VILLA [14]
87.9 97.5
98.8
76.3
94.2
96.8
OSCAR [28]
73.5
92.2
96.0
57.5
82.8
89.8
ERNIE-ViLL ViL [67]
88.7
98.0
99.2
76.7
93.6
96.4
Unicoder- VL [25]
86.2
96.3
99.0
71.5
91.2
95.2
62.3
87.1
92.8
48.4
76.7
85.9
ViLT [23]
83.5
96.7
98.6
64.4
88.7
93.8
61.5
86.3
92.7
42.7
72.9
83.1
Uni-Perceiver [72]
87.9
98.2
99.1
74.9
93.5
96.0
64.7
87.8
93.7
48.3
75.9
84.5
CLIP [38]
88.6
98.5
99.4
72.4
92.3
96.6
67.3
85.4
92.4
54.3
83.5
90.0
Ours
89.2
98.9
99.4
75.7
94.4
96.8
70.2
89.2
94.4
57.6
83.9
90.4
learning rate of le-5 for image and text encoders, and 1e-3 for the multi- encoder. Cosine learning rate with linear warmup is used in the training schedule. Weight decay and gradient clip are also adopted. See more details in the supplemental material.
Pre-train Dataset. Three knowledge graph datasets are adopted in the pre-trate pre-training process. Visu- alSem [2] is high-quality
knowledge graph dataset for vision and language concepts, including entities with multilingual glosses, multiple illustrative images, and visually relevant rela- tions, covering total number of 90k nodes, 1.3M glosses and 938k images. 13 semantic relations are used to connect different entities in the graph, while the entities in VisualSem are linked to Wikipedia articles, WordNet [34], and high-quality images from ImageNet [10]. Visual Genome [24] is knowledge-bart
scene graph dataset that connects structured image concepts with semantic relations. Visual Genome serves as the benchmark for various vision tasks, e.g., visual grounding, and scene graph generation. ConceptNet [46] is a knowledge graph that connects words and phrases of natural language with labeled edges. Its knowledge is collected from many sources including expert-creater resources and crowd-sourcing built on only language modality.
Besides the three knowledge graph datasets, we also train our model on two widely adopted image- text datasets that share the similar data distribution with the training data in CLIP. We practically add COCO Caption [8] and CC3M [42] to the training set, while large-scale datasets like CC12M [4] or YFCC [21] are not considered to maintain training efficiency.
Downstream Task. To validate the effectiveness of our framework, we conduct experiments on various downstream tasks, including multi-modal tasks like text and image retrieval, visual question answering, and uni-modal tasks like image classification and natural language understanding.
5.2 Multi-modal Tasks
Visual question answering Visual Entailment. We also validate the ef- fectiveness of Knowledge-CCL CLIIP on other vision-language tasks, includ- ing VQA [15] and SNLI-VE [64]. We show the comparison results in Tab. 2. Compared to competitive baselines including VILLA [14] and
Table 2: Fine-tuned results on other V-L. tasks.
VQA
SNLI_V VE
Method
test-dev
test-std
val
test
UNITER [9]
72.70
72.91
78.59
78.28
VILLA [14]
73.59
73.67
79.47
79.03
OSCAR [28]
73.16
73.44
ALBEF [26]
74.54
74.70
80.14
80.30
Uni-Perceiver [72]
73.4
74.1
ALBEF [26], Knowledge- CLIP with VIT-L/14 shows better performances under all settings, while the smaller model also achieves competitive re-
FLAVA [45]
72.8
78.89
CLIP [38]
74.10
73.56
79.51
80.01
Ours
76.11
75.24
80.52
80.97
sults. Compared to the original CLIP model, our pre-trained model practically improves its transfer- ability on downstream tasks, especially on the datasets like VQA that requires reasoning ability.
Image and text retrieval. We first conduct experiments on Flickr30k [37] and COCO Caption [8]
datasct to show the performances of our model on image-text retrieval tasks. Given input sets X
8

---

Table 4: Fine-tuned language understanding results on GLUE dataset. The best result is shown in blue and the better result between CLIP and our approach shown in bold.
CoLA Mce.
SST-2
RTE
MRPC Acc./F1
QQP
MNLI Acc
QNLI
Method
Acc.
Acc.
Acc./F1
Acc
VilBERT [32]
36.1
90.4
53.7
69.0/79.4
88.6/85.0
79,9
83.8
VI.-BETT [48]
38.7
89.8
55.7
70.6/81.8
89.0/85.4
81.2
86.3
UNITER [9]
37.4
89.7
55.6
69.3/80.3
89.2/85.7
80.9
86.0
SiMVLM [60]
46.7
90.0
63.9
75.2/84.4
90.4/87.2
83.4
88.6
FLAVA [45]
50.7
90.9
57.8
81.4/86.9
90.4/87.2
80.3
87.3
CLIP [38]
42.1
90.5
59.2
82.4/87.0
90.4/87.1
80.9
87.1
Ours
50.4
91.2
62.4
83.5/87.6
90.5/87.9
83.6
89.5
and Y/ of images and texts, we use Knowledge-CLIP to extract features for each input, and model the joint probability with the cosine similarity between image and text pairs. We summarize the comparison results of Knowledge-CLLe MLIP with competitive baselines in Tab. 1. It is shown that our model consistently achieves better results over the original CLIP on both datasets, while comparable with competitive baselines like OSCAR.
5.3 Uni-modal Tasks
Image Classification. To further demonstrate the generalization power of Knowledge-CL.n we compare the performances of pre-train models the ImageNet classification task [10]. We summarize the comparison results in Tab. 3, and show that Knowledge-CLIP can also handle vision tasks well. We argue the improvements over baselines may attribute to the scene graphs in our pre- pre-training dataset, which emphasize the visual concepts in the images.
Table 3: Fine-tuned results on ImageNet.
Method
Acc(%)
DeiT [53]
83.4
CLIP [38]
84.2
Ours
84,4
Language Understanding. We validate the generalization performance of Knowledge-CLIP for language understanding tasks on the widely adopted GLUE dataset [55]. Specifically, we conduct experiments on 7 tasks GLUE and summarize the comparison results Tab. 4. shown that our model achieves comparable performances with competitive baseline models. Also, for tasks like QQP and MNLI that require sentence-pan eatence--pir matching. Knowledge-CLIP shows higher performances, due to the existence of language triplets in the pre-training dataset.
5.4 Ablation Studies
To validate the effectiveness of the components in work, carefully design several settings, including (1) CLIP+continuous learning: we train vanilla CLIP (pretrained weights as initialization) on knowledge datasets adopted in our work; (2) Knowledge-CLIN Chaee--CLPP-(1111 12, t3): remove the training objectives respectively our work to analyze the contribution of each loss. For all experiments, we adopt smaller model (ViT-B/32) worth noticing that KD loss plays a vital role in the continuous learning scheme, without which will lead to significant performance drop due the model forgetting problem. Therefore, we use KD loss in all the ablation settings for fair comparison.
the image encoder of CLIP in the ablation study. Also, it is
We show the comparison results two representative tasks in Tab. 5, including the image/text retrieval task on Flickr30K, and the visual question answering task VQA. Several observations be made from the ablation: (1) All three training objectives (E2E, E2R, G2E) contribute to improving the model performance. Training the model without any of the objectives leads to inferior performances downstream tasks. We argue that the E2E, E2R, and G2E loss promote the model from different perspectives by focusing on semantic understanding of concepts, complicated relations between entities, and structural information. Therefore, all three objectives are necessary for the framework and contribute to the improvement respectively. (2) By comparing the first and second row, can see that simply training the CLIP model with extra time and data fails to improve the generalization performance. It also demonstrates that the improvements mainly come from the injected knowledge information rather than the continuous learning scheme.

---

Table 5: Ablation studies of continuous learning / training objectives. We report results on Flickr30K retrieval task and VQA task with ViT-B/32 as image encoder.
Flickr30K retrieval
VQA
Model
KG data E2E E2R G2E
Text (R@) 1) Image (R@) 1) test-dev test-std
CLIP
84.2
63.1
68.9
69.2
CLIP+Continuous Learning
84.5
63.0
69.1
69.5
Knowledge-CLiP..
v v
85.0
64.6
70.4
71.1
Chowledge-CLIPe.
V
/
v
85.7
66.0
71.2
69.9
Showledge-CLIPe
84.9
65.8
70.2
70.4
Knowledge- CLiP (Full)
v
86.3
67.2
72.5
72.7
We also conduct an ablation study on the KD loss adopted for continuous learning and summarize the results in Tab. 6. The model achieves lower results after removing the KD loss, indicating its vital role in the continuous learning scheme. We argue the reason for this phenomenon is that the model suffers from the forgetting problem, which is widely spotted in the field of lifelong learning and continuous learning.
Table 6: Ablation studies of the KD loss.
Flickr30K retrieval
Model
Text (R @ 1) Image (R @ 1)
Ours w/o KD
62.5 82.4 86.3
Ours w/ KD
67.2
5.5 Analysis on particular semantics
We also conduct experiments carefully selected data which may better reflect how a vision- language model understands particular type of input. Specifically, we select questions in the VQA dataset that contains (1) Negations; (2) Color attributes; (3) Position attributes; (4) Sizes. We summarize the comparison results CLIP and our model on these sub-datasets in Tab.
As we can observe, our model achieves consis- tent improvements over CLIP on these specially designed datasets and shows significantly better results. Regarding questions with negation, our model achieves 2. % higher accuracy. Regarding color and position attributes, our model shows even higher improvements. We believe these compar- isons on different 'semantic domains' demonstrate the effectiveness of injecting knowledge information into the current vision- vision-language pretraining framework which practically enhances the model perception of semantic understanding.
Table Ablation studies on semantic inputs.
Dataset
CLIP
Knowledge- CLII
Negation Color
64.7
66.8(42.1)
54.2
59.9(45.7
Position
61.2
68.3(47.1)
Size
72.1
63.4(+1.))
Conclusion
In this paper, propose novel vision- lanoualan pretraining framework that incorporates knowledge information model the semantic connections between vision and language entities. We introduce three types of graph-structured datasets into the training process, and adopt a multi-modal encoder to model the joint distribution of entities and their semantic relations. Extensive experiments on various downstream tasks including multi-modal, uni-modal, and graph-based tasks validate the transfer and generalization ability our model. Our approach now limited injecting knowledge information into the CLIP models. However, our training objectives and new knowledge graph datasets are technically compatible with other large-scale pretraining frameworks. We will explore the possibility of further applications in the future.
7
Acknowledgement
This work supported in part by the National Key R&D Program China under Grant 2020AAA0005000, the National Natural Science Foundation of China under Grants 62022048, Guoqiang Institute of Tsinghua University and Beijing Academy Artificial Intelligence. We also appreciate the generous donation of computing resources by High-Flyer AI.
10

---

References
[1] Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint 1XXi::908.000000 2019.
[2] Houda Alberts, Teresa Huang. Yash Deshpande, Yibo Liu, Kyunghyun Cho, Clara Vania, and lacer Calixto. Visualsem: high-quality knowledge graph for vision and language. arXiv preprint arXiv:2008.091500 09100, 2020. 2, 5,
[3] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Dural Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703, PMLR, 2020.
[6] Wenhu Chen, Wenhan Xiong, Xifeng Yan, and William Wang. Variational knowledge graph reasoning. arXiv preprint arXiv:/803.0663 065811 2018.
[7] Xiaojun Chen, Shengbin Jia, and Yang Xiang. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications, 141:1 12948, 2020.
[8] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 104.001504.00335, 2015. 8
and Jingjing Liu. Uniter: Learning universal image-text representations. 2019. 1, 8, [9] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng,
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and patlern recognition, pages 248-255 Icee, 2009. 2, 8,
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Ming-Wei
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 5
[13] Hao Fei, Yafeng Ren, Yue Zhang, Donghong Ji, and Xiaohui Liang. Enriching contextualized language model from knowledge graph for biomedical information extraction. Briefings in bioinformatics, 22(3):bbaal 10, 2021.
[14] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 33:66 18666666288 2020. 8
[15] Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 6904-6913, 2017. 8
[16] Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong, and Qing He. A survey on knowledge graph-based recommender systems. IEEE Transactions on Knowledge and Data Engineering, 2020. 3

---

[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
[18] Xiao Huang, Jingyuan Zhang, Dingeheng Li, and Ping Li. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining. pages 105-113 2019. 2, 3
[19] Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding. Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems, 34, 2021.
[20] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019. 3
[21] Sebastian Kalkowski, Christian Schulze, Andreas Dengel, and Damian Borth. Real-time analysis and visualization of the yfcc 100m dataset. In Proceedings of the 2015 workshop on community-organzzed multimodal mining: opportunities for novel solutions, pages 25-30, 2015.
8
[22] Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11487-11496, 2019.
3
[23] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021. 8
[24] Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017, 3. 5.8
[25] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross- modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336-11833 1134,, 2020. 8
[26] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021. 8
[27] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 198.v:1908.03557, 2019.
[28] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision- vision-language tasks. In European Conference Computer Vision, pages 121-137. Springer, 2020. 1, 8
[29] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence, 2015. 2
[30] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference artificial intelligence, 2015. 3
[31] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
12

---

[32] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for iaiio-and-languwee tasks. Advances in neural information processing systems, 32, 2019. 9
[33] Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. Fi-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages I 18051-18061, 2022. 3
[34] George Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995. 8
[35] Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $15-825, 2022. 3
[36] Xuran Pan, Zhuofan Xia, Shiji Song. Li Erran Li, and Gao Huang. 3d object detection with pointformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7463-7472, 2021. 3
[37] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phae to-paasse correspondences for richer amage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015. 8
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 9748-8763. PMLR, 2021.
[39] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. 3
[40] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint 1xXiv:5508.79900 2015. 5,5
[41] Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 8884, 2019. 3
[42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 2556-25666 2018.
[43] Baoxu Shi and Tim Weninger. Open-world knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 2
[44] Aman Shrivastava, Ramprasaath R Selvaraju, Nikhil Naik, and Vicente Ordonez. Clip-lite: Information efficient visual representation learning from textual annotations. arXiv preprint arXiv:21/2.07133 2021.
[45] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint 122042211..04482, 2021. 8,9 9
[46] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. Thirty-first AAAI conference on artificial intelligence, 2017. 3, 5, 8
[47] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei La Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre- training of generic visual-iinguistic representations. arXiv preprint arXiv: 08530, 2019. 3
13

---

[48] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre- training of generic visual- lisual-ligguistc representations. arXiv preprint arXiv:1908.085 2019.
[49] Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering techniques. Ad- vances in artificial intelligence, 2009, 2009. 3
[50] Hao Tan and Mohit Bansal. Lxmert: Learning coos-mmdaaltty encoder representations from transformers. arXiv preprint
2019. 1
[51] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 2019.
[52] Shaohua Tao, Runhe Qiu, Yuan Ping, and Hui Ma. Multi-modal knowledge-aware reinforcementt learning network for explainable recommendation. Knowledge- Based Systems, 227:107217, 2021. 3
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021. 9
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 3
[55] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.08800007466 2018. 9
[56] Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In Proceedings of the 27th ACM international conference information and knowledge management, pages 417 426, 2018.
[57] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai. Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to-sequence learning framework. arXiv preprint arXivv:202..0 03052, 2022.
[58] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):272--2743, 2017. 2
[59] Zhouxia Wang, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, and Liang Lin. Deep reasoning with knowledge graph for social relationship understanding. arXiv preprint 1807.00504 2018. 2
[60] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021. 9
[61] WX Wilcke, Peter Bloem, Victor de Boer, RH van Veer, and FAH van Harmelen. End-to-end entity classification on multimodal knowledge graphs. arXiv preprint 2xXx:20003.13382 2020.
3
[62] Yangxin Wu, Gengwei Zhang, Yiming Gao, Xiajun Deng, Ke Gong, Xiaodan Liang, and Liang Lin. Bidirectional graph reasoning network for panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 3
[63] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 4794 4803, 2022. 3
[64] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding, arXiv preprint arXiv:1901.06706 2019. 8
14

---

[65] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cmn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV). pages 670-685, 2018. 2,
[66] Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowledge graph and language understanding. arXiv preprint arXivv2010.00796. 2020. 3
[67] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced lision-language representations through scene graph. arXiv preprint arXiv:2006. 16934, 2020. 8
[68] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831-5840, 2018. 3
[69] Shengyu Zhang, Tan Jiang, Tan Wang, Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang. and Fei Wu. Devlbert: Out-o--dist visio-linguistic pretraining with causality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1744-1747, 2021.
[70] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. Variational reasoning for question answering with knowledge graph. In Thirty-second AAAI conference artificial intelligence, 2018. 2,
[71] Wentian Zhao, Yao Hu, Heda Wang, Xinxiao Wu, and Jiebo Luo. Boosting aatet image captioning with multi-modal knowledge graph. arXiv preprint arXiv:2107.11970, 2021.
[72] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang. and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. arXiv preprint IrXiv:2112.0152: 2021. 1,8
[73] Yushan Zhu, Huaixiao Zhao, Wen Zhang, Ganqiang Ye, Hui Chen, Ningyu Zhang, and Huajun Chen. Knowledge perceived multi-modal pretraining in e-commerce. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2744 27522, 2021.
[74] Mingehen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao. Kaleido-bert: lanionelanguage pre-training fashion domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12647-12657, 2021. 3
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We have addressed our contribution.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] have carefully read the ethics review guidelines and checked paper.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code will be released when the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5 and supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running ex- periments multiple times)? [No] We follow the experiment routine in the previous works.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section 5.
(b) Did you mention the license of the assets? [Yes] See Section 5.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you'l 'ou'r using/curating? [No] The dataset is readily available in public.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "xc-SPpb5GK8", "reviewer": "Reviewer_Foeh", "summary": "- The paper proposes a new Contrastive Language-Image Pre-Training method that incorporates semantic knowledge graphs (knowledge-CLIP). The method builds on the existing CLIP architecture and introduces three KG-aware pretraining objectives on top: Entity-Entity loss, Entity-Relation loss, Graph-Entity loss. The authors experiment with this method using multiple knowledge graphs and show improved performance across multimodal, unimodal, and KG-based downstream tasks.", "questions": "- Please find my main questions and suggestions mentioned in the \"Weaknesses\" section above. \n- Will the code/data/trained models be released?\n- For Figure 1b, what is the expected prediction when changing \"yellow\" to \"green\"? I wonder \"a photo of green __\" may not really make sense for the given image because there is no green object in the image?\n- For graph-based loss (Equation 11), do you compute Y(-,-,t) for all the triplets using knowledge-CLIP? Would  that be very expensive? Also, what is the size of the graph considered in the GNN - is it the entire KG or some subgraph?\n- A suggestion for the presentation: In Figure 2, maybe make the font consistent across A (left panel) and B (right panel)?", "limitations": "- The authors addressed the potential negative societal impact (in Appendix)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The paper is generally well-written and easy to understand. \n- The problem they are solving (incorporating background semantic knowledge into language-image model) and their proposed method (KG-aware model architecture and training objectives) are well motivated. The idea of considering a triplet set that contains different forms, (Img, Rel, Img), (Img, Rel, Text), (Text, Rel, Text), is interesting.\n- The experiments are solid (covers three types of KGs; covers both multimodal and unimodal downstream tasks) and shows that the proposed method makes moderate improvements over the baseline.\n\nWeaknesses\n- While the authors show improved numbers on benchmark datasets, it would be nice to also show and discuss how the proposed knowledge-CLIP model is *qualitatively* improving over the baseline CLIP. For example, in Intro and Figure 1, the authors motivates this paper by arguing that the baseline CLIP only captures text-image co-occurrence and fails to adjust for negation in text, etc. - is this issue solved in the proposed knowledge-CLIP model? Some existing work that combines text and KG (e.g. https://arxiv.org/abs/2104.06378) has done closely-related analyses such as adding negation or changing entities in text to see if the KG-augmented method can robustly handle them. It would be very interesting if the authors perform such analysis on the proposed knowledge-CLIP model that combines image, text and KGs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ps_2V7aRHK", "reviewer": "Reviewer_zySD", "summary": "This paper presents a vision-language pre-training framework that incorporates knowledge information by pre-training on multiple knowledge graph datasets (i.e., VisualSem, Visual Genome, and ConceptNet).\nThey unify all the possible triples of different modalities (e.g., image-relation-text, image-relation-image, text-relation-text), encode each modality with its corresponding CLIP encoder (i.e., Image encoder, Text encoder), and concatenate all the embeddings with special token <head>.\nThen, multi-modal encoder (i.e., 4 layer transformer model) encodes it and uses embedding of <head> token as a representation.\nFor the training objectives, the paper utilizes three different objectives including entity-entity, entity-relation, and graph-entity loss.\nThe pre-trained model consistently outperforms baselines including UNITER, OSCAR, CLIP, ALBEF on various multi-modal tasks.", "questions": "1. Figure 2: L_E2L -> L_E2R ?", "limitations": "I really like the approach but the lack of analysis is the limitation of this paper.\n\nI am willing to increase the score when more analysis about training objectives are provided.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. It is interesting that how framework encodes multiple features of different modalities into one representation.\n2. The performance shows the effectiveness of the approach.\n\n### Weakness\n\n1. Lack of analysis\n    - Want to see the performance with each training objective for understanding the effectiveness of each objective. (e.g., without L_E2E, without L_E2R, without L_G2E)\n    - Want to see the impact of L_KD, which is the loss term of KL distance between original CLIP and the proposed model.\n\n2. Lack of intuition for each objective\n    - E2E: Want to see the intuition behind exploiting contrastive learning between (head, relation)-masked triple and (tail)-masked triple.\n    - G2E: Want to see the intuition behind applying contrastive learning between before and after graph-propagation on (tail).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "i4PIH80GgJJ", "reviewer": "Reviewer_WaXR", "summary": "In order to address the semantic alignment issue in multimodal tasks, this paper suggests the Knowledge-CLIP framework. Experiments reveal that the proposed approach is practical.\n", "questions": "Please see and answer \"Strengths and Weaknesses\" for the issues I mentioned. Additionally, the reviewer suggests the authors take the time to consider why Multimodal Pre-training requires Graph (or knowledge).  The authors claimed that focusing on the graph can help semantic alignment at first, but the second half of the paper, just shows that continue learning with graph will have some improvement to CLIP.\n", "limitations": "I recommend the authors give more robust explanations. Besides, this article still has many self-justifying explanations so far, and I hope the author will give more clear and reasonable illustrations. Overall, this article does a lot of experimenting, which is why I gave it a original POSITIVE rating. However, the article's flaws are really obvious, and I hope that the author will thoroughly improve them based on the comments. I will further check the rebuttal.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(a) Model\n+ The use of \"graphs\" for multimodal pre-training is practical. In essence, the model benefits from having strong reasoning skills.\n- However, the details of the model are not well presented, including data processing, sample visualization, etc.\n\n(b) Experiments\n+ The experiments in the article are adequate and verify that the learning graph can help the model improve multimodal learning.\n- Which of these loss functions affects the results the most, and is it graph-based loss? If not, the model appears to be oriented east-west.\n- In Figure 1, because the template \"not a photo of ｛}\" is inherently misleading. In addition, the car below is between yellow and green in color. As a result, this is not a particularly good example to demonstrate what the author is trying to say.\n- There is not much evidence to demonstrate that the proposed methods can really help the model for modal alignment. It is difficult to verify this conclusion just by show only the SOTA results (without some newest SOTA baselines ) and lacking visual graphs/tables.\n- Except for VQA, there is no obvious advantage of the effect of the model in this paper.\n\n(c) writing\n+ The reviewer likes the motivation in the Introduction section.\n- There aren't enough ablation experiments to confirm that every design is rational. \n- Many earlier works, such as VLPs in e-commerce, also address the issue of alignment from different techniques.  The \"co-occurence\" is the key factor that is the same with this article (no matter the word-patch alignment, or using RoI-tags). The paper's references, however, are insufficient.\n\n(d) others\n- The SOTA performance was obtained by ignoring a large number of SOTA models. The authors  should give a justifiable explanation.\n\n\nUpdate after rebuttal\n\nThanks for your detailed response. I stick to my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G2sZfx_Ng9u", "reviewer": "Reviewer_7sfH", "summary": "The paper extends the Contrastive Language-Image Pre-training (CLIP) model with a knowledge-based pre-training framework. The framework introduces knowledge-based objectives in the pre-training process and utilizes diverse types of knowledge graphs as training data.\n\nContributions:\n- A framework to utilize knowledge graph to vision-language pretraining\n- Continued pretraining to save computation resources\n- Improved performance upon CLIP on vision-language multi-modal tasks as well as uni-modal tasks.", "questions": "- Equation 13 is confusing to me. Does it have a simple but meaningless solution that G^L(t) = Y(-,-,t)?\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The framework is intuitive and clearly presented.\n- The improvement of the CLIP model is significant and promising.\n\nWeakness:\n- The lack of proper ablation study. For the vision-language or uni-modal tasks in the evaluation, not much reasoning is required to perform the task. So, it is important to answer the question: if the improvement in Knowledge-CLIP comes from the knowledge-based objective or just from more training data. An ablated baseline would be CLIP continued pretrained on the same datasets which Knowledge-CLIP was pretrained.\n- Another ablation study should be on the three objectives to measure their contributions to the final result.\n- The motivation in figure 1 is not accurate. The example only shows that CLIP fails to capture negation, not all semantics. From the same figure, we can see that CLIP performs well in differentiating cars from planes. Authors need to be more specific on their claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept