言語処理学会 第30回年次大会 発表論文集（2024年3月）

Improving the Image Discrimination Ability for CLIP-Model via
Semantic Graphs through Graph Convolutional Network
Sangmyeong Lee1 Seitaro Shinagawa1 Koichiro Yoshino1,2 Satoshi Nakamura1
1 Nara Institute of Science and Technology 2 Guardian Robot Project, RIKEN
lee.sangmyeong.lo3@is.naist.jp

Abstract
The Contrastive Language-Image Pre-training (CLIP)model is based on plain textual inputs, leading to a challenge in handling structural ambiguity residing inside atext. This paper examines the eﬀectiveness of semanticgraphs, the graph format representation converted fromsyntax trees, using a graph convolutional network (GCN)
as the CLIP model’s input to address this challenge. Additionally, we leverage the integrated gradient methodology
to analyse how semantic graphs are interpreted within themodel’s architecture.

1

Introduction

Contrastive Language Image Pre-training (CLIP) [1] hasdemonstrated its eﬀectiveness in Vision and Language(V&L) tasks like few-shot and zero-shot classiﬁcation,leveraging large text-image pair datasets, applied to models like Stable Diﬀusion [2], one of the state-of-the-art
text-to-image generation models.
However, CLIP model’s heavy dependence on plain textual inputs poses challenges in capturing semantic nuancesfrom structural information in input texts. Among thesechallenges, we focus on structural ambiguity, where a sentence can be interpreted in multiple syntactic ways. InFigure 1 (A), the position of the bag can diﬀer, either on achair or in the man’s arms, based on the syntactic structure.
This raises a concern for the CLIP model’s ability to clearly
discriminate ambiguous vision and language pairs to meetthe user’s intention.
In the realm of linguistics, structural information of language is expressed using linguistic formalism, a systematic representation of the lingual structure. Our previous
research [3] attempted to insert the syntax tree, the mostrepresentative type of formalism, into the CLIP model’sFigure 1(A) : Multiple meanings of the text “Danny approached the chair with a bag.” based on the syntactic structure,(B) : Textualised syntax tree from the same input text, (C) : Semantic graph representing the same meaning with (B).

text encoder by simply treating it as a text consisting ofwords, POS tags, and brackets (Figure 1 (B)), outperforming the conventional CLIP model in image discrimination.
While successful, textualised syntax trees had the following problems:• With additional brackets and POS tags, the syntax
tree’s sequence length had potential threats to exceed
the CLIP model’s limitation.
• Tokenisation was performed in an unintentional manner, leading to wrong inference.
In this paper, we try another formalism called semanticgraphs (Figure 1 (C)). Semantic graphs abstract away thecore meaning of a sentence by representation of predicates(e.g. verbs) and arguments within (nouns). We expect thesemantic graphs to be free from the problems above hence
leading to better discrimination performance. We try out a― 2052 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Graph Convolutional Network (GCN) [4] as the encoding
strategy and undergo the analysis of the model’s inference
principle.

2

Related Studies

2.1

Limitations of the CLIP model’s Plain
Textual Inputs

While our research isn’t directly related with this aspect,
the CLIP model’s ability to correctly associate images withthe right arrangements of words, known as Visio-Linguistic
Compositionality, is reported to be poor in the current CLIPmodel [5]. As a response, there have been attempts todeal with this challenge by leveraging scene graphs, whichshows the visual structural information between the objects
appearing inside the image [6, 7]. Our research is diﬀerent
from these research in following points:

Figure 2 Overview of the proposal. The sentence is transformed into a semantic graph and then encoded through the GCNencoder.

acquire characteristic vectors from the graph. We combine
these vectors with the text vectors independently acquired
from the conventional CLIP’s text encoder, through concatenation and linear projection. As the graph parser, weemploy HanLP [12], and the pre-trained CLIP vision and• Ambiguity is a subject beyond compositionality. Even

text encoders are from Huggingface1）.

if the compositionality is solved, disambiguation re-Our GCN is based on Morris et al.’s work [13], wheremains a challenge.

a single node vector undergoes an update according to the• While scene graph processing is usually done by inlation, our graph methodology considers the overall

formula (from the kth layer to the (k+1)th):∑()𝑥 𝑖𝑘+1 = 𝑊1 𝑥𝑖𝑘 + 𝑊2
𝑒 𝑗,𝑖 · 𝑥 𝑗

ﬂow through nodes and edges.

This formulation involves two essential weight matrices:• Instead of visual structural information, ours employ

𝑊1 for the self-recurrent computation and 𝑊2 for managing

dividually encoding partial triplets of objects and re-the linguistic structural information.

2.2

(1)

𝑗 ∈ 𝑁 𝑒𝑖𝑔ℎ𝑏𝑜𝑢𝑟 𝑠 (𝑖)the weights of neighbouring edges from the node (𝑒 𝑗,𝑖 ).

GCN for Natural Language Processing

GCN has been applied in the ﬁeld of Natural Language Processing by encoding graph-structured linguistic

4Experiments
There are two research questions our experiments aims

to observe:

formalisms. Examples include syntax trees based on de-• Can integrating semantic graphs into the CLIP model

pendency grammar [8, 9], semantic graphs [10, 11], etc.

enhance its ability to discriminate vision and language

Our proposal is profoundly based on this idea, by con-pairs with structural ambiguity? (Section 4.1)

structing a semantic graph from a sentence then passing it• Does leveraging semantic graphs result in sound dis-through GCN.

crimination quality even in general scenarios irrele-3

vant to ambiguity? (Section 4.2)

Proposed Method
The overview of our proposal is illustrated in Figure 2.

We compare three models as follows:

Compared with the conventional CLIP model, the proposal• CLIPplain : Conventional CLIP model with plain text

has two key distinctions. First, during the ﬁne-tuning pro-

inputs, which is our baseline.

cess, we keep the CLIP model’s vision component param-• CLIPtree as text : Conventional CLIP model architecture ﬁne-tuned with linearised textual syntax trees aseters ﬁxed, since the extensively pre-trained vision encoder

inputs[3], which is the other baseline.

is expected to yield high performance without extra training. Second, out model presumes a graph parser transform-• CLIPGCN : GCN applied to a semantic graph, which

ing the textual input into the graph, and a GCN network tois our proposal.
1） https://huggingface.co/openai/clip-vit-base-patch32― 2053 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

text-to-image
DataLAVA

Model

Accuracy (%)

CLIPplain

50.0

0.264

0.708

CLIPtree as text

74.72

0.405

0.753

0.882

0.388

0.803

0.899

CLIPGCN

77.53

0.298

0.657

0.775

0.365

0.657

0.775

CLIPplain

NA

0.391

0.654

0.765

0.408

0.680

0.787

NANA

0.371

0.645

0.761

0.410

0.683

0.795

0.198

0.477

0.617

0.234

0.519

0.661

COCO CLIPtree as text
CLIPGCN
Table 1

4.1

image-to-text

Recall@1 Recall@5 Recall@10 Recall@1 Recall@5 Recall@100.905

0.270

0.753

0.893

Experimental results divided based on datasets. CLIPplain and CLIPtree as text are the baselines.

Structural Disambiguation Experiment

dataset equipped with a substantial number of vision andlanguage pairs depicting various situations and contexts.

The focus of this experiment is to leverage the semantic graphs to accurately discriminate vision and language

This dataset has no language data other than plain texts,
hence we use HanLP parser mentioned in Section 3. Aspairs with structural ambiguity. As a dataset, we employ

for the metrics, we use Recall@K, but not the discrimi-Language And Vision Ambiguities (LAVA) corpus [14],nation accuracy since this experiment has little to do withwhich consists of structurally ambiguous texts with twoambiguity.

possible interpretations, two corresponding images, two5corresponding syntax trees, and two corresponding seman-Experimental Results

already oﬀered in the corpus, we don’t use the parser pre-5.1 Structural Disambiguation Experimental Results

viously mentioned in this experiment. Evaluation metrics

The upper part in Table 1 shows the disambiguation ex-

tic parsed information. Since linguistic formalisms are

perimental results on LAVA corpus. In assessing discrim-

we employ are as follows:
Discrimination Accuracy

evaluates the model’sination accuracy, our proposal outperformed the baselines,

ability to successfully discriminate ambiguous vision andwith 77.53 in accuracy, suggesting the proposal’s eﬀec-language pairs. At every time step, the model is given
two vision and language pairs sharing the same plain text

tiveness. On the other hand, in Recall@K, CLIPtree as text

but with diﬀerent interpretations from structural ambigu-shows largely the best results. And except for the casewhere K equals 1, our proposal is even outperformed byity. We count the correctly matched pairs, and the accuracy

the CLIPplain .

is computed as the ratio of the number of correct matches

5.2 Generality Evaluation Experimental
Resultsover the total data pairs.
Recall@K

evaluates the model’s performance in theThe bottom part in Table 1 shows the generality evalu-

context of image retrieval, where we search for the righttion results on COCO dataset. In this section, our proposal

pair for the input from all the test data. With the values ofshowed the worst performance. While there has been a

K set as 1, 5, and 10 in advance, for each K, we count thedecline across almost every aspect compared with LAVA,

number of inputs for which the correct match was foundour proposal showed the worst decline. One notable thing

within the top K search results. We evaluate using thisis the performance of CLIPtree as text , of which the perfor-

metric bidirectionally—text-to-image and image-to-text.

mance decline is less severe than that of our proposal.

4.2

Also, CLIPtree as text shows the best performance in image-

Generality Evaluation Experimentto-text direction. Considering as well the CLIPtree as text ’s

The objective of this experiment is to assess the model’s
performance in diverse scenarios, aiming to determine its

best performance across the Recall@K in Section 5.1, one
could speculate this performance owing to the model’sability to generalize across various contexts without be-preservation of the conventional CLIP model’s pre-traineding excessively tailored to speciﬁc datasets, such as LAVA.

structure.

To explore this question, we use Microsoft COCO [15], aOverall, our proposal showed the best performance in― 2054 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 3 Discriminating examples where the scenario featuresa single person. Both graphs are generated from the same sentence, presented in the bottom. The numbers on the red arrows
indicate the model’s inference probability, and nodes enclosed
in red circles highlight the actual diﬀerences between the topand bottom graphs. Two diﬀerent arrows represent the model’s
matching and the actual right pair. In this case, both inferences
are successful, with the top graph of the high probability and thebottom graph of the low probability.

Figure 4 Discriminating examples where the scenario features
two people. The Figure structure is as same as that of Figure 3.
Both graphs are matched to the right images.

tence “Danny looked at Andrei having a yellow chair.”,the diﬀerence in meaning is ‘who’ has the bag, Andrei
or Danny. As the LAVA corpus assigns distinct personalnames, treating them as proper nouns instead of providing
general absolute characteristics like appearance descrip-

discrimination accuracy but the lowest in generality. This

tions, it becomes challenging to distinguish individuals,

discrepancy in performance of the proposal necessitatessuch as determining who is Andrei and who is Danny, evenfurther analysis.

with the assistance of semantic graphs. Consequently, in6

scenarios involving more than two people, the model be-Analysis

came excessively tailored to precisely diﬀerentiate between

For our GCN proposal model, we employed the Inte-these pairs. In Figure 4, while the matching is successful

grated Gradients method [16], which involves interpolating between the input and an empty zero baseline input ofwith high inference probability, the upper example doesn’tshow the model’s attention given to the right part to focus

the same size, accumulating gradient values for each edge.

on. Throughout the text examples, the model usually chose

The focus of our analysis is to see if the model’s attention

an extreme strategy to focus on every edge in one pair andis given to the adequate parts of the graph in disambigua-

pay attention to random few edges in the other. Overall, ittion using the LAVA corpus. The diﬀerence between two

could be said that while leveraging semantic graphs showed

diﬀerent graphs from the same sentence is enclosed within

successful discrimination performance both in quantitative

red circles in Figure 3 and 4.

and qualitative manners for certain cases, innate noise in-Figure 3 shows the successful discrimination examples

cluded in the data hindered the model’s correct inference,

where the scenario covers a single person. The diﬀerence

calling out a need for a more robust dataset.

in graphs appear in meaning as the position of the bag,

7Conclusion

whether on the chair on in the man’s arms. In Figure 3
it is evident that the model is concentrating on the rightThis paper introduces the integration of semantic graphs

part hence leading to successful matching. Additionally,

into the CLIP model to enhance its discrimination perfor-

it is notable that the bottom example shows less attention
which is demonstrated by the less thickness of edges, given

mance for disambiguation. Our experiments demonstratecompared with that given to the upper example, and the in-superior accuracy, highlighting the proposal’s strength.
However, limitations, such as the model’s generality andference probability is signiﬁcantly lower (0.546). This sug-

potential impact of noisy parsed linguistic formalism, re-gests the attribution of semantic graph in its right intention

main unexplored. The exclusive focus on CLIP raises ques-in discrimination tasks.

tions about the generalisability of our ﬁndings to other Vi-

However, there was a problem observed in scenariossion and Language models. Future research will address

covering two people as shown in Figure 4. In the sen-these limitations to further reﬁne and extend our approach.

― 2055 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgement
This work was supported by JSPS KAKENHI Grant[13]Numbers 21K17806.

References[14][1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
Krueger, and Ilya Sutskever. Learning transferable visualmodels from natural language supervision. In ICML, 2021.
[2] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick
Esser, and Björn Ommer. High-resolution image synthesis
with latent diﬀusion models. In CVPR, pp. 10674–10685,
2021.
[3] 李 相 明, 品 川 政 太 朗, 中 村 哲. Clip に お け る テ キ
ストの構文情報理解による画像識別能力の向上.
画像の認識・理解シンポジウム. [Lee Sangmyeong,
Seitaro Shinagawa, and Satoshi Nakamura (2023). Improving Image Discrimination Ability through Understandingof Textual Syntactic Information in CLIP. Meeting on Image Recognition and Understanding (MIRU)], 7 2023.
[4] Thomas Kipf and Max Welling. Semi-supervised classiﬁcation with graph convolutional networks. ArXiv, Vol.
abs/1609.02907, , 2016.
[5] Yutaro Yamada, Yingtian Tang, and Ilker Yildirim. Whenare lemons purple? the concept association bias of clip.
ArXiv, Vol. abs/2212.12043, , 2022.
[6] Yufen Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang,Xinfeng Zhang, Weijie Chen, Zeng Zhao, Tangjie Lv,Zhipeng Hu, and Wen Zhang. Structure-clip: Enhance multi-modal language representations with structureknowledge. ArXiv, Vol. abs/2305.06152, , 2023.
[7] Harman Singh, Pengchuan Zhang, Qifan Wang,Mengjiao MJ Wang, Wenhan Xiong, Jingfei Du, andYu Chen. Coarse-to-ﬁne contrastive learning in imagetext-graph space for improved vision-language compositionality. ArXiv, Vol. abs/2305.13812, , 2023.
[8] Diego Marcheggiani and Ivan Titov. Encoding sentenceswith graph convolutional networks for semantic role labeling. In EMNLP, 2017.
[9] Emanuele Bastianelli, Andrea Vanzo, and Oliver Lemon.
Encoding syntactic constituency paths for frame-semantic
parsing with graph convolutional networks. ArXiv, Vol.
abs/2011.13210, , 2020.
[10] He Cao and Dongyan Zhao. Leveraging denoised abstract
meaning representation for grammatical error correction.
In ACL, 2023.
[11] Weiwen Xu, Huihui Zhang, Deng Cai, and Wai Lam. Dynamic semantic graph construction and reasoning for explainable multi-hop science question answering. ArXiv,Vol. abs/2105.11776, , 2021.
[12] Han He and Jinho D. Choi. The stem cell hypothesis:
Dilemma behind multi-task learning with transformer encoders. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, EMNLP, pp.
5555–5577, Online and Punta Cana, Dominican Republic,[15][16]― 2056 ―November 2021. Association for Computational Linguistics.
Christopher Morris, Martin Ritzert, Matthias Fey,William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural:
Higher-order graph neural networks. In AAAI, 2018.
Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz,and Shimon Ullman. Do you see what i mean? visualresolution of linguistic ambiguities. In EMNLP, 2015.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James
Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, andC. Lawrence Zitnick. Microsoft coco: Common objects incontext. In ECCV, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).