PolCLIP: A Unified Image-Text Word Sense Disambiguation Model via Generating Multimodal Complementary Representations (2024)

Abstract: Word sense disambiguation (WSD) can be viewed as two subtasks: textual word sense disambiguation (Textual-WSD) and visual word sense disambiguation (Visual-WSD). They aim to identify the most semantically relevant senses or images to a given context containing ambiguous target words. However, existing WSD models seldom address these two subtasks jointly due to lack of images in TextualWSD datasets or lack of senses in VisualWSD datasets. To bridge this gap, we propose PolCLIP, a unified image-text WSD model. By employing an image-text complementarity strategy, it not only simulates stable diffusion models to generate implicit visual representations for word senses but also simulates image captioning models to provide implicit textual representations for images. Additionally, a disambiguation-oriented image-sense dataset is constructed for the training objective of learning multimodal polysemy representations. To the best of our knowledge, PolCLIP is the first model that can cope with both Textual-WSD and Visual-WSD. Extensive experimental results on benchmarks demonstrate the effectiveness of our method, achieving a 2.53% F1score increase over the state-of-the-art models on Textual-WSD and a 2.22% HR@1 improvement on Visual-WSD.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10676–10690 August 11-16, 2024 ©2024 Association for Computational Linguistics
Word sense disambiguation (WSD) can be viewed as two subtasks: textual word sense disambiguation (Textual-WSD) and visual word sense disambiguation (Visual-WSD). They aim to identify the most semantically relevant senses or images to a given context containing ambiguous target words. However, existing WSD models seldom address these two subtasks jointly due to lack of images in TextualWSD datasets or lack of senses in VisualWSD datasets. To bridge this gap, we propose PolCLIP, a unified image-text WSD model. By employing an image-text complementarity strategy, it not only simulates stable diffusion models to generate implicit visual representations for word senses but also simulates image captioning models to provide implicit textual representations for images. Additionally, a disambiguation-oriented image-sense dataset is constructed for the training objective of learning multimodal polysemy representations. To the best of our knowledge, PolCLIP is the first model that can cope with both Textual-WSD and Visual-WSD. Extensive experimental results on benchmarks demonstrate the effectiveness of our method, achieving a 2.53% F1score increase over the state-of-the-art models on Textual-WSD and a 2.22% HR@1 improvement on Visual-WSD.

heading: 1 Introduction
text: Understanding and identifying the intended meaning of words with multiple senses (i.e., polysemy) is a significant challenge in natural language processing (Navigli, 2009). This promotes in-depth research on word sense disambiguation (WSD), which has recently been extended to multimodal downstream tasks (Bevilacqua et al., 2021). Techniques for WSD are critical for enhancing the accuracy and effectiveness of text understanding and
*Corresponding author.
information retrieval tasks such as machine translation (Raganato et al., 2019), image-text retrieval (Chen et al., 2020), and large language model inference (Kritharoula et al., 2023).
Theoretically, WSD can be divided into two subtasks: textual word sense disambiguation (TextualWSD) (Bevilacqua et al., 2021) and visual word sense disambiguation (Visual-WSD) (Raganato et al., 2023). Given a context containing an ambiguous target word, the goal of Textual-WSD is to select the most semantically appropriate one from a set of candidate senses, while the goal of VisualWSD is to choose the most semantically suitable one from a set of candidate images. Due to the distinct modalities, these two subtasks typically require specialized training datasets and methods (Bevilacqua and Navigli, 2020; Blevins and Zettlemoyer, 2020; Kwon et al., 2023). Nevertheless, they can be unified as a Multimodal-WSD task if the senses and images in existing WSD datasets are aligned, as shown in Figure 1. The task objectives of Multimodal-WSD cover: 1) in cases where candidates include both senses and images, a Multimodal-WSD model is required to identify the most semantically relevant senses and images simultaneously for ambiguous target words within a given context; 2) in cases where candidates include either senses or images, the model is still
10676
required to identify the most semantically relevant senses or images for ambiguous target words within a given context. Such task objectives are also in line with the practical scenarios of current large visionlanguage models (Liu et al., 2024). Technically, developing a generic Multimodal-WSD model can realize the unification of WSD tasks and activate the potential of multimodal applications in understanding polysemy knowledge.
In the Textual-WSD datasets (Raganato et al., 2017), merely textual senses serve as candidates (i.e., image-missing), while in the Visual-WSD datasets (Raganato et al., 2023), only images serve as candidates (i.e., sense-missing). This results in the challenge of modality missing at the data level, limiting the unification of these two WSD subtasks. Furthermore, multimodal representations have been demonstrated to carry richer semantic information compared to unimodal representations in recent WSD works (Gella et al., 2016, 2019). However, constrained by model architecture, existing Textual-WSD models (Conia and Navigli, 2021; Maru et al., 2019; Huang et al., 2019) cannot supplement candidate senses with image information, and Visual-WSD models (Yang et al., 2023; Zhang et al., 2023; Dadas, 2023) cannot supplement candidate images with descriptions. This poses technical difficulties in developing a unified framework for Multimodal-WSD.
To address these issues, we propose PolCLIP, a unified image-text WSD model which is proficient in multimodal polysemy processing and is built upon CLIP (Radford et al., 2021) architecture. By employing an image-text complementarity strategy, it simulates stable diffusion models (Ho et al., 2020) (generating images based on texts) and image captioning models (Ramos et al., 2023) (generating descriptions based on images). The core idea of this strategy is to make PolCLIP initially focus on the key information of original unimodal senses or images, and then re-utilize the text or image encoder to generate implicit imagetext complementary representations. Two widely used WSD datasets (SemCor (Miller et al., 1993) and VWSD-KB (Yang et al., 2023)) are integrated into a disambiguation-oriented image-sense dataset for the training objective of learning aligned multimodal representations. Moreover, a fine-tuned GPT-3.5 model is utilized to generate lexical definitions for semantic enhancement in testing phase. The main contributions of this work are summarized as follows:
• A unified image-text WSD model is proposed, which is the first model to jointly cope with Textual-WSD lacking images and VisualWSD lacking senses.
• An image-text complementarity strategy is introduced to simulate stable diffusion models and image captioning models for addressing the modality missing issues in unimodal WSD datasets.
• A disambiguation-oriented image-sense dataset is constructed to provide a benchmark for the Multimodal-WSD task.

heading: 2 Related Work
text: Textual-WSD was mainly tackled by knowledgebased methods and supervised methods (Bevilacqua et al., 2021). Knowledge-based methods (Maru et al., 2019; Scozzafava et al., 2020) typically used external dictionary resources to provide sense lists for ambiguous words to resolve polysemy. Supervised methods (Huang et al., 2019; Wang and Wang, 2020) generally used pre-training language models to maximize the similarity probabilities between contexts and candidate senses in a feature space. BEM (Blevins and Zettlemoyer, 2020) and SACE (Wang and Wang, 2021) adopted biencoders and only retained the representations corresponding to ambiguous words. They achieved state-of-the-art results on English all-words benchmarks at that time. Moreover, the full utilization of visual features for verb sense disambiguation has attracted increasing interest (Gella et al., 2016, 2019). EViLBERT (Calabrese et al., 2020b) obtained better results by learning task-agnostic multimodal sense representations, compared to methods built solely on language models. Although these methods primarily leveraged visual information to bolster performance on Textual-WSD, they could not be applied to Visual-WSD straightforwardly.
Visual-WSD was introduced in SemEval-2023 Task 1 (Raganato et al., 2023). The VisualWSD mainstream approaches employed VisionLanguage Pre-training models (VLPs) for imagetext retrieval. FCLL (Yang et al., 2023) proposed a fine-grained image-text contrastive learning mechanism and won first place in SemEval-2023 Task 1. Moreover, large language models (LLMs) were widely used to enrich the semantic information of contexts (Ghahroodi et al., 2023; Yang et al., 2024). Calling APIs was a commonly adopted strategy, where simple prompts were designed to
guide LLMs to return the interpretations of ambiguous target words in contexts (Kritharoula et al., 2023). However, Visual-WSD models depended on the prior knowledge of VLPs, which were pretrained with objectives biased towards image-text understanding rather than WSD. This resulted in Visual-WSD models struggling to effectively resolve Textual-WSD.
Data was a key factor to unify these two WSD subtasks and develop a generic Multimodal-WSD model. WordNet (Miller et al., 1990) was a large lexicographic database and a standard inventory for English WSD. It contained approximately 120,000 synsets. BabelNet (Navigli and Ponzetto, 2010; Navigli et al., 2021) was the most popular multilingual dictionary, which was semi-automatically mapped to other resources to acquire encyclopedic terms. It covered over 500 languages and was upgraded to version 5.3 recently. Researchers could access various possible resources about ambiguous words by BabelNet, including example sentences, parts of speech, textual senses, and images. By linking WordNet with Wikipedia through BabelNet, BabelPic (Calabrese et al., 2020a) expanded non-concrete image-sense pairs, paving the way for our work to construct larger disambiguationoriented multimodal datasets.
Inspired by FCLL, we used a dual-stream architecture with contrastive learning to align the knowledge of word-sense pairs and word-image pairs from the proposed disambiguation-oriented dataset. Previous research viewed LLMs as common tools for text semantic enhancement. Thus, LLMs were also adopted in our testing phase.

heading: 3 Method

heading: 3.1 Task formulation
text: Textual-WSD and Visual-WSD can be unified as a Multimodal-WSD task which is a token classification problem. A given context c generally contains at least one ambiguous target word wtarget. For Textual-WSD, there is a set of word senses S = {s1, s2, . . . , ŝ, . . . , sn} as candidates, where ŝ denotes the most semantically relevant sense to wtarget. Following Eq. 1, a Textual-WSD model is required to learn a similarity function F to retrieve ŝ from candidate senses. For Visual-WSD, there is a group of images I = { i1, i2, . . . , î, . . . , in } as
candidates, where î represents the most semantically relevant image to wtarget. Following Eq. 2, a Visual-WSD model is required to learn a similarity function F to retrieve î from candidate images.
ŝ = argmaxF ( c, wtarget, S ) (1)
î = argmaxF ( c, wtarget, I ) (2)

heading: 3.2 The PolCLIP model
text: The framework of the PolCLIP model is shown in Figure 2. It utilizes an image-text complementarity strategy and is built upon CLIP architecture. It employs 12-layer transformers as the text encoder and 24-layer visual transformers as the image encoder. A context with an ambiguous target word wtarget is input into the text encoder to generate a complete context representation ec = {[CLS], ew1 , . . . , ewtarget , . . . , ewn , [SEP ]}, where ewtarget is the representation corresponding
to wtarget. The candidate senses of wtarget are input into the text encoder to output a sense vector vsense. The candidate images of wtarget are fed into the image encoder to output an image vector vimage.
A Sense-to-Image Generation (SIG) module and an Image-to-Sense Generation (ISG) module are designed to generate implicit image-text complementary information, which is expressed in vectors. Specifically, the SIG module consists of a shared 2-layer self-attention module, a sense-toimage projector, and the last four layers of the image encoder. The sense vector vsense is input to SIG. Its key information is condensed through the self-attention module and the linear-layer projector. After that, the highly compressed sense information is transformed into a sense-generated image vector vsense-genimage with visual knowledge through the last four layers of the image encoder. Similarly, the ISG module is composed of the same shared 2-layer self-attention module, an image-to-sense projector, and the last four layers of the text encoder. The image vector vimage is input to ISG and then converted into an image-generated sense vector vimage-gensense with textual knowledge.
In order to make the generated implicit imagetext complementary information actually beneficial to semantic augment, following ALBEF (Li et al., 2021), a text momentum encoder and an image momentum encoder are employed to generate pseudo-target vectors. Specifically, the candidate senses and images from the same batch are input separately into the text and image momentum encoders to output a pseudo-target sense vector vpseudosense and a pseudo-target image vector v
pseudo image . These two pseudo-target vectors supervise
v image-gen sense and v sense-gen image to be close to ground truth. The text and image momentum encoders retain the prior knowledge of the backbone model to counteract the issue of catastrophic forgetting (Li et al., 2023). Therefore, the pseudo-target vectors gain improvement continuously with these two momentum encoders being optimized at a small pace. The similarity between vimage-gensense and v pseudo sense is calculated by Eq. 3-4. Also, the similarity between v
sense-gen image and v pseudo image is calculated by Eq. 5-6. s is a similarity function. PISG and PSIG are the softmax-normalized similarities used to supervise the ISG module and the SIG module.
s (Sgen , Spse ) = vimage-gensense · ( vpseudosense )T (3)
PISG = exp (s (S gen , Spse ))
∑N n=1 exp (s (S gen , Spse )) (4)
s (Igen , Ipse ) = v sense-gen image ·
( v
pseudo image
)T (5)
PSIG = exp (s (I gen , Ipse ))
∑N n=1 exp (s (I gen , Ipse )) (6)
A shared 4-layer cross-attention module serves as a fusion module. It integrates the original unimodal sense/image representations and the generated implicit image/sense representations into semantically enriched multimodal representations. vsense serves as Q and v sense-gen image serves as K and V . They are fed into the fusion module and then a sense-guided multimodal vector vsense-guimulti is calcu-
lated by softmax ( QKT√ dk V ) , where dk denotes the dimension of 768. This vsense-guimulti achieves an effective interaction of the original sense representation with the implicit sense-generated image representation. Similarly, vimage serves as Q and v image-gen sense serves as K and V . They are fed into the fusion module and then an image-guided multimodal vector vimage-guimulti is output by the same cross-attention calculation process. This vimage-guimulti achieves an effective interaction of the original image representation with the implicit image-generated sense representation.
To avoid the key information of the ambiguous target word wtarget being smoothed out, we directly select the representation of wtarget as the anchor vector for retrieval, instead of simply averaging the complete context representation ec or taking [CLS]. Following Eq. 7-8, this anchor ewtarget is used to calculate the similarity with the senseguided multimodal vector vsense-guimulti . The similarity between the anchor ewtarget and the image-guided multimodal vector vimage-guimulti is calculated by Eq. 9- 10. PW2S and PW2I are the softmax-normalized anchor-to-sense similarity and the anchor-to-image similarity. The PolCLIP model can identify the most semantically appropriate senses and images based on these two similarities.
s(W,S) = ewtarget · ( v
sense-gui multi
)T (7)
PW2S = exp(s(W,S))∑N n=1 exp(s(W,S))
(8)
s(W, I) = ewtarget · ( v
image-gui multi
)T (9)
PW2I = exp(s(W, I))∑N n=1 exp(s(W, I))
(10)
Algorithm 1: Pseudocode of Training PolCLIP data : a context c with an ambiguous target word wtarget;
the candidate senses S and the candidate images I;
1 while c, wtarget , S, I do 2 ec ← Text_Encoder(c); # the complete context representations 3 ewtarget ← ec; # the anchor vector based on wtarget 4 # the original unimodal sense/image representations 5 vsense ← Text_Encoder(S); 6 vimage ← Image_Encoder(I); 7 # the generated implicit image/sense representations 8 vsense-genimage ← SIG(vsense); 9 vimage-gensense ← ISG(vimage);
10 # the semantically enriched multimodal representations 11 vsense-guimulti ← Fusion(vsense , v sense-gen image ); 12 vimage-guimulti ← Fusion(vimage , v image-gen sense ); 13 # the anchor-to-sense and anchor-to-image similarities 14 sim(W2S)← ewtarget · (vsense-guimulti ) T ; 15 sim(W2I)← ewtarget · (viamge-guimulti ) T ; 16 17 # the pseudo sense/image representations 18 vpseudosense ← Text_Momentum_Encoder(S); 19 vpseudoimage ← Image_Momentum_Encoder(I); 20 # the SIG and IGS similarities 21 sim(SIG)← vsense-genimage · (v pseudo image ) T ; 22 sim(ISG)← vimage-gensense · (vpseudosense )T ; 23 # the two generation-based loss 24 LSIG ← CrossEntropyLoss(sim(SIG), labels(SIG)); 25 LISG ← CrossEntropyLoss(sim(ISG), labels(ISG)); 26 # the two understanding-based loss 27 LW2S ← CrossEntropyLoss(sim(W2S), labels(W2S)); 28 LW2I ← CrossEntropyLoss(sim(W2I), labels(W2I)); 29 end
Four contrastive losses (Hadsell et al., 2006) are defined to optimize four training objectives jointly, comprising two generation-based objectives (SIG loss and ISG loss) and two understanding-based objectives (W2S loss and W2I loss).
The two generation-based objectives make the generated implicit image-text complementary information close to ground truth, to ensure that the enriched multimodal representations are semantically correct. The contrastive loss LSIG is defined as a cross-entropy H between the sense-generated image vector vsense-genimage and the pseudo-target image vector vpseudoimage :
LSIG = E(Igen,Ipse)∼DH ( YSIG,PSIG ) (11)
Y indicates the ground-truth multi-label one-hot similarity, where negative pairs have a probability of 0 and the positive pairs have a probability of 1. Similarly, the image-generated sense vector v image-gen sense and the pseudo-target sense vector v pseudo sense are used to calculate LISG:
LISG = E(Sgen,Spse)∼DH ( YISG,PISG ) (12)
The two understanding-based objectives ensure that the PolCLIP model accurately identifies the
semantically optimal senses and images. The anchor ewtarget and the sense-guided multimodal vector v
sense-gui multi are used to calculate LW2S :
LW2S = E(W,S)∼DH ( YW2S ,PW2S ) (13)
Also, the anchor ewtarget and the image-guided multimodal vector vimage-guimulti are used to calculate LW2I :
LW2I = E(W,I)∼DH ( YW2I ,PW2I ) (14)
Finally, the full training objective of PolCLIP is:
L = LSIG + LISG + LW2S + LW2I (15)
In summary, the entire training process is performed under the first objective of the MultimodalWSD task. The pesudocode of training the PolCLIP model is abstracted in Algorithm 1.

heading: 3.3 Inference of the PolCLIP model
text: The existing WSD test suites suffer from the modality missing issues, resulting in Multimodal-WSD models being evaluated separately under TextualWSD and Visual-WSD settings. Therefore, the inference process of the PolCLIP model is tested only under the second objective of the MultimodalWSD task.
To further stimulate the potential of the PolCLIP model in understanding polysemous text, a semantic enhancement is implemented for contexts during the inference procedure. Different to those methods that call APIs, we develop a disambiguation-oriented GPT-3.5 (D-GPT) to generate intended lexical definitions of a word in contexts. Fine-tuning on a random selection of 50,000 data from SemCor, D-GPT is developed based on the gpt-3.5-turbo-1106 model which is one of the latest fine-tunable GPT models released by OpenAI. Each piece of fine-tuning data consists of a query and an output. The query includes a context with a specified ambiguous word. The output is the ground-truth sense of the ambiguous word in the context. The fine-tuning experiment took approximately 175 minutes. More fine-tuning details are provided in Appendix A. After fine-tuning, DGPT generates lexical definitions based on queries. For example, a query “What does the word ‘bank’ mean in the context ‘They pulled the canoe up on the bank’?” is fed into D-GPT. Then, D-GPT possibly generates an interpretation “The word ‘bank’ refers to the side or edge of a river, lake, or other body of water”.
Algorithm 2: Pseudocode of PolCLIP Inference input : an augmented context c with an ambiguous target word wtarget;
the candidates Cand with senses or images; output : the ranked candidates Candranked ;
the semantically optimal sense or image Obest;
1 ec ← Text_Encoder(c); # the complete context representations 2 ewtarget ← ec; # the anchor vector based on wtarget 3 if only senses in Cand then 4 # for Textual-WSD 5 vsense ← Text_Encoder(Cand); 6 # the sense-generated image representations 7 vsense-genimage ← SIG(vsense); 8 # the sense-guided multimodal representations 9 vmulti ← Fusion(vsense , vsense-genimage );
10 else 11 # for Visual-WSD 12 vimage ← Image_Encoder(Cand); 13 # the image-generated sense representations 14 vimage-gensense ← ISG(vimage); 15 # the image-guided multimodal representations 16 vmulti ← Fusion(vimage , vimage-gensense ); 17 end 18 similarity ← ewtarget · (vmulti)T ; 19 Candranked ← topk(similarity); # k is the number of candidates 20 Obest ← arg max(Candranked); # the semantically optimal sense or image
During the testing phase, all the original contexts in WSD test sets are concatenated with the lexical definitions generated by D-GPT, to create semantically augmented contexts. For instance, an augmented context is “They pulled the canoe up on the bank. The word ‘bank’ refers to the side or edge of a river, lake, or other body of water”. These augmented contexts are subsequently fed into the PolCLIP model. The trained SIG and ISG modules support the PolCLIP model to address MultimodalWSD even when any modality is missing. The inference procedure of the PolCLIP model is abstracted in Algorithm 2.

heading: 3.4 Training data
text: The PolCLIP model relies on large-scale aligned image-sense pairs to learn multimodal polysemy knowledge. Thus, we construct a disambiguationoriented image-sense dataset by integrating SemCor (Miller et al., 1993) and VWSD-KB (Yang et al., 2023), to achieve the training objective of unified image-text WSD. SemCor is the most prevalent dataset for training Textual-WSD models. VWSDKB contains multimodal data such as words, senses, and images. The offline BabelNet v5.2 and BabelPic are used to collect relevant images to the senses in SemCor and VWSD-KB. The detailed construction process is provided in Appendix B.
After construction, all word senses in SemCor and VWSD-KB are aligned with at least one image and at most five images. We filter out the collected images that are pornographic, violent, or invalid
and conduct a manual validation to ensure there is no data leakage. The SemCor associated with images is called Image-Enhanced SemCor. Table 1 displays the statistical details of Image-Enhanced SemCor and VWSD-KB, comprising a total of 274,505 English instances (each instance includes a context with at least one ambiguous target word). An example of the disambiguation-oriented imagesense dataset is shown in Figure 1. Given a context “He had seen the Andromeda tree in Japan”, there are four candidate senses for the ambiguous target word “Andromeda”, and each sense corresponds to five images. The most semantically relevant sense and images to this context are Sense 4 and its five associated images.

heading: 4 Experiments and Results

heading: 4.1 Datasets
text: Due to the training objective of unified imagetext WSD, we opt not to use the validation sets from Textual-WSD or Visual-WSD. We allocate 80% of the combined Image-Enhanced SemCor and VWSD-KB datasets as the training set and reserve the remaining 20% as the validation set. XL-WSD (Pasini et al., 2021), an extra-large evaluation framework for Textual-WSD, is employed to evaluate the model performance on Textual-WSD. XL-WSD is widely used since it encompasses six English all-words Textual-WSD benchmark datasets, including SensEval-2 (SE2, (Palmer et al., 2001)), SensEval-3 (SE3, (Snyder and Palmer, 2004)), SemEval-2007 (SE7, (Navigli et al., 2007)), SemEval-2010 (SE10, (Agirre et al., 2010)), SemEval-2013 (SE13, (Navigli et al., 2013)), and SemEval-2015 (SE15, (Moro and Navigli, 2015)). These six benchmark datasets comprise a total of 8,517 English instances for testing. SemEval-2023 (SE23, (Raganato et al., 2023)) is used to assess the model performance on VisualWSD, as it is currently the most widely used VisualWSD benchmark containing 463 English instances.

heading: 4.2 Implementation details
text: Settings. Our model is implemented on Pytorch 2.0.1 and 4 RTX 4090 GPUs. Both the text encoder and image encoder are initialized by CLIPViT-L/14 (Radford et al., 2021). All parameters of the text encoder are optimized to preserve the maximum capability of the model to process ambiguous knowledge at the text level, while the image encoder is completely frozen to reduce the computational cost. The sense batch size is set to 50, the image batch size is set to 250. Following ALBEF (Li et al., 2021), the momentum is set to 0.005. AdamW is applied to optimize model parameters with a learning rate of 1e-04 and weight decay of 0.05. The image resolution is specified as 224×224, and the maximum text length is set to 77. The training process of the PolCLIP model requires approximately 2.5 GPU hours per epoch, with a total of 20 epochs. F1-score is used to evaluate the model performance on Textual-WSD. Hit Rate at 1 (HR@1, i.e., accuracy) and Mean Reciprocal Rank at 10 (MRR@10) are used to assess the model performance on Visual-WSD.
Baselines. We train PolCLIPbase using ImageEnhanced SemCor and PolCLIPlarge using the combination of Image-Enhanced SemCor and VWSD-KB. In the testing phase, both PolCLIPbase and PolCLIPlarge are integrated with D-GPT for semantic enhancement. Our model is compared
with recent state-of-the-art (SOTA) methods including (1) SOTA models in Textual-WSD: BEM (Blevins and Zettlemoyer, 2020), SACE (Wang and Wang, 2021) and Z-Reweighting (Su et al., 2022), (2) SOTA models in Visual-WSD: FCLL (Yang et al., 2023) and UVWSD (Kwon et al., 2023), (3) SOTA models in image-text learning tasks: Openai/CLIP-VIT-L/14 (Radford et al., 2021), Laion/CLIP-VIT-L/14 and Openai/CLIPVIT-H/14 (Schuhmann et al., 2022). More details about baseline models are provided in Appendix C. For a fair comparison, these baseline models are retrained using Image-Enhanced SemCor and VWSD-KB.

heading: 4.3 WSD results
text: The comparison results between PolCLIP and the baseline models on WSD benchmark test sets are shown in Table 2. The PolCLIP model achieves the state-of-the-art performance. SACE and FCLL are the second best models for Textual-WSD and Visual-WSD, respectively. Without D-GPT, PolCLIPlarge reaches an F1-score of 82.60% on all the Textual-WSD test data, which is 1.51% higher than SACE. It attains an HR@1 of 82.28% and a MRR@10 of 87.98% on Visual-WSD, which are 0.91% and 0.29% higher than FCLL respectively. With D-GPT, the performance of both PolCLIPbase and PolCLIPlarge is enhanced thanks to the semantically augmented contexts with lexical definitions.
In this situation, PolCLIPlarge gains an F1-score of 83.62% on Textual-WSD, which is 2.53% higher than SACE. It attains an HR@1 of 83.59% and a MRR@10 of 90.07% on Visual-WSD, which are 2.22% and 2.38% higher than FCLL respectively.
Specifically, without training, Openai/CLIP-VITL/14, Laion/CLIP-VIT-L/14, and Openai/CLIPVIT-H/14 (collectively called CLIPs) obtain zeroshot F1-scores below 50% on all the Textual-WSD test data, due to the pre-training data and goal of CLIPs do not target WSD. Conversely, UVWSD obtains over 80% zero-shot HR@1 on Visual-WSD. While only using the Image-Enhanced SemCor as the training set, PolCLIPbase outperforms CLIPs, BEM, SACE, and Z-Reweighting on Textual-WSD, even though its performance on Visual-WSD is slightly inferior to FCLL. When the combination of Image-Enhanced SemCor and VWSD-KB is used as the training set, PolCLIPlarge shows further improvement over PolCLIPbase and surpasses all baseline models. Additionally, the effectiveness of the disambiguation-oriented image-sense dataset is proven, with the performance of all the baseline models on WSD benchmarks being bolstered.

heading: 4.4 Ablation study
text: An ablation study is conducted to reveal the contribution of each module and the results are reported in Table 3. For the two generation-based training objectives, the Sense-to-Image Generation module is removed first, which corresponds to LSIG. In this scenario, the F1-score of PolCLIPlarge on Textual-WSD decreases by 8.08% and the HR@1 on Visual-WSD drops by 1.07%. This indicates that the implicit image information generated by the SIG module aids PolCLIPlarge in acquiring enriched multimodal representations. Secondly, the Image-to-Sense Generation module is removed, which corresponds to LISG. The HR@1 of PolCLIPlarge on Visual-WSD decreases by 5.82% and the F1-score on Textual-WSD drops by 1.43%. This demonstrates that the implicit sense information generated by the ISG module also facilitates PolCLIPlarge obtaining deep polysemy knowledge.
Regarding the two understanding-based training objectives, the alignment process between the anchor focused on ambiguous target words and candidate images is eliminated first, which corresponds to LW2I . This means that PolCLIPlarge exclusively trains for Textual-WSD. At this point, the HR@1 of PolCLIPlarge on Visual-WSD is only 9.29%. PolCLIPlarge is regarded as a model that
makes random selections for Visual-WSD, since it is required to choose one from ten candidate images for each instance. However, the trade-off problem caused by multi-task training objectives allows its F1-score on Textual-WSD has a 0.35% improvement. Secondly, the alignment process between the anchor and candidate senses is eliminated, corresponding to LW2S . This means that PolCLIPlarge exclusively trains for Visual-WSD. In this situation, the F1-score of PolCLIPlarge on Textual-WSD is only 19.73% but the HR@1 on Visual-WSD increases by 0.44%. The explanations are similar. An additional experiment, which investegates the generality of D-GPT for WSD, is provided in Appendix D.
Overall, the two generation-based modules actually facilitate PolCLIP acquiring multimodal polysemy knowledge. The two understanding-based alignment processes are the most critical components, since they maximize the similarities between contexts and senses/images in a feature space.

heading: 4.5 Optimal layer number for optimizing image encoder
text: To reduce the computational cost, we opt not to optimize all parameters of the image encoder. With all parameters of the text encoder being optimized, the last 0/4/8/12 layers of the image encoder are separately optimized to investigate their impact on the
model performance. The results of PolCLIPlarge with different layer numbers for optimizing the image encoder are displayed in Figure 3. When zero layers are optimized (meaning the image encoder is completely frozen), PolCLIPlarge has the smallest parameter size and the SOTA results on WSD benchmarks. Therefore, this model configuration is selected as our best model (as reported in Table 2). It is interesting that when more layers are optimized, the model performance gradually improves in a small way for Textual-WSD, but drops significantly for Visual-WSD. This is contrary to our expectations. Theoretically, optimizing more layers of the image encoder should enhance the model ability to capture image knowledge. We speculate that redundant knowledge, introduced by some noisy image-sense pairs in the training set, increases the model’s training burden. Thus, refining this disambiguation-oriented image-sense dataset would be valuable.

heading: 4.6 Analysis on model performance for different PoS
text: Since some words may present different parts of speech (PoS) in contexts, exploring the model performance for ambiguous target words with different PoS is beneficial to reveal the unique advantages of the PolCLIP model. The F1-score results of SACE and PolCLIPlarge, trained on the combination of Image-Enhanced SemCor and VWSDKB, are shown in Table 4. Compared with SACE, PolCLIPlarge has improvements of 1.39%, 0.15%, 3.36% and 5.72% in NOUN, VERB, ADJ and ADV respectively. One of the challenges in WSD tasks is the difficulty of understanding non-concrete words accurately, which are often adjectives or adverbs. PolCLIPlarge happens to have a more obvious improvement in adjectives and adverbs. Therefore, we believe that PolCLIPlarge has favorable adaptability and flexibility for ambiguous target words with different PoS, due to its generation-based advantages. On the one hand, it can supplement tangible senses or images with semantically concrete images or descriptions. On the other hand, it can
also supplement non-concrete senses or images with semantically abstract images or descriptions. The case study and visualizations of the implicit image-text complementary information generated by the SIG and ISG modules for concrete and nonconcrete examples are provided in Appendix E.

heading: 5 Conclusion
text: This paper proposes a unified image-text WSD model PolCLIP, which achieves state-of-the-art performance on Textual-WSD and Visual-WSD benchmark datasets. Extensive experimental results prove the effectiveness of our imagetext complementarity strategy. A series of indepth explorations of the model architecture demonstrate the Sense-to-Image Generation module and the Image-to-Sense Generation module can generate effective multimodal complementary representations. The disambiguationoriented image-sense dataset empirically facilitates WSD models understanding of multimodal polysemy knowledge. This may provide a benchmark for the future Multimodal-WSD task. Source codes and datasets are publicly released at: https://github.com/CharlesYang030/PolCLIP.
Limitation
In this work, we did not explore the MultimodalWSD models on multilingual data. The proposed image-text complementarity strategy was employed to address the modality missing issues in unimodal WSD datasets. However, the PolCLIP model still lacked the ability to generate realistic senses and images that can be intuitively validated in terms of their semantics. Moreover, DGPT was built upon GPT-3.5 which was a closedsource model and may be changed, updated, or even discontinued. In future work, we plan to further expand the disambiguation-oriented imagesense dataset to cover more languages. We will also develop a large generic model suitable for Multimodal-WSD.

heading: Acknowledgements
text: The work is supported by grants from National Natural Science Foundation of China (62372189), the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/FDS16/E09/22), and the Scientific Research Innovation Project of Graduate School of South China Normal University (2024KYLX090).

heading: A Fine-tuning D-GPT
text: A fine-tuned disambiguation-oriented GPT-3.5 (DGPT) is developed to generate lexical definitions for ambiguous target words during the testing phase. The gpt-3.5-turbo-1106 model is chosen as the backbone model, since it is one of the latest fine-tunable GPT models released by OpenAI and outperforms several open-source LLMs with smaller parameter sizes in terms of inference capabilities. Constrained by fine-tuning costs, we randomly collect 50,000 data from SemCor to serve as the fine-tuning dataset. Each piece of fine-tuning data consists of a query and an output. The OpenAI fine-tuning platform requires users to simply package their fine-tuning corpus into message-style data and upload it for end-to-end fine-tuning. Following OpenAI’s guidelines1, the format of message-style data for fine-tuning D-GPT is shown in Figure 4. After fine-tuning, D-GPT generates lexical definitions based on queries. For example, a query “What does the word ‘bank’ mean in the context ‘They pulled the canoe up on the bank’?” is fed into D-GPT. Then, D-GPT possibly generates an interpretation “The word ‘bank’ refers to the side or edge of a river, lake, or other body of water”. Even though the output of D-GPT seems to be the answer to generative WSD tasks, we aim to use the output to build a semantically augmented context because WSD is defined as a classification task in this work.

heading: B Construction of the disambiguation-oriented image-sense datasets
text: The disambiguation-oriented image-sense dataset was constructed based on SemCor (Miller et al., 1993) and VWSD-KB (Yang et al., 2023) datasets.
1The fine-tuning guidelines of OpenAI: https:// platform.openai.com/docs/guides/fine-tuning
Specifically, the offline version of BabelNet2 v5.2 was employed to collect a list of relevant images based on each sense in these two datasets. If there were more than five available images in the list, the top five were selected; otherwise, all images were retained. However, a minority of the senses failed to associate with any image through BabelNet, as they were typically non-concrete, like expressing sadness. Thus, BabelPic (Calabrese et al., 2020a), an image-text dataset for non-concrete concepts, was utilized to find images for a part of nonconcrete senses based on babel-ids. Furthermore, those senses that we had collected relevant images were set as an internal knowledge base. For each sense si that was not included in either BabelNet or BabelPic, RoBERTa3 was used to identify the three senses most semantically similar to si within this internal knowledge base, based on text similarity. The first image from each of these three most similar senses was aggregated as the set of images corresponding to si. The entire construction process enabled all word senses in SemCor and VWSD-KB to be aligned with at least one image and at most five images.
After construction, we had conducted a comparative analysis between the proposed training dataset and the existing benchmarks. For Textual-WSD, we examined all contexts in our training dataset and the test set, ensuring there were no duplicates. For Visual-WSD, all images were uniformly transformed into 3×224×224 tensors, and then paired with their respective contexts to form <context, tensor> pairs (Mo et al., 2023; Chen et al., 2024). There were no duplicate <context, tensor> pairs. Through this manual verification, we ensured there was no data leakage.
2BabelNet (updated to version 5.3): https://babelnet. org/
3The hugging face link of RoBERTa: https:// huggingface.co/FacebookAI/roberta-large

heading: C Baselines
text: The details of baselines are as follows: SOTA models in Textual-WSD: BEM (Blevins and Zettlemoyer, 2020) adopted two text encoders and focuses on the representations of ambiguous target words rather than the complete context representations. SACE (Wang and Wang, 2021) employed an interactive context exploitation method and selects similar sentences from the same document to enhance context representations. ZReweighting (Su et al., 2022) utilized a strategy for adjusting training on imbalanced datasets at the word level. These three models obtain outstanding performance on Textual-WSD benchmarks when
training exclusively on SemCor.
SOTA models in Visual-WSD: FCLL (Yang et al., 2023) employed a fine-grained image-text contrastive learning mechanism and benefited from VWSD-KB. It won first place in SemEval-2023 Task 1. UVWSD (Kwon et al., 2023) did not necessitate training but achieved remarkable performance by employing Bayesian inference to incorporate sense definitions.
SOTA models in image-text learning tasks: Openai/CLIP-VIT-L/14 (Radford et al., 2021), Laion/CLIP-VIT-L/14 and Openai/CLIP-VIT-H/14 (Schuhmann et al., 2022) all employ a dual-stream architecture to learn image-text knowledge, simi-
lar to our PolCLIP model. The first model is pretrained on over 400 million image-text pairs, and the latter two are pre-trained on the English subset of LAION-5B (Schuhmann et al., 2022), a large publicly available image-text dataset.

heading: D Generality of D-GPT
text: An additional experiment is conducted to explore the generality of D-GPT for WSD tasks. Specifically, D-GPT is integrated with SACE and FCLL, and its impact on the performance of these two models and PolCLIPlarge is illustrated in Figure 5. D-GPT indeed enhances the performance of these three WSD models. This indicates that using lexical definitions generated by D-GPT to create semantically augmented contexts is a generalpurpose and convenient pipeline for WSD tasks. It can be applied to various WSD models. Furthermore, compared to the evaluation results without D-GPT, PolCLIPlarge shows an improvement of 1.02% F1-score and 1.31% HR@1. These are respectively higher than the 0.44% F1-score increase of SACE on Textual-WSD and the 0.64% HR@1 increase of FCLL on Visual-WSD. This also leads us to believe that PolCLIP could gain more when dealing with contexts that are semantically more accurate, thanks to the image-text complementarity strategy.

heading: E Effectiveness of the SIG and ISG modules
text: In order to further intuitively reveal the effectiveness and importance of the two generation-based modules (i.e., SIG and ISG), the generated implicit image-text complementary information is visualized. Two groups of concrete and non-concrete examples are collected from the test set. Each group of examples contains a sense and an image. They are fed into the trained SIG and ISG modules respectively, and then a sense-generated image vector and an image-generated sense vector are output. All senses and images in the training set are transformed into vectors by text and image encoders, serving as two separate candidate pools. By calculating vector similarity, the top-5 most similar images can be identified from the image candidate pool based on the sense-generated image vector. Also, the top-5 most similar senses can be identified from the sense candidate pool based on the image-generated sense vector. Visualizations of these two groups of concrete and non-concrete
examples are shown in Figure 6. For the concrete example, the top-3 images are semantically consistent with Sense 1. Even if the last two images do not depict the shape of a bell, they are related to music or sound, which is one of the functions of bells. Based on Image 1 related to milk, the retrieved five senses are all semantically correct. For the non-concrete example, the top-3 images are semantically relevant to Sense 2 related to beauty. Even based on Image 2, which primarily shows a man’s face expressing pleasure, the top-3 senses accurately capture concepts of relaxation, laughter, and beard.
In summary, the SIG and ISG modules are reliable in generating effective multimodal complementary information.

heading: Future Work
text: This may provide a benchmark for the future Multimodal-WSD task. Source codes and datasets are publicly released at: https://github. com/CharlesYang030/PolCLIP. Limitation
In this work, we did not explore the MultimodalWSD models on multilingual data. The proposed image-text complementarity strategy was employed to address the modality missing issues in unimodal WSD datasets. However, the PolCLIP model still lacked the ability to generate realistic senses and images that can be intuitively validated in terms of their semantics. Moreover, DGPT was built upon GPT-3. 5 which was a closedsource model and may be changed, updated, or even discontinued. In future work, we plan to further expand the disambiguation-oriented imagesense dataset to cover more languages. We will also develop a large generic model suitable for Multimodal-WSD