SRCB at SemEval-2023 Task 1: Prompt Based and Cross-Modal Retrieval Enhanced Visual Word Sense Disambiguation
Year: 2023

Abstract:
The Visual Word Sense Disambiguation (VWSD) shared task aims at selecting the image among candidates that best interprets the semantics of a target word with a short-length phrase for English, Italian, and Farsi. The limited phrase context, which only contains 2-3 words, challenges the model’s understanding ability, and the visual label requires image-text matching performance across different modalities. In this paper, we propose a prompt based and multimodal retrieval enhanced VWSD system, which uses the rich potential knowledge of large-scale pretrained models by prompting and additional text-image information from knowledge bases and open datasets. Under the English situation and given an input phrase, (1) the context retrieval module predicts the correct definition from sense inventory by matching phrase and context through a biencoder architecture. (2) The image retrieval module retrieves the relevant images from an image dataset. (3) The matching module decides that either text or image is used to pair with image labels by a rule-based strategy, then ranks the candidate images according to the similarity score. Our system ranks first in the English track and second in the average of all languages (English, Italian, and Farsi).

heading: 1 Introduction
text: Word Sense Disambiguation (WSD) is a challenging task of Natural Language Processing, which aims at identifying the polysemic word with correct interpretation (Bevilacqua et al., 2021). It has numerous applications, such as machine translation, search engines, and text understanding. The traditional WSD task provides the predicting word with its context, which is supposed to be a sentence containing the word, and chooses the proper sense from the predefined inventories (Miller, 1995; Navigli and Ponzetto, 2012). The model is asked to
∗Contribution during Internship in Samsung Research China-Beijing.
extract the semantic feature of the target word from the particular text and match it with the most practical sense. The WSD task can be processed as a classification problem among the senses as labels in the given inventory by taking advantage of the neural language model’s powerful semantic understanding capability. The appearance of Visual WSD connects the different modalities and gives the word sense a new presentation form. The sense inventory now is a set of images instead of text definitions, and the target word has to be paired with the corresponding image. The Vision-Language model, CLIP (Radford et al., 2021), has shown its outstanding performance on unifying image-test modalities, and it provides potential knowledge for semantic understanding required by disambiguation.
The SemEval-2023 Task-1 Visual Word Sense Disambiguation (VWSD) (Raganato et al., 2023) is a multilingual WSD task, for English, Italian, and Farsi, with the phrase as textual context and images as visual senses. Most of the system setup in this paper is under the English track. The other two languages are translated to English and only apply a partial sub-system without image sense involvement. Under the English situation, the phrase contains the target word and has a length of 2-3, which differs from the traditional WSD task with the sentence as context. This challenges the encoder to extract sufficient semantic information from the target word from the short text. Image sense labels require a highly unified image-text connection to produce the final output. For Farsi and Italian, although the specific language version of CLIP and multilingual CLIP are available (Bianchi et al., 2021; Sajjad Ayoubi, 2022; Carlsson et al., 2022), the actual performance in the current task is limited by the scale of pre-training data.
Our system makes the best use of the potential knowledge of the pre-trained model by prompting rather than continuous training and fine-tuning. We
439
gather information on the target word from semantic networks and online dictionaries. The most pertinent information is selected by matching the augmented phrase through a biencoder architecture and regarded as the prompt. This biencoder architecture was proposed in BEM (Blevins and Zettlemoyer, 2020),which jointly trains context and gloss in the same vector space. SimCSE (Gao et al., 2022) involves contrastive loss to maximize the similarity of sentences with identical meanings, which makes it a powerful tool for matching phrases and definitions. Therefore, we use SimCSE as the backbone of our biencoder architecture. The later work (Wu et al., 2022) improves the performance of matching text with different lengths. There is a recent multimodal WSD dataset (Calabrese et al., 2020) that uses images to represent the sense of the concept. Beyond the text information, we use additional images from the open image dataset. Different types of words require different modalities to express. Therefore, we design a rule-based process that decides whether text prompt or images are used for final sense representation. Finally, we use a huge version of CLIP to extract features and rank the candidate images according to similarity scores.

heading: 2 System Overview
text: The system description in this section is under the English situation. Italian and Farsi are translated to English and only apply a partial sub-system without image sense involved. For English, we divide the VWSD task into three main modules, including (1) context retrieval module, (2) image retrieval module1, and (3) matching module, as shown in Figure 1. The two retrieval modules perform disambiguation among senses and are regarded as a WSD-specific modules.

heading: 2.1 Context Retrieval Module
text: In this module, the most suitable text prompt interpreting the meaning of the target word will be selected. We use the definitions and synonyms as the context and match them with the given phrase, which is a similar setup to the traditional WSD task. However, this approach still needs more information on the short-length phrase when matching with context. We use translation to gain extra knowledge by back-translating the phrase from another language to English with multiple online translators (Luan et al., 2020) and concatenate the results. Then, we put the augmented phrase and the customized context into the biencoders, which
1This module is not applied on Italian and Farsi.
separately encode phrase and context as shown in the Text Encoder part of Figure 1. To further improve the WSD performance, we apply weighted similarity to minimize the distance to the phrase. We disambiguate the confused senses of the target word and maximize the similarity with the phrase with equation 1,
Simi = Xi ·X⊤context Simfinal = Simphrase − α ∗ Simword
(1)
and α determines the disambiguation weight. As a result, compute the similarity between phrase and candidate contexts, which gives the nearest embeddings to represent the correct sense.

heading: 2.2 Image Retrieval Module
text: Instead of constraining in text space, we are inspired to obtain cross-modal information by BabelPic (Calabrese et al., 2020), a multimodal dataset for non-concrete concepts. Although the domain of BabelPic does not match with the task dataset, it is observed that there are a considerable amount of specific entities in both training and testing datasets that benefit from image information. Therefore, we collected the extra image data using clip retrieval (Beaumont, 2022), which retrieves images according to the similarities between phrase and LAION image embeddings.

heading: 2.3 Matching Module
text: With the text context and image sense as the inputs, the final module will match them with candidate image labels and produce the image ranking according to similarities. The decision function chooses either text context or image sense to be used for final matching. We investigated the BabelNet information and found that it provides valuable properties of target words. We designed a series of rules to decide using text or images based on the word properties. The selection strategy is designed according to the sense properties of BabelNet shown in Figure 2, and an overall decision process is based on the error analysis of training data. For instance, named entities of geographic places and biological creatures benefit from images and have a higher similarity than text context. However, concepts have various visual representations, and images might involve more mistakes than text prompts. Then the final context is processed through the corresponding encoder with the image labels. The encoders are from a large pre-trained VLM, and
we believe that the pre-training data contains potential knowledge that benefits WSD purpose and the alignment capability, which is significant to obtain correct text-image pairs. Therefore, we use a huge version of CLIP trained on LAION-5B as our module backbone for feature extraction.

heading: 2.4 Sense Inventory and Data Augmentation
text: The WSD task resource is an inventory containing all possible senses of the target word, and the widely-used sources are semantic networks such as WordNet and BabelNet. As the given phrase is too short for the model to extract information, we collected extra data from Wikipedia, WordNet, BabelNet, and other online dictionary using the target word (phrase if available). The details of data collection are introduced in Section 3.1. The online dictionaries involve extra text knowledge over the traditional knowledge base, which is proved to be useful (Bevilacqua et al., 2021). The quality of sense inventory significantly affects the WSD results, and a good inventory should contain sufficient and understandable information for the model to match with phrases and image labels. Our sense inventory contains abundant and high-quality information on the predicting words after a postprocessing strategy.

heading: 3 Experimental Setup
text: This section will discuss the detailed data usage and experiment setup. As there is a distribution gap between training and testing data, most of the setup is based on the testing and customized validation sets.

heading: 3.1 Data Usage
text: Data splits There is a huge sense distribution gap between training and testing data, as shown in Table 1. The number of senses is referred to the online dictionaries rather than BabelNet, which prefers to contain as many senses as possible. Most of the training set is monosemy, which rates over 70% and most are under biology, rather than pol-
ysemy, which is expected to be processed by the WSD task. The reversed situation is applied to the testing set. To bridge this gap, we randomly select target words with certain number of senses and construct a validation set with the same distribution as the testing set.
Extra data The external gloss inventory is formed by context from multiple sources. For Wikipedia and online dictionaries, we only retrieve the definitions of the target word. For semantic networks, including WordNet and BabelNet, we gather synonyms, categories, domains, and types besides definitions.

heading: 3.2 Experimental Setup
text: The prompt template decides the exploring depth of the pre-trained model, and it requires the right information with the right position. Although the learnable prompt is proven to benefit downstream tasks, it has limited generalization on zero-shot problems (Zhou et al., 2022b,a). This task data is proved as zero-shot and long-tail, and we applied a fixed prompt template. The essential elements of the prompts are synonyms and glosses, which contributes a lot to matching phrase and images. By adding target phrase and connecting words, the final template is
[Phrase], as known as [Synonyms], is [Gloss].
e.g. Reflecting glass, also known as looking glass, is a mirror; usually a ladies’ dressing mirror
and the order of the elements has slight influences on the results.
We use Microsoft, Google translates, and DeepL to perform back-translation on phrases and concate-
nate the result to pair with prompt context through a SimCSE-based biencoder. For image collection, clip retrieval (Beaumont, 2022) is a tool that allows users to retrieve images with given text, which encodes text and images to clip embeddings and calculates the distances. Finally, the huge version of open-clip trained on LAION-5B (Cherti et al., 2022) is applied to match text-image or imageimage pairs and rank the candidates by similarity scores.
The evaluation measures below are used as marking metrics:
• Hit rate is the accuracy of correctly ranking the gold image in the first place.
• Mean reciprocal rank (MRR) is a measurement evaluating a ranked list as shown in the following equation 2.
MRR = 1
N
N∑
i=1
1
ranki (2)

heading: 4 Results and Discussion
text: In this session, we mainly analyze the English results of the approaches in our system and discuss how it contributes to the results. The training and testing data are further compared and analyzed in the behavior of different training strategies.

heading: 4.1 Final results
text: Our system achieves a hit rate of 84 on the English testing set, which ranks 1st out of 56 teams on the English leaderboard. The baseline approach uses the Most Frequent Sense from sense inventory as the input context and matches it with image labels. There is a huge gap between the results of the
training and the testing set, and the constructed validation set has limited ability to eliminate the gap, as shown in Table 2. By applying a SimCSE-based biencoder to select the correct sense, the results significantly increase the testing set, which is related to the more multi-sense words in the testing set than the training set listed in Table 1. Another major improvement is adding synonyms to the prompt, which has contributed a lot to WSD ability. After analyzing the CLIP results, we find out that the synonyms can match the pre-training data of CLIP and transfer the potential knowledge of pre-trained models. To further improve the WSD performance, we back-translate the phrase for augmentation and add weighted similarity to minimize the distance to the phrase, which leads to a hit rate of 78 with text only.
The involvement of image representation provides cross-modal information for disambiguation. The basement results of the image are not as expected, but the overlap of wrong instances is small, which means the combination of image and text
covers a broader range of correct instances. Those words represented well by visual information, such as named entities, are replaced by images and, as a result, we achieve the final result of 84 hit rate on the testing set.

heading: 4.2 Distribution Shift
text: The distribution of senses determines the contribution ratio of the WSD module and matching module. The training set has more single-sense words, with a proportion of 70%, than multi-sense words, and the testing set has the reversed situation as indicated in Table 1. The training data is more likely to be a text-image matching task rather than a WSD task with minor WSD contribution, and the modification of the matching module has more influence on the results. However, the testing data involves more ambiguity than the training set, and most words have multiple meanings. Therefore, after the testing data was released, we changed our strategy and put more effort into WSD performance, as shown in Table 2 index 1-5.

heading: 4.3 Fine-tuning and Pre-training
text: To investigate the value of training data, we tried both fine-tuning and continuous pre-training on CLIP with the training set. As the accuracy of the selected gloss cannot be guaranteed, the phrase context is the only available text data to be learned. Moreover, the testing set covers two more languages over English, making fitting more challenging.
Fine-tuning the CLIP classification layers with training data give 1% improvement on the validation set, which has the probability of dropping
down the zero-shot capability on unseen words. The training set shows its properties, including open domain, long tail, and zero-shot, as shown in Figure 3. The classification layers have limited capability of dealing with exceptionally sparsely distributed data.
Continuous Pre-training the CLIP with a contrastive loss between given phrase and images gives better results than fine-tuning classification layers, which has over 2% improvement. Rather than adapting specific data distribution, continuous training could be regarded as injecting more knowledge into the model, which might benefit generalization.

heading: 5 Conclusion
text: We describe our prompt and biencoder-based Visual WSD system and investigate how the potential knowledge of large-scale pre-trained VLM contributes to disambiguation and modal alignment. The well-designed prompt template connects the input phrase and the potential knowledge of the pretrained model, which also prevents the zero-shot generalization capability. The sense inventory is the fundamental element for WSD tasks, and context quality determines the performance boundary. The involvement of images in the WSD module further extends the coverage of sense interpretation ability. With the combination of text context and image representation, our system achieves a hit rate of 84 and ranks first in the English track.
In future work, more research on the modal fusion strategy will be completed, and the automatic fusion approach that applies to the different datasets will be designed. Rather than selecting a single modality based on rules, extracting useful information from both text and images could benefit the disambiguation performance.

heading: Future Work
text: In future work, more research on the modal fusion strategy will be completed, and the automatic fusion approach that applies to the different datasets will be designed. Rather than selecting a single modality based on rules, extracting useful information from both text and images could benefit the disambiguation performance