SUT at SemEval-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation
Year: 2023

Abstract:
Visual Word Sense Disambiguation (V-WSD) identifies the correct visual sense of a multisense word in a specific context. This can be challenging as images may need to provide additional context and words may have multiple senses. A proper V-WSD system can benefit applications like image retrieval and captioning. This paper proposes a Prompt Generation approach to solve this challenge. This approach improves the robustness of languageimage models like CLIP to contextual ambiguities and helps them better correlate between textual and visual contexts of different senses of words.

heading: 1 Introduction
text: There are different senses associated with polysemous English words (Bevilacqua et al., 2021). The task of determining a specific sense of a word has been one of the most significant challenges in Natural Language Processing. Several studies have been conducted on determining a word’s sense based on its sentence. With the emergence of multimodal works, visual and acoustic information can be used to solve a word’s ambiguity. These models can be used to expand capability of models in limited textual context. Using a proper Visual Word Sense Disambiguation model can contribute to the advancement of different areas, such as image captioning, image retrieval, and other multimodal problem settings.
Our main approach is to tackle the problem of V-WSD (Raganato et al., 2023) using prompt engineering. We used ChatGPT (OpenAI, 2022) to generate a desired prompt for a given sense. Using CLIP model (Radford et al., 2021) we extract text and image embeddings for the given input and compare their cosine distance. Our model works for
*The first two authors contributed equally and listed randomly.
English, Farsi and Italian inputs with average MRR of 71.13% ranking 9th in SemEval final results.
Using ChatGPT to expand contextual information increased model’s accuracy. However, since ChatGPT usage has limitation on number of requests, the model input is limited.
Our code is publicly available on github1.
Related Works
There are substantial works on word sense disambiguation in natural language processing (Navigli, 2009). A word sense disambiguation can be done using supervised (Wang and Wang, 2021), (Bevilacqua and Navigli, 2019), (Hadiwinoto et al., 2019), or knowledge-based models (Moro et al., 2014), (Agirre et al., 2014). Knowledge-based works use lexicon graph-based information (such as WordNet) to extract a word’s sense. Supervised methods return the sense of a word based on the presented context (Bevilacqua et al., 2021).
The upper bound accuracy for this WSD is 80 percent which is human accuracy on annotated data (Navigli, 2009). State-of-the-art models (Bevilacqua and Navigli, 2020), (Barba et al., 2021) have reached this accuracy. In V-WSD we use visual information beside contextual data in order to improve model accuracy (Barnard and Johnson, 2005), (Su and Jurie, 2011). There are several datasets for V-WSD. Most of these datasets concentrate on verbs (Gella et al., 2016), (Gella et al., 2019). MuWoSeD (Anonymous, 2023) is a general visual word dataset we use to finetune our models.

heading: 2 Our System

heading: 2.1 Overview
text: The overview of our approach to tackling the problem of V-WSD is based on the idea of prompt engineering. Prompt Engineering helps the model
1https://github.com/language-ml/SemEval-VWSD
2160
overcome the disambiguation present in the textual contexts by adding details and clarification in the given prompts to the model to find the corresponding visual context. Large Language Models (LLMs) can help us better understand the context of a sense. As a result, we use ChatGPT as our LLM for generating prompts for each of the senses. Our approach does not need any fine-tuning technique for inference, resulting in more efficiency for the model.

heading: 2.2 Prompt Generation Module
text: Here we detail the Prompt Generation module: we first give the word and corresponding sense to the ChatGPT to generate a desired prompt for that sense of a word. This is done by first transforming the textual context into a template of the form “What does the [WORD] mean in [CONTEXT]?”. Then, we get the response from ChatGPT that contains some extra information regarding that word, which can guide the multimodal model to correlate visual and textual contexts better. Furthermore, due to the limit of requests and the large number of words in the test dataset, we batched these words in a request to reduce the number of requests to avoid the ChatGPT limit.

heading: 2.3 Multimodal Module
text: We use the CLIP model (Radford et al., 2021) as our Multimodal Module to have a shared embedding space between texts and images. This module produces an embedding for texts and images such that related texts and images have more cosine similarity than others. CLIP model (Radford et al.,
2021) uses a text encoder and image encoder and projects the output vectors to a shared space, calculates the similarity between any two pairs of text embedding and image embedding of a batch of N pair samples into an N ∗ N matrix with cosine similarity, and attempts to increase the value of N elements in the diagonal matrix while decreasing the value of the other N ∗N .
As the CLIP model has been trained on a large amount of data, it should have learned the shared embedding between the image and the text well enough to differentiate between the meanings of ambiguous words and related images.
We provided a visual representation of our approach in Figure 1. We provide the generated prompt and candidate images to the CLIP model, as depicted in part(d) and part(e) of Figure 1, and based on the cosine similarity between the text em-
bedding and each image embedding, we determine a ranking for the related images, as depicted in part(f) of Figure 1.

heading: 3 Experimental Setup and Evaluation
text: For our experiments, each dataset instance has three main parts: the ambiguous word, limited textual context, and ten image candidates, one of which corresponds to the desired visual context, and there were three languages in the test dataset: English, Italian, and Farsi. We first batched the dataset words and their textual contexts and got their corresponding prompts. We then used different backbones of the CLIP model as the Multimodal Module of our system. Additionally, we took a mean pooling of all of these backbones to predict the correct image. For the Farsi and Italian words, we translated them into English and then used the translated text as the input for our Multimodal Module.
We used ResNet-50 and ResNet-101 (He et al., 2016) and ViT (Dosovitskiy et al., 2020) backbones for the CLIP models. We also used fine-tuned models on the MuWoSeD dataset (Anonymous, 2023), a Multimodal Word Sense Disambiguation dataset. We fine-tuned models in 5 epochs on this dataset. We test various models using the same two ranking metrics specified in the task. The first metric is Hits@1. This metric measures the frequency with which the ground truth appears in the top position. Mean Reciprocal Rank (MRR) is our second metric, calculated using the following formula:
1
N
N∑
i=1
( 1
ranki )
where N refers to the number of test data and ranki refers to the rank of ground truth in i-th test data.

heading: 4 Results
text: The results for our models are shown in Table 1. For the English language, our model achieved Hits@1 of 0.67 and MRR of 0.79 in zero-shot settings. When fine-tuned on the training dataset of the task, it increased to 0.69 and 0.80 for Hits@1 and MRR, respectively. The pre-trained CLIP on the MuWoSeD dataset got Hits@1 of 0.68 and MRR of 0.80, which shows an increase compared to the zero-shot setting.
For the Farsi language, our model achieved Hits@1 of 0.50 and MRR of 0.64, suggesting that
this language is more challenging for the model. Moreover, our model produced Hits@1 of 0.53 and MRR of 0.68 for the Italian language.
Overall, our model achieved Hits@1 of 0.57 and MRR of 0.71. This suggests that our zero-shot approach performs fairly well, given the fact that it is not fine-tuned on a specific dataset and can perform fairly well across different domains. In addition, it consumes less energy since it does not need any data for training. We also evaluated the pre-trained MuWoSeD on this task dataset and observed that the pre-trained model on the same task but with different data increased the accuracy in the zero-shot setting.

heading: 5 Discussion and Conclusions
text: In this paper, we proposed a model to solve the Visual Word Sense Disambiguation challenge using Large Language Models to generate prompts for different textual contexts of words to add extra information to them to make multimodal models select the corresponding image of that context better.
For the English language, we used ChatGPT to generate those prompts and then gave those prompts to different backbones of CLIP models to detect the related image of that sense both in zeroshot and fine-tune settings. It shows that fine-tuning on the training dataset improves the model’s performance. Moreover, using a pre-trained model on the MuWoSeD dataset in zero-shot settings leads to better results than the original CLIP model in the zero-shot setting.
We first translated textual contexts for the Farsi and Italian languages and then used the result as the text input of our multimodal models in zero-shot settings. Using a zero-shot setting significantly reduces the consumed energy while maintaining its performance at a fairly good level in different domains.
Future works can include prompt generation in multilingual settings to add extra information to the textual input of the multimodal models.

heading: Future Work
text: Future works can include prompt generation in multilingual settings to add extra information to the textual input of the multimodal models