Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval
Year: 2023

Abstract:
In this study, we propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation (WSD) component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). We confirmed previous findings — human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts yield the worst results. Our code is available at https:// github.com/pxm427/WSD-for-IR.

heading: 1 Introduction
text: The SemEval 2023 shared task VWSD1 combines WSD and Image Retrieval (IR), which aims to select a correct image among ten candidates using a phrase containing ambiguous words. Neural models are likely to be attracted by frequent tokens, labels, and senses of ambiguous words, particularly in limited contexts. We determined that CLIP (Radford et al., 2021) fails to find the correct images using phrases with the ambiguous words of frequent senses even enhanced with contrastive learning on large-scale data.
As shown in Figure 1, given phrase “Andromeda tree”, the pretrained CLIP selected incorrect images that focused on either constellation “Andromeda” or part of “tree”, neglecting the phrase’s meaning entirely. This sample demonstrates ambiguity as a challenge for state-of-the-art neural models. Therefore, we exploited GPT-3 (Brown et al., 2020), a large language model (LLM), for its pretrained knowledge as implicit sense disambiguation and phrase context enrichment for this task.
Prompt engineering boosts model performance and plays a crucial role in applying LLMs to many NLP tasks, which lessens training and testing discrepancies resulting from human-like languages
∗Now at Hitotsubashi University 1https://raganato.github.io/vwsd/
(Liu et al., 2023). A prompt refers to a text or a set of instructions that guides the model to generate a specific type of response or output. A prompt can be a question, statement, keyword, or sequence of words that provide context and information to the model. Recent research on prompt techniques demonstrates that well-designed prompts spur the potential of neural models without modifying their parameters (Jin et al., 2022). We are curious about how prompts can improve the performance of CLIP on VWSD.
Our main contributions are as follows:
1. We explored different templates for queries and observed their effects on VWSD. The quotes and highlighting ambiguity are effective.
2. We adopted GPT-3 as a key VWSD component to generate queries, which improved the performance in terms of accuracy.
3. We demonstrated that our prompt techniques are effective for finetuning CLIP.
417

heading: 2 Method
text: Each VWSD phrase contains an indication of the ambiguous word(s). As illustrated in Figure 2, we first introduce a baseline CLIP that takes text input as either VWSD short phrases or a longer query enhanced with templates for ambiguous words. Then, we leverage GPT-3 to further enrich these queries for CLIP.
CLIP with phrase and queries In a shared feature space, CLIP provides a joint embedding representation for each (image, text) pair. The joint embedding representation allows for semantic similarity comparisons between images and text, that is
similarity score = CLIP(Eimage , Etext) ,
where Eimage and Etext are the embeddings obtained from its image and text encoders, respectively. We focus on text input for VWSD. text takes the form of either a single VWSD phrase or list of queries that bears the phrase and indication of the ambiguous words in the phrase. Table 1 lists our nine templates for creating the queries. Take “Andromeda tree” as an example; “Andromeda” fits the slot [ambiguous word(s)] and “tree” fits [rest of word(s)]. Template #1 appears to be logically contradicted with #2. This is because some ambiguous words in the phrase do not fit slot [ambiguous word(s)], but fits slot [rest of word(s)]. Moreover, template #3 is for both ambiguous words and rest of words to improve the coverage. These different templates semantically
fit different phrases in VWSD, and their performance with CLIP are similar. We select the maximum of the similarity scores from all the queries, and we want the image with the highest score.
Query Enrichment with GPT-3 GPT-3 is a powerful language model that can perform various NLP tasks such as language generation, text classification, and question answering. It was designed to improve upon the limitations of previous language models by training a large-scale neural network on massive amounts of text data. This allows GPT-3 to understand the context and generate coherent and contextually relevant responses to text-based inputs, making it useful for a wide range of NLP applications. Additionally, GPT-3 can be finetuned on specific NLP tasks to further enhance its ability to perform various language-related tasks.
As shown in Table 2, we induce additonal knowledge from GPT-3 by posing different questions for VWSD phrases which was used to finetune the CLIP: 1) a direct query, 2) a query with double quotes to highlight the phrase, and 3) adding an explicit phrase to separate the phrase ambiguity (i.e., ambiguous word(s)) from others based on 2). These three types are concatenated to the original phrase as a query to finetune CLIP for better performance.

heading: 3 Experiments

heading: 3.1 Settings
text: In this study, we used data resources including images and phrases released by SemEval-2023 Task 1. Here, we chose only the English version from
all the language versions (English, Farsi, Italian). We divided the official training data into a training set (11,869) and development set (500). Finally, we evaluated our finetuned models on the test data (463), whose contents are different from the training data.
We employed pretrained CLIP as our baseline to calculate the similarity score between the image
and text. In the baseline, we used only a phrase as the input of the text component, and the performance is not good. We further finetuned the CLIP to improve the performance by expanding the phrases to queries, and even enriching queries with GPT-3.
For training, we set the batch size to 100 with 10 epochs and used a learning rate of 1e-7. For GPT-3,
we chose text-davinci-003, which was considered the most capable GPT-3 model. text-davinci-003 exhibited a better performance with a higher quality, longer output, and better instruction-following than the other models.

heading: 3.2 Results & Analysis
text: Table 3 lists the results based on the baseline model and finetuned models using different prompt types.
As shown in Table 3, the accuracy of the finetuned models on Dev and Test performed better than the baseline model. This proves that finetuning can improve the performance of CLIP. All the results on the Test were much lower than those on the Dev. This may be because the Dev data was obtained from the training dataset, which is thematically different from the Test dataset.
For different prompt types, the accuracy on the Dev varied. The finetuned CLIP adapting prompt type Direct had the lowest overall performance with 90.20, and prompt type Double quotes had the highest overall performance with 93.20. A speculative reason for this was the lower knowledge quality when selecting the prompt type Direct because GPT-3 tended to not consider the phrase entirely when asking directly, thereby generating inaccurate knowledge. For prompt type Explicit phrase, it could reach a point of 92.20.
On the Test, including the baseline, prompt type Explicit phrase exhibited the best performance, which could reach up to 66.95. This indicates that the knowledge generated by GPT-3 was beneficial. Conversely, the performance of prompt type Direct was worse than the baseline, which may indicate that poor knowledge can introduce negative effects.
Finally, we conducted ensemble experiments between each prompt type. The results demonstrated an improvement in accuracy for all prompt types except Explicit phrase.

heading: 3.3 Ablation Study
text: To investigate the benefit of the effect of queries, knowledge from GPT-3, and finetuning, we conducted ablation experiments. We counted the number of answers that improved or worsened in terms of the change of the gold answer rank. Table 4 shows that samples that improve are increasing.
Query templates. As shown in Table 4, the baseline results were the lowest: 71.50 and 58.53 on Dev and Test, respectively. This is because the phrase was too short to carry meaningful information. Therefore, when creating a sentence including a target phrase as a query, more contextual information can be obtained. Consequently, the score increased by 0.9 and 2.59 on Dev and Test, respectively, compared with the baseline.
Prompt engineering. To better use of the information in context, we have added knowledge from GPT-3 based on the prompts. The score particularly increased to 80.40 on Dev, which proves that adding knowledge from GPT-3 improves the performance.
Finetuning. In the finetuning section, we first finetuned CLIP with only queries. After finetuning, the accuracy on Dev increased by 15.80 compared with CLIP with queries, which proves the importance of finetuning. Paying attention to the results on Test in CLIP with queries and GPT-3 knowledge is also important. Table 4 shows the score of 64.58 was 3.24 points higher than the result of CLIP finetuned with queries. This is partially explained by the knowledge from GPT-3 being partially effective. We further finetuned CLIP with prompts and GPT-3. The best performance reached scores of 92.20 and 66.09 on Dev and Test, repectively, which illustrates the usefulness of GPT-3.

heading: 4 Related Work

heading: 4.1 Knowledge generated from LLMs
text: Unlike WordNet (Miller, 1994) and SemCor (Miller et al., 1993), recent large-scale language models (LLMs) provide an easy explanation for ambiguous words. In particular, LLMs are well suited for disambiguation tasks. For example, BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have a general language understanding ability that has been demonstrated to capture word senses (Coenen et al., 2019).
Recently, Brown et al. (2020) proposed Generative Pretrained Transformer-3 (GPT-3), an LLM trained on a massive amount of data, for various NLP tasks such as dialogue generation (Zheng and Huang, 2021; Lee et al., 2022). It demonstrates an understanding of logical reasoning and external knowledge, which has made it applicable GPT-3 to solving complex problems involving cause-and-effect relationships (Liu et al., 2022). When provied with a proper prompt or asked a human-like question, a pretrained GPT-3 model responds with fluent and relevant text as an answer, which shows passable “logic” and details for disambiguation. The quality of the answer depends on the prompt and question. However, to the best of our knowledge, no research has been conducted on VWSD using LLMs. In this study, we rely on LLM output as external knowledge for VWSD.

heading: 4.2 Image Retrieval
text: Recently, IR has undergone dramatic shifts from approaches handcrafted with global and local descriptors, to convolutional neural networks (He et al., 2016) with adaptive local descriptors, to recent non-convolutional models with one global descriptor, such as a Vision Transformer (Dosovitskiy et al., 2021, ViT). Experimental evaluations (Gkelios et al., 2021) show that ViT achieves competitive results at a low complexity and even finetuning is not required, which makes it an attractive choice as a baseline model for IR.
Recently, researchers began leveraging natural language descriptions in computer vision to improve performance. He and Peng (2017) and Liang et al. (2020) showcased the utilization of natural language descriptions and explanations to enhance the fine-grained visual classification of birds. Radford et al. (2021) presented CLIP in a zero-shot setting, which demonstrated the model’s substantial potential for widely-applicable tasks such as IR
(Mori et al., 1999).

heading: 5 Conclusion and Future Work
text: We explored the effects of a query on VWSD and used GPT-3 as a key to generate queries. After finetuning CLIP with queries generated from GPT-3, we determined that queries generated by GPT-3 using prompts improved the performance in terms of accuracy.
In the future, we plan to apply some other multimodal models and compare the results with those of existing works. We also intend to adopt GPT-4 2 to generate knowledge considering both textual and visual cues for VWSD.

heading: Future Work
text: In the future, we plan to apply some other multimodal models and compare the results with those of existing works. We also intend to adopt GPT-4 2 to generate knowledge considering both textual and visual cues for VWSD