OPI at SemEval-2023 Task 1: Image-Text Embeddings and Multimodal Information Retrieval for Visual Word Sense Disambiguation
Year: 2023

Abstract:
The goal of visual word sense disambiguation is to find the image that best matches the provided description of the word’s meaning. It is a challenging problem, requiring approaches that combine language and image understanding. In this paper, we present our submission to SemEval-2023 visual word sense disambiguation shared task. The proposed system integrates multimodal embeddings, learning to rank methods, and knowledge-based approaches. We build a classifier based on the CLIP model, whose results are enriched with additional information retrieved from Wikipedia and lexical databases. Our solution was ranked third in the multilingual task and won in the Persian track, one of the three language subtasks.

heading: 1 Introduction
text: Visual word sense disambiguation (VWSD) is a task in the field of multimodal natural language processing, in which the goal is to identify the intended meaning of a target word in a given context by selecting the most appropriate image from a set of candidate images. Finding images corresponding to the correct meaning of the word might improve the performance of methods combining text and visual information such as image search engines, visual question answering, or image generation models.
SemEval-2023 workshop hosted a task on visual word sense disambiguation. The task involved selecting the best matching image out of ten candidates given a short textual description. The descriptions usually consisted of two words: the target word and the context word (Raganato et al., 2023). For example, the phrase andromeda tree contains the ambiguous target word andromeda and the context word tree, which indicates a specific meaning of the target word. The task organizers provided three datasets, of which the trial and training datasets contained phrases in English, while
the test dataset was multilingual and consisted of English, Italian, and Persian subsets. Participants were allowed to submit their solutions for a particular language or for all three languages. The systems were ranked according to the average accuracy score from three language-specific subtasks.
In this paper, we describe our system for the VWSD shared task. The backbone of our solution is a classifier using multimodal CLIP embeddings (Radford et al., 2021; Cherti et al., 2022), which has been enriched with features extracted from Wikipedia and dictionaries. These knowledge sources are used to retrieve textual and image data, providing additional information useful for determining the correct meaning of the target word. Our system was ranked third in the multilingual task and took first place in the Persian subtask. The source code of our system is publicly available, as well as the fine-tuned models and other resources required to reproduce our experiments.1

heading: 2 System description
text: The proposed approach consists of several modules which together constitute a visual word sense disambiguation system. The three core components of our approach are: 1) a classifier based on CLIP image-text embeddings, 2) a Wikipedia retrieval module, 3) a learning to rank (LTR) model whose role is to generate the final ranking of images based on the features provided by the other modules. A high-level overview of the system is shown in Figure 1.

heading: 2.1 CLIP-based classifier 1
text: CLIP (Contrastive Language-Image Pretraining) is a method for learning multimodal embeddings for language and vision by aligning the representations of images and their captions. The original CLIP models published by OpenAI (Radford et al., 2021)
1https://github.com/sdadas/vwsd
155
were trained on a dataset of 400 million image-text pairs. Recently, CLIP architectures trained on even larger datasets and with more parameters have been released by the LAION group, achieving state-ofthe-art results on several image retrieval and zeroshot image classification tasks (Cherti et al., 2022). Visual word sense disambiguation can be viewed as a text-to-image matching task, the type of problems for which CLIP is particularly effective. Therefore, we chose this model as the basis for our classifier.
We utilize CLIP in zero-shot mode, using a pre-trained checkpoint, to assign a score for each context-image pair from the data sample. Specifically, we compute vector representations of textual context c and image x, and then calculate the similarity between these vectors using the following formula:
score(c,x) = sim(c,x)− p(x) (1) in which sim(c,x) denotes a standard cosine similarity and p(x) is a score penalty for the image x. The penalty is calculated for each image as the mean similarity between that image and all the contexts in the dataset, normalized by the frequency of image occurrence. The rationale for using penalties is the observation that some images have high cosine similarity to many contexts, leading the model
to incorrectly prefer them for the majority of samples in which they appear. The penalty lowers the similarity for these cases without affecting the results for the other images. We calculate it using the following formula:
p(x) =   1 |C| ∑
ci∈C sim(ci,x)
  · card(x)
max xj∈X card(xj)
(2) in which C is the set of all contexts, X is the set of all images, and card(x) denotes the number of samples in which image x appears.

heading: 2.1.1 Multilingual classification
text: Publicly available CLIP models were trained on a set of English image captions, and are therefore not adapted for generating vector representations for texts in other languages. Consequently, the described method cannot be applied directly to Italian and Persian. However, it is possible to use transfer learning methods to train a multilingual or language-specific text encoder aligned with the representations generated by the original model. Such methods have been used in the past to create multilingual versions of CLIP (Reimers and Gurevych, 2020; Carlsson et al., 2022). The basic idea is to fine-tune a language model using bilin-
gual or multilingual corpora. The original CLIP model generates a vector representation of the English text, while the language model produces a representation for the translation of that text. The difference between these vectors is then used to compute mean squared error (MSE) loss and the language model is optimized to approximate the representations generated by CLIP.
We employed this technique to train Italian and Persian text encoders, using OpenCLIP H/14 (Cherti et al., 2022)2 as the teacher model and XLM-R large (Conneau et al., 2020)3 as the student model. To train the encoders, we collected 10.5 million English images captions, which we then translated to Italian and Persian using publicly available neural machine translation models (Tiedemann and Thottingal, 2020; Khashabi et al., 2021). The dataset for training was obtained from the following three sources:
• English subset of Wikipedia-based Image Text (WIT) dataset (Srinivasan et al., 2021).
• SBU Captions dataset (Ordonez et al., 2011).
• A subset of 7 million English captions from Conceptual 12M (Changpinyo et al., 2021).
For multilingual classification, we use the same scoring procedure as described in the previous section. The only difference is that we replace the original CLIP text encoder with our fine-tuned models.

heading: 2.1.2 Context augmentation 4
text: One way to improve the performance of the described classifier is to expand the textual context with additional phrases associated with the actual meaning of the target word, which is expected to increase the similarity between the context and the correct image. We can do this with lexical databases by finding the sense of the target word and then extracting additional information from the definition of that sense. In our solution, we use multilingual resources available in Extended Open Multilingual WordNet (Bond and Paik, 2012; Bond and Foster, 2013). Specifically, we utilize the following lexical resources:
• For English, we use Princeton WordNet database (Miller, 1995).
• For Italian, we use two lexical databases: one 2https://huggingface.co/laion/ CLIP-ViT-H-14-laion2B-s32B-b79K 3https://huggingface.co/xlm-roberta-large
included in MultiWordNet (Pianta et al., 2002) and another one from EuroWordNet (Toral et al., 2010).
• For all three languages, we employ additional multilingual resources: Wiktionary and the Common Locale Data Repository (CLDR).
Our context expansion procedure works by appending alternative names extracted from a specific word sense, as well as from senses that are linked to it through hypernym, instance hypernym, member meronym, or substance meronym relations. For example, the context andromeda tree is expanded to: andromeda tree, andromeda, japanese andromeda, lily of the valley tree, pieris japonica, shrub, bush.
In order to find the correct sense, we retrieve a list of available senses of the target word from all lexical databases for a specific language and then compare the descriptions of these senses with the context word. The description is constructed from definitions, alternative names, and examples of use of a given sense, as well as senses linked to it by hypernym or instance hypernym relations. In our solution, we implemented two algorithms for matching sense and context:
• Exact matching, which involves finding exact occurrences of the context word in the sense description. The similarity between the context and the description is computed as the number of matched words divided by the total number of words in the description.
• Similarity matching, involving the comparison of word vectors extracted from the word embedding model. In this method, we convert the context word and words from the sense description into their vector representations using multilingual FastText models (Grave et al., 2018). The similarity between context and sense is calculated as the maximum cosine similarity between the representation of the context word and the representations of all the words from the description.
We select the sense with the highest similarity to the context. For English, only exact matching is used. This method, however, has a low recall for languages other than English. For Italian and Persian we use exact matching first, and if no sense is found, we use similarity matching as a fallback method.

heading: 2.1.3 Drawbacks of CLIP-based methods
text: Although CLIP offers high zero-shot performance for the visual word sense disambiguation task, we also noticed certain problems in using this model that we could not fully eliminate. We share our observations below, which may provide suggestions for future research:
• The model is sensitive to images containing text. It also tends to assign high scores to images, which contain the target or context word. For example, for the context blue mood, it assigned the highest similarity to an image showing just the word blue on a blue background.
• The model performs best with images, which directly show the object being described. However, it has trouble modeling more abstract relationships between textual context and image, especially when the context describes non-physical concepts such as emotions, actions, or events.
• The model has a bias toward more commonly used word senses. As a result, in some cases even expanding the context with additional phrases directing the model to the correct prediction does not help, it still chooses the image relating to the more popular meaning of the target word.

heading: 2.2 Wikipedia retrieval module 2
text: Apart from the classifier described in the previous sections, our solution also includes a Wikipediabased retrieval module, which returns an independent set of scores for each context-image pair. To apply this method, we first download publicly available Wikipedia dumps for the languages of interest, and then create BM25 (Robertson et al., 2009) indexes with texts extracted from Wikipedia articles. For each document, we include a set of URLs to images attached to the article. We utilize WIT (Srinivasan et al., 2021) dataset to obtain a mapping between articles and images.
During inference, we use the following procedure to process the data sample, consisting of textual context and a set of images:
1. Full context is used to query the index. In response, we retrieve the top 10 articles sorted by their relevance to the query. If no relevant documents are found, we retry the search using only the target word as a query.
2. We download all the images attached to the retrieved articles. Next, we transform both the down-
loaded images and the images from the sample to their vector representations using the CLIP model.
3. We compare the sets of sample and article vectors using cosine similarity. The final score for each sample image is equal to the maximum of all similarities to the retrieved images.

heading: 2.3 Learning to rank 5
text: The last element of our solution is the learning to rank model (LTR), which leverages the results returned by the other modules of the system to generate the final ranking of images. At the same time, it is the only component, which requires finetuning, as the modules described previously operate in zero-shot mode. Our approach is based on the LambdaMART algorithm (Burges, 2010), which transforms the ranking problem into a pairwise classification task. It uses a loss function that compares the relative ordering of two items, rather than absolute scores. This allows the model to better capture the relative importance of different items in the ranking process.
In our case, each data sample is represented by ten vectors consisting of numerical features, with each vector describing a comparison between the context and one of the sample images. We use the training set provided by the task organizers to optimize the model. The numerical features are computed from the outputs of the CLIP classifier and the Wikipedia retrieval module, as well as calculated from the dataset statistics. In the case of scoring modules, the following features are extracted from each: the score assigned to the image, the average and maximum of the scores assigned to the other images from the same sample, the difference between the current score and the average, the difference between the current score and the maximum. We also include the penalty value p(x) for the image, extracted from the CLIP-based classifier. As for other features, we include the following values in the model 3 :
• Similarity values computed by CLIP between the image vector and the individual word vectors from the sample - separately for the target word and the context word.
• Two frequency-related features, calculated as the logarithm of the number of occurrences of the image and the context word in the entire dataset.
Input features and hyperparameters of the LTR model are detailed in the Appendix.

heading: 3 Experiments and results
text: This section contains a discussion of the official results of the visual word sense disambiguation task. We have also included a description of other variants of our system which were not used in the submitted solution. We conducted post-evaluation experiments using the gold labels provided by the organizers to analyze the results obtained by alternative versions of our approach.

heading: 3.1 Official results
text: The shared task consisted of three language subtasks, and the final ranking of the submitted solutions was based on the average of the results obtained in these subtasks. The primary metric used to evaluate the systems was accuracy, but the organizers also reported mean reciprocal rank (MRR) as an additional metric. 54 teams participated in the shared task. Our solution was ranked third in the main classification and won the Persian language subtask. The results of the top three ranked solutions, the official baseline, and the best results for each language subtask are shown in Table 1.
The team which won the task achieved consistently high accuracy in all three languages, despite not winning on any of the subtasks. The other teams, including us, scored lower in one or more languages. The weak point of our solution was Italian, on which we ranked only 9th, with a difference of 14% accuracy to the winning system. The performance of the best solutions on the Italian subtask turned out to be as high as on English, which was a surprise considering that no training data was available for this language.

heading: 3.2 Post-evaluation results
text: One of the most important components of our approach is the CLIP model, used for both text-to-
image and image-to-image comparisons. In our solution, we employed OpenCLIP H/14 model published by LAION, which until recently was the largest CLIP variant available. In 2023, an even larger G/14 model was released. To study the impact of the selected CLIP version on the accuracy of the whole system, we tested our solution on available OpenAI and LAION models. The results of this experiment are shown in Table 2. Since we do not have multilingual versions of these models, the results presented are for the English subtask only. The conclusions of the experiment are consistent with results on other datasets found in the literature. LAION models achieve significantly higher accuracy than OpenAI models, and larger models outperform smaller ones. The only surprising finding is the weaker performance of the largest G/14 model. It is possible that other hyperparameters of
our system would need to be readjusted in order to achieve the optimal performance for the largest model.
Another aspect of the system we examined is the context expansion method. In the submitted solution, we used a method based on WordNet and other lexical resources. While developing the system, we also explored an alternative technique using a sequence-to-sequence model. We employed the recently released Flan-T5 (Chung et al., 2022) for this task. Our approach was to send the following prompt to the model: What is the meaning of [context]? In response, the model would generate a definition of the given word sense, which we added to the context. The advantage of this approach is that it allows the context to be expanded for every sample, unlike dictionary-based methods which only expand the context with known definitions. The main disadvantage is the quality of the generated answers. In some cases, they were incorrect, which had a negative impact on the accuracy of the system. Therefore, we decided not to include this method in our solution.
Table 2 shows the results of the three context expansion methods. We tested the performance of the approach based on the T5 model and lexical resources. We also tested a hybrid approach, in which we first try to expand the context using WordNet and other databases, and if that fails, we use the T5 model. As we expected, the standalone T5 model turned out to be worse than the lexical method. However, the hybrid approach managed to improve the accuracy of the English subtask over our original solution.

heading: 3.3 Ablation study
text: As part of the experiments, we performed an ablation study to better understand the impact of the various elements on the accuracy of our solution. The results are shown in Table 3.
The experiment involved performing an evaluation on a system in which a specific component was disabled. We disabled the following functionalities: penalties p(x) used in the CLIP-based classifier (no penalties), learning to rank model (no LTR), context expansion (no expansion), and Wikipedia retrieval module (no Wikipedia). We also tested a version of the system stripped of all the above components, based only on the CLIP model (CLIP only). In cases where LTR module is disabled, we instead use a simple heuristic to select the best matching image. We choose the image found by the Wikipedia retrieval module if the value assigned to that image is higher than 0.9, and the value assigned to the all other images is lower than 0.8. Otherwise, we select the highest rated image by the CLIP-based classifier.
As we can see, each of the components we proposed contributed to the performance of the final solution. The simplest version of the system, based only on the CLIP model, performs at least 10% worse than the submitted solution. However, we can also observe that the effect of each functionality varies for different languages. For example, without Wikipedia, the English and Italian subtasks only lose approximately 2% accuracy, while the Persian subtask scores 12% lower. Context expansion is a feature, which has a significant performance impact on each of the three languages.

heading: 4 Conclusion
text: In this paper, we described our solution for the visual word sense disambiguation shared task. Our system was ranked third in the multilingual track and won in the Persian track, one of the three language-specific subtasks. In the publication, we demonstrated how to build a system, which incorporates different approaches to the problem: imagetext embeddings, lexical resources, image and text retrieval. We showed that each of these components can improve the performance of the overall solution. We have also pointed out some drawbacks of our approach, which can be a starting point for creating better methods in the future.

heading: A Hyperparameters and features
text: https://www.sbert.net/examples/training/ multilingual/README.html#training

heading: Future Work
text: We have also pointed out some drawbacks of our approach, which can be a starting point for creating better methods in the future