SemEval-2023 Task 1: Visual Word Sense Disambiguation
Year: 2023

Abstract:
This paper presents the Visual Word Sense Disambiguation (Visual-WSD) task. The objective of Visual-WSD is to identify among a set of ten images the one that corresponds to the intended meaning of a given ambiguous word which is accompanied with minimal context. The task provides datasets for three different languages: English, Italian, and Farsi. We received a total of 96 different submissions. Out of these, 40 systems outperformed a strong zero-shot CLIPbased baseline (Radford et al., 2021). Participating systems proposed different zeroand few-shot approaches, often involving generative models and data augmentation. More information can be found on the task’s website: https://raganato.github.io/vwsd/.

heading: 1 Introduction
text: Word Sense Disambiguation (WSD) is the task of associating a word in context with its intended sense, generally from a pre-defined sense inventory. While there has been significant progress in the last few years (Bevilacqua et al., 2021; Loureiro et al., 2021), mainly powered by progress in language models, WSD has been mainly limited to settings with textual content only. In most settings, WSD uses sense inventories obtained from lexical resources, such as WordNet (Miller, 1998). However, in real-world scenarios, WSD is often associated with other modalities, such as images. The Visual Word Sense Disambiguation (Visual-WSD) task aims at filling this gap: given a word and some limited textual context (often a single word), the task is to select among a set of candidate images the one which corresponds to the intended meaning of the target word. Figure 1 provides a simplified overview of the task.
To make the task more challenging, the dataset is constructed with the following objectives in mind:
(1) at least one image associated with an incorrect meaning of the word is listed among the options; (2) minimal context (one or two words) is provided for the target word to increase the disambiguation difficulty; and (3) other negative examples are sampled from the same domain as the target sense. Objective (1) guarantees that the task pinpoints systems’ abilities in not only identifying the correct class, but also in doing that at the more fine-grained word sense level. Objective (2) ensures a challenging disambiguation setting in which the system is provided with a very limited set of contextual triggers
2227
to identify the intended meaning. This is relevant given the well-known disambiguation issues for image recognition models, as also observed for recent image generation systems, such as DALLE-2 (Ramesh et al., 2022; Rassin et al., 2022). In objective (3), we build a ‘challenge set’ where we source images from the same domain as the domain of the target sense, and include a large number of these incorrect images for disambiguation.
The Visual-WSD task can be viewed from two different perspectives. The task is a fine-grained image classification task (in a constrained setting) where the model needs to have knowledge about the different meanings of an ambiguous word. Alternatively, Visual-WSD is a novel disambiguation task where the sense distinctions are given in the visual modality rather than the conventional textual definitions obtained from lexical sense inventories (Miller et al., 1990; Navigli and Ponzetto, 2012).
Task formulation. Given a target ambiguous word (e.g., coach) and a limited context (e.g. passenger) along with ten images, the Visual-WSD task consists of identifying the most appropriate image for the intended meaning of the ambiguous word. Figure 1 shows a simplified summary of the task with three examples.
Applications. Multimodality is highly intertwined with different aspects of our day-to-day lives. In fact, retrieving and understanding text with textual data is relevant for a myriad of tasks, from object detection to image retrieval (Calixto and Liu, 2017; Gella et al., 2017; Li et al., 2020). Our task enables the in-depth investigation of one aspect often neglected, which is the inherent ambiguity in language (Calabrese et al., 2020a).
Multilinguality. In addition to English, we provide labeled data (context words for given ambiguous targets) for Italian and Farsi. This data serves to analyse the performance of models in languages other than English, and in practical cross-lingual settings where initial training data might not be available for a given language.

heading: 2 Data
text: In this section we detail the data collection and annotation process carried out for the training and testing data. We then provide some statistics on the datasets.

heading: 2.1 Data sources
text: The data for this task is mostly obtained from Wikidata,1 OmegaWiki,2 and BabelPic (Calabrese et al., 2020b). Wikidata and OmegaWiki are both collaborative projects to produce a free and open knowledge base and dictionary respectively. Each article, in both resources, can be considered as a concept (or named entity) for which one or multiple related images are provided. BabelPic provides images associated to BabelNet (Navigli and Ponzetto, 2012), a large multilingual encyclopedic dictionary, about non-concrete or abstract concepts.3 We use BabelNet as a bridge to link the three resources: Wikidata, OmegaWiki and BabelPic. In the following section we describe the semi-automatic process to construct the Visual-WSD dataset.

heading: 2.2 Construction procedure
text: Each instance in the dataset consists of a target word in the context of one or two trigger words, associated with 10 different images: one image corresponding to the intended meaning, and the others referring either to the other meanings of the ambiguous target word, to similar words from the same domain, or other randomly-selected concepts from the used resources.
Training data. We provide silver training data in English by leveraging BabelNet semantic network structure. Specifically, we first collect a list of senses, either ambiguous or monosemous, belonging to the WordNet portion of BabelNet, together with their associated picture provided by BabelNet. Context words are provided based on each concept hypernym (e.g., chef cook as the context for the word chef ). Moreover, to avoid human faces as potential target images, we also filter out senses denoting people through the associated WordNet category.4 This training data is intended for silver data only.
Testing data. We provide gold testing data in English, Farsi and Italian. In detail, we first collect a list of ambiguous word senses (the potential target words) with their respective images and definitions from BabelNet. Then, for each word sense, we ask an annotator to provide one or two trigger words (the context) that are enough to identify the
1https://www.wikidata.org/ 2http://www.omegawiki.org/ 3For our task, we use the gold version only. 4https://wordnet.princeton.edu/
documentation/lexnames5wn
intended meaning of the word sense when considering the association between definition and image. These trigger words were also selected to be challenging enough as not to give away the meaning of the image in isolation (i.e., the target word is generally necessary to understand the full context). This step is to ensure a challenging text disambiguation task. In the case of English, annotators were proficient English speakers, including the authors of this paper, while in the case of Farsi and Italian, native speakers performed the task. To build the negative sample images, we pick them in a manner to ensure we mitigate model bias to unwanted dataset artifacts. To this end, we construct an extension set containing both a set of random images and images that correspond to words other than the target that belong to the same domain (e.g., in Figure 1 we add “zebra” for the target word “crane”). These domains, referred to as BabelDomains (CamachoCollados and Navigli, 2017), extend the Wikipedia featured article page domains5 to cover most WordNet and Wikipedia. Finally, for each previously validated instance, a human annotator selects nine images from the extension set of candidate images, fulfilling the criteria of selecting related images but different from the target one.

heading: 2.3 Statistics
text: Table 1 shows the number of instances in each split, trial, training, and test. The trial data provided with a few gold examples only, was used for early development purposes during the initial phase of the shared task. Similarly to the test set, trial instances were validated by annotators. English is the largest dataset, including more than 12 thousand silver training instances, and 463 gold test annotations. Concerning Farsi and Italian languages, we provide gold testing data only, with 200 and 305 annotations respectively.
5https://en.wikipedia.org/wiki/ Wikipedia:Featured_articles

heading: 3 Evaluation
text: We first introduce the metrics that are used to evaluate the participating systems (Section 3.1). Then, we provide a brief description of the baseline system (Section 3.2) as well as the participting systems (Section 3.3), and present the overall results of the task (Section 3.4).

heading: 3.1 Evaluation metrics
text: We use mean reciprocal rank (MRR) and hit rate at 1 (HIT@1) as the evaluation metrics. Given r = [r1, . . . , rn] as the image ranking predictions provided by a given system, MRR is defined as:
MRR = 1
n
n∑
i=1
1 ri (1)
where n is the number of images. HIT@1 is the ratio of instances ranking the true image as first, that is defined as:
HIT@1 = 1
n
n∑
i=1
11(ri) (2)
where 11(r) is an indicator function that returns 1 if r is 1, and otherwise 0. HIT@1 can also be viewed as accuracy when only one prediction is provided.

heading: 3.2 Baselines
text: As a baseline for the Visual-WSD task, we use CLIP (Radford et al., 2021), a recent languagevision multi-modal embedding model. We compare the CLIP embedding of the query phrase with the CLIP embedding of each candidate image, and the candidate image with the highest cosine similarity to the query is considered as the prediction. We use the original CLIP for English and multilingual CLIP released by Sentence Transformers (Reimers and Gurevych, 2019) for non-English, where the model’s weights are shared via HuggingFace (Wolf et al., 2020).6

heading: 3.3 Participating systems
text: Most of our participants’ submissions used pretrained vision-and-language models in a zero-shot
6Specifically, we used the openai/ clip-vit-large-patch14-336 model for CLIP and sentence-transformers/ clip-ViT-B-32-multilingual-v1 for multilingual CLIP. The code to reproduce the baseline can be found at https://github.com/asahi417/ Visual-WSD-baseline.
setting, similarly to our baseline (i.e., CLIP). A common approach across submissions involved using generative models and data augmentation. For text inputs, participants used pretrained language models to generate sentences that include the context words, and back-translation (Sennrich et al., 2016) for data augmentation. In the multimodal setting, participants used pretrained text-to-image models (Rombach et al., 2021) to generate candidate images, and used measures of distance between the generated images and the candidate images as a way to find the correct image for some text input. Finally, participants have also used lexical knowledge graphs such as WordNet to enrich input text with sense information, as well as external resources such as Wikipedia as a source from which to retrieve sentences.
In the following, we describe the top four performing systems in more detail, which include the top-performing models for each individual language.
TAM of SCNU (Yang et al., 2023) This system is based on a Fine-grained Contrastive LanguageImage Learning (FCLL) model that learns finegrained image-text knowledge through a new finegrained contrastive learning mechanism. The contextual information is enriched by establishing a relationship between concepts and sentences. Finally, this model benefits from a newly constructed multilingual and multimodal knowledge base for ambiguous words. This system achieved the top overall result in the task with an average 72.56 accuracy (or Hits@1) performance, which highlights its robustness across languages.
Samsung Research China - Beijing (Zhang et al., 2023b) In this model, definitions and synonyms of the target word are collected from WordNet, BabelNet, Wikipedia and vocabulary.com to build the reference sense inventory. Moreover, images of phrases are collected from the LAION open dataset (Schuhmann et al., 2021) using CLIP retrieval (Cherti et al., 2022). Then, the most suitable definition from the sense inventory is selected using a biencoder architecture with SimCSE (Gao et al., 2021) as the backbone. The matching model is a large version of CLIP trained on LAION-2B from Open CLIP. For Farsi and Italian, senses are directly translated from English. This system achieved the overall second best performance and the best result by a substantial margin in the English test set.
zywiolak (Dadas, 2023) This hybrid system combines multimodal embeddings and knowledgebased approaches. The main classifier is based on the CLIP model, whose results are enriched with additional information retrieved from Wikipedia and lexical databases. The various modules of the system are integrated using a learning to rank (LTR) model. This model takes as input a feature vector
describing the results of the individual components of the system and outputs a relevance ranking of candidate images. This system ranked third overall, while being the top performing system in Farsi.
Rahul (Patil et al., 2023) This system presents an ensemble of different neural models. First, CLIP models are used for English, with text-to-text translation models for Farsi-to-English and Italian-toEnglish. Additionally, this system integrates multilingual BERT-base embeddings (Devlin et al., 2019) for text and ResNet101 embeddings (He et al., 2016) for the image. This system ranked fourth overall in terms of accuracy (Hits@1) and first in the Italian test set.

heading: 3.4 Results
text: Table 2 shows the overall results (averaged over the three languages, English, Italian and Farsi) of all participating systems. Across the board, systems in the top block, i.e. top five submissions, achieve better results than all the others. Tables 3, 4, 5 show
language-specific performances for each submission (English, Farsi and Italian, respectively). As can be seen, although top systems for English and Italian achieve similar results, their performance appears to be relatively weaker in managing Farsi data.

heading: 4 Conclusions
text: In this paper, we presented the first task on Visual Word Sense Disambiguation. Instead of the usual single-modality tasks of textual word sense disambiguation and image recognition, our proposal merges these two paradigms into a single unified task. In particular, WSD is reframed by replacing the usual sense inventory based on lexical resources by a dynamic inventory based on images. The task received 96 submissions. The evaluation showed promising results with 40 submissions outperforming a very competitive image recognition baseline based on a zero-shot CLIP model for different languages. Nonetheless, given the recency and challenging nature of the task, there is clear room for improvement for future work.

heading: Future Work
text: Nonetheless, given the recency and challenging nature of the task, there is clear room for improvement for future work