UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation
Year: 2023

Abstract:
We describe the systems of the University of Alberta team for the SemEval-2023 Visual Word Sense Disambiguation (V-WSD) Task. We present a novel algorithm that leverages glosses retrieved from BabelNet, in combination with text and image encoders. Furthermore, we compare language-specific encoders against the application of English encoders to translated texts. As the contexts given in the task datasets are extremely short, we also experiment with augmenting these contexts with descriptions generated by a language model. This yields substantial improvements in accuracy. We describe and evaluate additional V-WSD methods which use image generation and text-conditioned image segmentation. Overall, the results of our official submission rank us 18 out of 56 teams. Some of our unofficial results are even better than the official ones. Our code is publicly available at https://github.com/ UAlberta-NLP/v-wsd.

heading: 1 Introduction
text: This paper addresses our work on SemEval-2023 Task 1: Visual Word Sense Disambiguation1 (Raganato et al., 2023). The V-WSD task is closely related to WSD, and similarly involves understanding and classifying the meaning of a polysemous word in context. The distinction is in how classes are defined: In WSD, a system has access to a sense inventory that enumerates the possible senses of each word, and the task is to classify the focus word according to the sense that best corresponds to its intended meaning. In V-WSD, a system is given a set of candidate images, and the task is to select the image which depicts the intended meaning of the focus word.
The multi-modal nature of V-WSD introduces challenges not encountered in WSD. First, image
1https://raganato.github.io/vwsd/
processing is generally more computationally intensive than text processing. Second, a V-WSD system must represent the meanings of both images and text, and must have mechanisms to compare these multi-modal semantic representations. Last, since the candidate images in V-WSD are not restricted to a sense inventory, they may exhibit highly variable levels of sense granularity.
The V-WSD task is motivated by cases where textual context alone is insufficient to disambiguate a word. In such cases, visual context may be available to facilitate disambiguation. For example, the word play is ambiguous in the context “that was a good play,” as it may refer to a theatrical performance or an action in a sport. However, an associated image of a stage or a sports field would enable a V-WSD system to disambiguate play.
We propose a novel V-WSD algorithm that ranks candidate images by embedding images and wordsin-context in a shared semantic space, while also taking advantage of lexical knowledge bases commonly used in WSD. In particular, our method uses sense glosses of the focus word to create representations of the possible meanings that word may have. Our algorithm is flexible, It includes several optional modules, as well as hyper-parameters that facilitate customization, optimization, and detailed analysis.
We test various configurations of our method and analyze their performance. Our three principal conclusions are as follows: First and foremost, the
2043
augmention of the original textual context plays a crucial role in improving performance. Second, there is a considerable gap between English and non-English performance, indicating that bias towards English models extends to the multi-modal setting. Third, we observe a major distribution shift between the train and test sets, which is confirmed by our ablation study.

heading: 2 Related Work
text: Recent work on WSD can be divided into supervised and knowledge-based systems. Supervised WSD methods depend on large training corpora in which some or all of the content words have been tagged with their correct senses (Blevins and Zettlemoyer, 2020; Barba et al., 2021). Knowledgebased methods depend on other sources of linguistic knowledge (Wang and Wang, 2020). In general, knowledge-based methods are outperformed by contemporary supervised methods (Pasini et al., 2021). Today, state-of-the-art WSD systems approach accuracy limits imposed by inter-annotator agreement (Maru et al., 2022).
Research on the incorporation of visual information for WSD is relatively sparse. Barnard et al. (2003) propose a statistical model that associates image regions and words to predict word senses. Loeff et al. (2006) apply spectral clustering to group similar images corresponding to the same senses. Saenko and Darrell (2008) employ an unsupervised approach to assign senses to images using surrounding texts and dictionary definitions, and then train a visual SVM classifier to disambiguate unseen images. Gella et al. (2019) introduce the task of visual verb sense disambiguation, in which one image is selected based on a given context. Vascon et al. (2021) propose a graph-based semisupervised transductive learning method for visual verb sense disambiguation.
Multi-modal foundation models (Bommasani et al., 2021) such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) can represent both text and images in a shared embedding space. Recent work2 (Bianchi et al., 2021; Sajjad Ayoubi, 2022) improves the text encoder by using a pretrained text-only encoder such as BERT (Devlin et al., 2018).
2https://github.com/moein-shariatnia/ OpenAI-CLIP

heading: 3 Task & Dataset
text: Task Definition: Given a focus word w in a short context c, and a set of candidate images I , the task is to select the image i∗ ∈ I which best represents the meaning of w in c. For example, given the context “baseball bat” with bat as the focus word, a V-WSD system should choose the image that depicts the bat used in baseball (Figure 1).
Dataset: The training data provided for this shared task consists of a silver dataset with 12,869 V-WSD instances. Each sample is a 4-tuple ⟨f, c, I, i∗ ∈ I⟩ where |I| = 10. The contexts are generally very short, often just a single word in addition to the focus word. We randomly select 10% of the training data for use as a development set. The test dataset consists of 968 instances, of which 463 are English, 200 are Farsi, and 305 are Italian. We observe that many of the incorrect candidate images in the training data have nothing to do with any sense of the focus word. However, in the test data, we observe that this is less often the case, making the test set considerably more difficult.
Evaluation Metrics: The primary metric is the hit rate, which is equivalent to top-1 accuracy, or simply accuracy. This is the proportion of instances for which the system selects the correct image. We also compute the mean reciprocal rank (Voorhees and Tice, 2000) which represents how highly VWSD systems rank the ground-truth image, on average.
Language Distribution: We observed some instances where the context contained non-English words. To estimate the prevalence of this phenomenon, we randomly selected 100 instances from the training set and manually identified the language of each. For example, the focus word shorea in “shorea genus” comes from new Latin, and refers to a genus of mainly rainforest trees. Table 1 shows the frequency of each language in our sample.

heading: 4 Method
text: In this section, we describe the key components of our systems, including an algorithm that combines text and image similarity measures.

heading: 4.1 Algorithm
text: We propose an algorithm to select a single image from a set of candidates that best matches the context. To reiterate the problem, we are given a context c containing a focus word w and a set I of candidate images. We assume that we also have a non-empty set G containing possible glosses of w; in practice, we obtain G from BabelNet using the freely available API.3
Our algorithm makes calls to two similarity functions: The first is simL, a written language similarity function, which takes as input two text strings and returns a value indicating the semantic similarity between them. The second is simV L, a vision-to-written language similarity function, which takes as input an image and a text string and returns a value indicating the similarity between what the image depicts and what the text describes.
With these functions, for each candidate image i ∈ I , and for each gloss g ∈ G of the focus word w, we compute the pairwise similarity between:
1. The image and context: sic = simV L(i, c)
2. The image and gloss: sig = simV L(i, g)
3. The context and gloss: scg = simL(c, g)
This allows us to identify the pair of a candidate image i∗ and gloss g∗ that maximizes a weighted average of these three similarity scores. Algorithm 1 shows the pseudocode for this algorithm.
Hyperparameters: Our algorithm depends on three weight hyperparameters: wic, wig, and wcg. They represent the weights for image-context, image-gloss, and context-gloss similarity, respectively. Table 2 shows the results of the hyperparameter binarized grid search performed on a 500- sample of the training set. Based on our development experiment results, we decided to set all hyperparameter weights to 1 for simplicity, except where otherwise noted. We discuss the hyperparameters further in Section 6.6.
3https://babelnet.org/guide
Algorithm 1 Candidate Image Scoring 1: c← the context of the focus word 2: G← list of glosses for the focus word 3: I ← list of candidate images 4: for i in I do 5: sg ← 0 6: for g in G do 7: sig ← wig · simV L(i, g) 8: scg ← wcg · simL(c, g) 9: sg ← max(sg, sig + scg)
10: scores[i]← sg + wic · simV L(i, c) 11: return scores
Context Augmentation: For each instance, we prompt InstructGPT (Brown et al., 2020; Ouyang et al., 2022) to generate a definition for the context phrase. We use the following prompt template: “For each line, define the phrase:” followed by the contexts, one per line. For example, the context “baseball bat” is augmented to become “baseball bat: a bat used to hit a baseball during the game of baseball.” The use of this additional context is described in Section 5.3
Supplementary Training Data: We speculate that the size of the training dataset may be a limiting factor in the accuracy of our method. We, therefore, experiment with augmenting the training data with additional data derived from BabelPic Calabrese et al. (2020), a multi-modal resource which maps a subset of BabelNet synsets to sets of one or more images. For each pair of a synset and an image, we enumerate a lemma from the base synset and a lemma from a related synset. The two lemmas are concatenated, starting with the lemma from the related base synset, to form a two-word context. We then select nine other random images from BabelPic, forming an instance comparable to those in the training set: a two-word context with a
single focus word, with ten images, one depicting the correct sense of the focus. We create 54,968 instances this way and experiment with adding this dataset to the training data at training time.
Glosses: For each instance, we enumerate the BabelNet (Navigli and Ponzetto, 2012) glosses corresponding to each sense of the focus word. If there are multiple glosses for a single sense, we pick the first and add it to the set G. This prevents senses from being over-represented due to the number of glosses in BabelNet.

heading: 5 Systems
text: In this section, we describe our systems for the V-WSD task, Our official system submissions are based on our primary systems: TR and LANGSPEC. We also describe two alternative systems, which do not use Algorithm 1. Both perform worse than the primary systems, but their results are nevertheless valuable for the purpose of analysis. We also present a supplementary method, which can be optionally used in combination with our other systems. Non-English instances are translated using DeepL4 for Italian and ChatGPT5 for Farsi.

heading: 5.1 Primary Systems
text: TR: Image Scoring with Translations If the input instance is not English, we translate it into English. Then we apply Algorithm 1. We compute simV L using embeddings from CLIP (Radford et al., 2021), an English-only model which encodes text and images in a shared embedding space. We compute simL using BERT (Devlin et al., 2018) as an English-only text encoder. We set the weight parameters: wic, wig, and wcg to 1 in this specific case.
LANGSPEC: Image Scoring with LanguageSpecific Models This system is similar to TR, except that non-English instances are not translated into English. This is our only system which directly operates in other languages. Given a nonEnglish instance, we replace CLIP and BERT with language-specific models to compute simV L and simL. For English instances, this method is the same as TR. For Italian, we use CLIP-Italian (Bianchi et al., 2021) to compute simV L and Italian BERT6 to compute simL. For Farsi, we use
4https://www.deepl.com/translator 5https://openai.com/blog/chatgpt/ 6https://huggingface.co/dbmdz/
bert-base-italian-xxl-uncased
CLIPfa (Sajjad Ayoubi, 2022) to compute simV L and ParsBERT (Farahani et al., 2021) to compute simL.

heading: 5.2 Alternative Systems
text: GEN: Generative Image Model This method takes a different approach compared to TR and LANGSPEC; it does not use Algorithm 1. Instead, we provide the context (translated into English, if needed, as outlined above) as input to Stable Diffusion (Rombach et al., 2022), a generative model which takes a text prompt as input and produces candidate images to depict what the text describes. For each context, we generate 15 images using 20 diffusion steps each. We set the guidance scale hyperparameter to 7.5. For each candidate image, we compute its cosine similarity with each generated image based on embeddings produced by CLIP. The candidate with the highest similarity to the generated images is chosen as the output.
SEG: Text-Conditioned Image Segmentation As with GEN, this method does not use Algorithm 1. Instead, we use a zero-shot image segmentation system (Lüddecke and Ecker, 2022) to segment images based on the provided context. This system produces a mean mask value, which we use as a measure of similarity between the context and the segmented image; we return the image with the highest mean mask value, given the context.

heading: 5.3 Supplementary Method
text: DEF: Generating Additional Context TR, GEN, and SEG make use of the input context, translated to English as needed. However, the contexts provided in the official dataset for this task are extremely short. With DEF, we generate additional context by using the original context to prompt InstructGPT for a more extensive description, as described in Section 4.1. We then concatenate the generated text to the context and pass this augmented context to TR, GEN, or SEG. We refer to the methods using this supplementary method as TR+DEF, GEN+DEF, and SEG+DEF, respectively. We do not combine DEF with LANGSPEC, as we observe that InstructGPT is less robust to short non-English contexts.
For TR+DEF, we set wig and wcg to 0, as the improved context obviates the need for their corresponding terms in Algorithm 1. GEN+DEF and SEG+DEF, being based on GEN and SEG, do not depend on Algorithm 1.

heading: 6 Experiments
text: In this section, we present, discuss, and analyze our results.

heading: 6.1 Results
text: Table 3 shows our performance on the test set. We find that accuracy has a 99.46% Pearson correlation with mean reciprocal rank, and so for conciseness, we report accuracy alone. The translation-based systems, TR and TR+DEF, yield the best results. One explanation for this outcome is the disproportionate amount of English training data available to the models we build upon: CLIP and BERT. The higher performance of these models on English appears to compensate for the noise introduced by the translation process. We discuss this further in Section 6.4.
An interesting trend is the benefit of context augmentation, (Section 5.3). Between TR and TR+DEF, we observe a 3% average improvement in accuracy. We observe a similar trend in GEN versus GEN+DEF and SEG versus SEG+DEF.
We further observe that accuracy on English instances is highest, accuracy on Farsi instances is lowest, while accuracy on Italian instances is in between both. This corresponds to the quality and quantity of resources available for each language. We undertake more thorough analyses in the next section.

heading: 6.2 Distribution Shift
text: As shown in Table 4, we observe a clear disparity in polysemy, and the proportion of focus words which are nouns, between the training and test sets. This difference is especially notable when considering the performance gap between the sets.
Zero-shot vs. Fine-tuning: We observe that finetuning on the training set leads to a drop in performance on the test set (Figure 2). This may be due to the divergence between the training and test datasets outlined above.

heading: 6.3 Traditional WSD
text: Although both the V-WSD and WSD tasks have some similarities, we found that some ideas drawn from WSD prove ineffective for V-WSD.
Using Glosses: We observe empirically worse performance when using glosses in our algorithm. Specifically, with TR on the English test set, we obtain a hit rate of 61.1% when we do not use glosses and 56.8% when we do. Such a steep drop (4.3%) is surprising, especially since most state-ofthe-art WSD systems explicitly use glosses in their methods.
We posit that sense disambiguation in V-WSD is more focused on homonymy than polysemy and, as a result, can be less nuanced than in WSD. For example, apple could refer to a fruit or a tree. In an image depicting both, the focus may be unclear. In WSD, this distinction is critical since tree and fruit are distinct senses. In V-WSD, however, we can make a correct prediction without deciding between both senses. As a result of this lower granularity, glosses become less important.
Performance of WSD Systems on Context: We manually disambiguated the sense of the focus word in a randomly-selected set of 16 instances from the training set. We then applied a state-ofthe-art WSD system, ConSec (Barba et al., 2021), to these instances. We observe that ConSec sense predictions were accurate 50% of the time, falling considerably below its reported accuracy of 82%.

heading: 6.4 English Hegemony
text: Natural language processing research often focuses on the English language, at the expense of other languages (Magueresse et al., 2020). The relative performance of TR and LANGSPEC reflects this phenomenon: Translating non-English text to English, in order to apply an English encoder, can be expected to introduce some noise due to translation errors and information loss. However, we observe that this pipeline approach produces better results than using an Italian or Farsi encoder directly. This suggests that the field’s focus on English has yielded English encoders which are much better than those available for Italian or Farsi. We speculate that advancing the state-of-the-art for non-English encoders may yield even better performance, by avoiding the need to translate to English.

heading: 6.5 Image Generation
text: As shown in Figure 3, when applying our image generation system (GEN), we observe an increase in performance as we generate more images. Although the performance jump when transitioning from 1 to 5 images is most pronounced, we see benefits from scaling until a certain point, 10 images, where the trend becomes unreliable.

heading: 6.6 Text-Conditioned Image Segmentation
text: With SEG, we can sometimes robustly segment images and predict masks indicating the correct
image, conditioning on the full context. However, this method sometimes forms incorrect semantic representations. Appendix A details more examples of SEG’s usage. In addition to Figure 4, we present more extensive examples in Appendix A.

heading: 6.7 Algorithm Hyperparameters
text: Our algorithm uses three weights hyperparameters to balance pairwise similarities. We set all weights to 1 based on Table 2. Comparison of results with wcg set to 0 or 1 suggests that simL(c, g) does not improve performance. Two reasons support this finding. Firstly, images encode richer representations, producing more precise simV L(i, g) and simV L(i, c), while both context and glosses are discrete textual features, introducing uncertainty to simL(c, g). Secondly, we use CLIP and BERT to calculate simV L and simL, respectively. CLIP’s multi-modal pre-training may offer better similarity scores, fitting this task better. Understanding these findings more deeply is an interesting avenue for future research.

heading: 7 Conclusion
text: In this paper, we outlined our work on the recently proposed task of V-WSD. We found that many ideas from traditional WSD are difficult to adapt to V-WSD, and, moreover, WSD systems are generally not useful for V-WSD. We were particularly surprised to find that, unlike in WSD, glosses appear to be unhelpful for V-WSD. Contrariwise, our innovation of augmenting the context did yield substantial gains in accuracy.
Further research will be needed to establish the connection between V-WSD and the broader field of lexical semantics. We speculate that developing systems for joint WSD and V-WSD may yield improvements in one or both tasks. Our work here serves as a proof-of-concept establishing the utility of language models and lexico-semantic resources in the developing task of V-WSD.

heading: Acknowledgements
text: This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), and the Alberta Machine Intelligence Institute (Amii). We also thank Behzad Shayegh for providing advice on the Farsi test set.

heading: A Text-Conditioned Image Segmentation
text: A.1 Success Mode In the successful case of this system, we see that we are able to segment the object based on the text provided properly. See the figures below for details.
A.2 Failure Mode In the failure case of this system, we see that we are unable to confidently segment the object based on the text provided. See the figures below for details.

heading: Future Work
text: Understanding these findings more deeply is an interesting avenue for future research