ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation
Year: 2023

Abstract:
Successful word sense disambiguation (WSD) is a fundamental element of natural language understanding. As part of SemEval-2023 Task 1, we investigate WSD in a multimodal setting, where ambiguous words are to be matched with candidate images representing word senses. We compare multiple systems based on pre-trained CLIP models. In our experiments, we find CLIP to have solid zero-shot performance on monolingual and multilingual data. By employing different fine-tuning techniques, we are able to further enhance performance. However, transferring knowledge between data distributions proves to be more challenging.

heading: 1 Introduction
text: Determining the correct meaning of an ambiguous word in a given textual context is a fundamental element of successful understanding of natural language. Formulated in the task of Word Sense Disambiguation (WSD), it remains a key challenge of Natural Language Processing (NLP). Recent successful approaches tackling this task often leverage large pre-trained language models (Bevilacqua et al., 2021). With the rise of large pre-trained vision-and-language (V&L) models such as CLIP (Radford et al., 2021), the question arises how well these models can tackle WSD in a multimodal setting.
Towards answering this question, Task 1 of SemEval-2023 proposes a visual WSD task (Raganato et al., 2023), combining ambiguous contextualized words with images encoding different word senses. Instead of selecting the correct word sense from a sense inventory, a system now has to retrieve the best matching image from a set of candidate images, thereby bridging the gap between vision and language.
In this paper, we present our contributions to the SemEval-2023 Visual-WSD task. Our work
∗* equal contribution
focuses on empirically comparing the WSD performance of multiple CLIP models in different transfer learning settings. This includes the investigation of zero-shot transfer and adapters (Rebuffi et al., 2017; Houlsby et al., 2019) as a parameter-efficient means of transfer learning. We especially aim to highlight to which degree simple techniques are sufficient for successful task performance. Our best submitted system ranks 9th out of 54 on the official task leaderboard (counting only one submission per person).

heading: 2 Background

heading: 2.1 Word Sense Disambiguation (WSD)
text: WSD is the task of determining the exact meaning of an ambiguous word in a given context. Typically, this is achieved by selecting the most suitable word sense from a pre-defined, static word sense inventory such as WordNet (Miller et al., 1990). Existing solutions for this task mainly follow two approaches: Knowledge-based approaches exploit the knowledge encoded into lexical resources, such as the graph structure of WordNet. Supervised approaches aim to learn a parameterized model, such as a neural network, that directly predicts the correct sense s given a word w and a context c in a classification setting (Hadiwinoto et al., 2019). Training these models in a supervised setup requires large sets of word-context pairs annotated with correct sense labels. One such dataset is SemCor (Miller et al., 1993), consisting of 200,000 sense annotations based on the WordNet sense inventory. Recent state-of-the-art WSD systems can often be described as a combination of knowledgebased and supervised approaches. E.g., EWISER (Bevilacqua and Navigli, 2020) combines a pretrained BERT encoder (Devlin et al., 2019) with a synset graph extracted from WordNet. ESCHER (Barba et al., 2021) on the other hand augment the word-context pair with textual definitions for all
1463
possible senses from a sense inventory.

heading: 2.2 SemEval-2023 Visual-WSD Task
text: Existing work on WSD almost unanimously focuses only on the text domain. The SemEval-2023 Visual-WSD task (Raganato et al., 2023) extends the scope of WSD by formulating it as a multimodal V&L problem. Similar to the classical WSD formulation, a pair (w, c) consisting of an ambiguous word w and a short textual context c is given as input. The goal now is to select an image from a set of candidates. More concretely, from a set of ten candidate images {I0, . . . , I9}, a system has to select the image Iy that best represents the intended meaning of w given c.
The Visual-WSD task provides a labeled English-language training dataset consisting of 12,869 (w, c) pairs along with 12,999 unique images representing word senses. The test split contains samples in three languages, including English (463 samples), Italian (305 samples) and Farsi (200 samples). Samples of all three languages share a set of 8100 test images.
Performance of WSD systems on the VisualWSD task is measured using two metrics. Accuracy (or hit rate at 1) is used to measure in how many cases a system correctly ranks Iy as the most likely candidate. Mean reciprocal rank (MRR) is used to evaluate the overall ranking quality of a system by taking into account at which position it ranks Iy.

heading: 3 Methodology

heading: 3.1 CLIP
text: CLIP (Radford et al., 2021) has been proposed as an approach to learn multimodal V&L representations by pre-training on a large-scale dataset of image-text pairs. The CLIP architecture follows a "late interaction" design, where image and text are encoded independently by a vision encoder V and a text encoder T . Both encoders are jointly optimized during pre-training to maximize the cosine similarity between the embeddings of both modalities for matching image-text pairs.
While Radford et al. (2021) show that CLIP has strong zero-shot capabilities on downstream tasks, subsequent work such as CLIP-ViL (Shen et al., 2021) incorporates the frozen CLIP vision encoder into existing architectures for various visionlanguage tasks. We directly leverage both pretrained CLIP encoders for transfer to the downstream task. As the formulation of the Visual WSD
task is similar to the pre-training task of CLIP, our fine-tuning objective closely follows CLIP’s pretraining objective. Instead of passing N imagetext pairs as done during pre-training, our input consists of one (w, c) pair alongside 10 images. The symmetric cross entropy loss employed during pre-training therefore collapses to a regular cross entropy loss (CE) over the candidate images. In simplified form, the full fine-tuning objective therefore can be formulated as:
L = CE(V ([I0, . . . , I9]) · T ([w, c]), y)
where · denotes the dot product. In our work, we select pre-trained checkpoints with Transfomer-based models (Vaswani et al., 2017) for V and T .

heading: 3.2 Adapter Methods
text: Adapters have been introduced to computer vision (Rebuffi et al., 2017) and NLP (Houlsby et al., 2019) as a parameter-efficient alternative to full fine-tuning for transfer learning. Initially focused on simple feed-forward bottleneck modules, the scope of parameter-efficient transfer learning methods has recently broadened to include a wide range of methods (Li and Liang, 2021; Hu et al., 2022; He et al., 2022; Liu et al., 2022) and extensions to new domains such as V&L tasks (Sung et al., 2021; Zhang et al., 2022; Lu et al., 2023). We refer to all of these approaches as adapter methods.
All adapter methods have in common that they introduce a small number of new parameters to specific locations in a pre-trained neural network. During training on the downstream task, only the newly introduced parameters are updated while all pre-trained model weights are kept frozen. In this work, we consider the following methods: Bottleneck adapters introduce bottleneck feedforward layer modules in each Transformer layer. They consist of a down-projection into a lower dimension, a non-linearity, an up-projection that projects back into the original hidden layer dimension and a residual connection. We follow the approach of Pfeiffer et al. (2021) in adding adapter modules sequentially after the feed-forward component of each Transformer layer. Low-Rank Adaptation (LoRA) (Hu et al., 2022) injects trainable low-rank decomposition matrices into Transformer layers. For a weight matrix W0, LoRA performs a re-parametrizations using lowrank matrices A ∈ Rr×k and B ∈ Rd×r of rank r such that W = W0 +BA. Following the original
implementation, we add LoRA to the query and value matrices of each Transformer self-attention layer.

heading: 3.3 Intermediate Task Transfer
text: Due to their modular nature, the parameters of multiple adapters can be composed to leverage the knowledge from multiple downstream tasks. In our work, we study whether enriching our visual WSD system with knowledge learned from a text-only WSD intermediate task s benefits final performance on the target task t. To achieve this, we employ sequential fine-tuning of adapters (Poth et al., 2021; Zmarsly, 2022), which first optimizes adapter parameters on s before subsequently fine-tuning the same parameters on t.
For s, we choose the SemCor dataset (Miller et al., 1993), which provides (w, c) pairs annotated with the sense key k best describing w in its context. We train a bottleneck adapter on s by letting a multiple choice model choose k from a list of 10 possible sense keys and transfer its parameters to our models for t.

heading: 4 Experimental Setup

heading: 4.1 Model Checkpoints
text: We evaluate the following pre-trained models:
OpenAI CLIP denotes the original model checkpoints provided by Radford et al. (2021). These checkpoints are pre-trained on 400M image-text pairs collected from the Internet by the CLIP authors. For our experiments, we select the model variants using Vision Transformer (ViT) (Dosovitskiy et al., 2021) as the vision encoder. Specifically, we select the variants with ViT-B/32 and ViT-L/14336 for zero-shot evaluation.
LAION CLIP denotes models pre-trained in an open-source replication of the original CLIP models (Cherti et al., 2022). The best performing models of this family are pre-trained on LAION-2B, the English subset of the LAION-5B dataset (Schuhmann et al., 2022), consisting of 2.3B English language image-text pairs collected from Common Crawl. We select multiple checkpoints using ViT as vision encoder for evaluation, including ViTB/32, ViT/L-14, ViT-H/14 and ViT-G/14. As of writing this paper, ViT-G/14 has the best zero-shot performance of all publicly available CLIP models1.
1https://laion.ai/blog/giant-openclip/
LAION Multilingual CLIP denotes multilingual variants of LAION CLIP pre-trained on the full LAION-5B dataset consisting of 5.85B image-text pairs in over 100 languages. These models use XLM-R (Conneau et al., 2020), a pre-trained multilingual language model, as text encoder. In our experiments, we select two checkpoints, one using ViT-B/32 and XLM-R Base and one using ViTH/14 and XLM-R Large.

heading: 4.2 Transfer Learning Methods
text: We compare multiple transfer learning methods on the pre-trained CLIP checkpoints. In increasing order of parameters updated during application on the Visual WSD task, these include: 1) Zero-shot transfer, i.e. evaluation on the downstream task without any training. 2) Projection layer tuning, i.e. fine-tuning of the final projection layers on top of CLIP’s vision and text encoders while keeping all other parameters fixed. 3) Adapter methods as described in § 3.2. 4) Partial fine-tuning, i.e. finetuning the full text encoder while keeping the vision encoder frozen. 5) Full fine-tuning, i.e. fine-tuning all CLIP parameters on the downstream task.
Their number of trainable parameters is compared in Table 1.

heading: 4.3 Training Details
text: To evaluate our models and compare different approaches during training, we keep a fixed set of 1286 (w, c) pairs (10%) from the training dataset, leaving us 11583 pairs to actually train models on.
In the provided datasets, each context contains the word it is paired with, making it sufficient to only use the tokenized context as input for our text encoder models without further preprocessing. We limit all tokenizers to a maximum sequence length of 20, which is enough to represent all samples from the different datasets.
Preprocessing of the input images consists of rescaling them to have a shorter side of 224 pix-
els, cutting out the center to form a 224x224 pixel square, and normalizing them. To save disk space and unnecessary operations during training, we rescale all images beforehand.
Full models are fine-tuned for 5 epochs, using an initial learning rate of 10−6. Adapters are trained for 30 epochs and start with a learning rate of 10−4. In both cases, the learning rate is decreased linearly after every epoch. After training, we keep the checkpoint scoring the highest accuracy on our leftout validation split for inference. We use a batch size of 8 for all training runs.
For all evaluated CLIP models, we rely on PyTorch implementations in the HuggingFace Transformers library (Wolf et al., 2020). Adapter method implementations are provided by the adaptertransformers library (Pfeiffer et al., 2020).

heading: 5 Results and Analysis

heading: 5.1 Results on Validation Data
text: We first analyze the performance of all evaluated methods on our held out validation set (Table 2).
Zero-shot Comparing the zero-shot performance
of the selected CLIP checkpoints, it is apparent that all pre-trained models perform reasonably well without fine-tuning on the training set. With accuracy scores between 72% and 81%, they substantially outperform a random baseline. Comparing between the evaluated models, performance increases with model size. The best zero-shot scores are obtained by OpenAI ViT-L/14-336 and LAION ViT-G/14, the largest evaluated representatives of their family. For base size models, we find that the multilingual LAION XLM-R Base slightly outperforms its monolingual counterparts. As we also suspect this model to have the highest potential on the multilingual test data, we focus on evaluating it in the following.
Fine-tuning Switching to the fine-tuned models, we observe that all evaluated methods are able to improve performance on the validation split. In general, accuracy scores on XLM-R Base increase by roughly 12 points compared to zero-shot performance of the same model. Interestingly, we find no substantial performance differences between the different transfer learning methods. Costly full fine-tuning of both the vision and text encoder of CLIP performs similar to various adapter methods and simple fine-tuning of CLIP’s vision and text projection layers. The picture for LAION XLMR Large is similar. Here, full fine-tuning slightly outperforms the other two methods, achieving our overall best scores of 91.91% accuracy and 95.37% MRR. From these results, we can conclude that it is possible to adapt CLIP to visual WSD efficiently without costly fine-tuning of all parameters on task data.

heading: 5.2 Results on Test Data
text: We transfer various approaches for the multilingual models to the test set. Table 3 shows accuracy and MRR scores per language as well as average scores across all languages.
Zero-shot Again, we find that zero-shot performance of the two evaluated multilingual LAION XLM-R models is substantially better than random across all languages. Similarly, they outperform the baseline provided by the task organizers, consisting of a zero-shot OpenAI ViT-L/14-336 for English and a zero-shot variant of OpenAI ViT-B/32 for Italian and Farsi. The latter has been made multilingual using Knowledge Distillation (Reimers and Gurevych, 2019). The best evaluated zero-shot model, LAION XLM-R Large, improves over the
baseline by 9 points on English and Farsi and by 39 points on Italian in terms of accuracy.
Fine-tuning Comparing the fine-tuned LAION XLM-R Base models, we find no meaningful improvements over zero-shot performance on the test set in most cases. Only LoRA improves over zero-shot performance by 2.4 accuracy points and 2 MRR points on average. On LAION XLM-R Large, LoRA and Bottleneck adapters improve slightly over zero-shot performance, reaching MRR scores over 70%.
Intermediate Task Transfer Results corresponding to the method described in § 3.3 are listed as SemCor Adapter. They perform similar to our other approaches, with the base variant even being outperformed by LoRA. We can conclude that knowledge transfer between text-only and multimodal WSD is non-trivial and requires further investigation.
Analysis In general, fine-tuning CLIP on the training set does not substantially improve results on the test set in our setup. To investigate potential causes for the limited knowledge transfer between training and test sets, we further analyze the data. Figure 1 compares the distribution of number of WordNet synsets2 per target word in both data splits. We can observe that the majority of target words in the training set are found in zero to two WordNet
2Synsets are groups of words sharing the same meaning within WordNet.
synsets. In contrast, samples from the test set are found in 6 synsets on average, with some samples occurring in 15 or more synsets. Samples in the training set thus are overwhelmingly specific in their meaning, leaving less room for ambiguity, while test samples show diverse sets of meanings that might prove more difficult to disambiguate. We hypothesize that a WSD system fine-tuned on the Visual-WSD train split using the presented simple techniques therefore could generalize poorly to the test split. We leave further investigation of this issue to future work.

heading: 6 Conclusion
text: As part of the SemEval-2023 Visual-WSD task, we investigated the multimodal WSD capabilities of various pre-trained CLIP models in a zero-shot and multiple transfer learning settings. We found that large pre-trained CLIP models perform reasonably well in the zero-shot setting, even across languages, yielding the conclusion that these models have substantial inherent visual WSD capabilities. Fine-tuning CLIP checkpoints using different techniques, including parameter-efficient adapter methods, leads to further improvements on indistribution validation data. However, improvements over zero-shot performance on the multilingual test set are little. We conclude that knowledge transfer to the test set is non-trivial and approaches beyond the simple ones tested might be required for further substantial improvements.

heading: Acknowledgements
text: We thank Myra Zmarsly and Gianluca Zimmer for their support on this project and reviewing our paper.

heading: Future Work
text: We leave further investigation of this issue to future work