TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation
Year: 2023

Abstract:
Visual Word Sense Disambiguation (WSD), as a fine-grained image-text retrieval task, aims to identify the images that are relevant to ambiguous target words or phrases. However, the difficulties of limited contextual information and cross-linguistic background knowledge in text processing make this task challenging. To alleviate this issue, we propose a Fine-grained Contrastive Language-Image Learning (FCLL) model, which learns finegrained image-text knowledge by employing a new fine-grained contrastive learning mechanism and enriches contextual information by establishing relationship between concepts and sentences. In addition, a new multimodalmultilingual knowledge base involving ambiguous target words is constructed for visual WSD. Experiment results on the benchmark datasets from SemEval-2023 Task 1 show that our FCLL ranks at the first in overall evaluation with an average H@1 of 72.56% and an average MRR of 82.22%. The results demonstrate that FCLL is effective in inference on fine-grained language-vision knowledge. Source codes and the knowledge base are publicly available at https://github.com/CharlesYang030/FCLL.

heading: 1 Introduction
text: The problem of word polysemy, first recognized in machine translation, is one of challenging tasks in Natural Language Processing (NLP). In the field of NLP, resolving word polysemy is generally regarded as Word Sense Disambiguation (WSD), and it still remains as one of the most challenging and pervasive linguistic phenomena in NLP at present (Bevilacqua et al., 2021). To promote the research of the issues, international competitions have been held such as SemEval1 to advance the state-of-theart in word sense analysis and to help create highquality annotated datasets in WSD. Since SemEval-
*Corresponding author. 1https://semeval.github.io/
2023, visual WSD has been introduced and it contains three tracks including English, Farsi and Italian (Raganato et al., 2023). Given a target word and some limited textual context, the visual WSD task is to select the one which relevant to the intended meaning of the word among a set of candidate images.
Theoretically, the general WSD task aims at making explicit the semantics of a word in context by identifying the most suitable meaning (called sense) from a pre-defined sense inventory (Bevilacqua et al., 2021), which only involves words and senses. Differently, visual WSD requires models to learn more fine-grained image-text knowledge to establish the relationship among words, senses and images. Meanwhile, the limitations of visual WSD are significant. Because it is difficult in most cases to determine the specific sense of a word from a phrase containing few words without additional contextual information, and even more difficult to further identify the images related to the word. In addition, words in text of different languages may have different senses, and the syntactic and structural differences among languages make models ineffective during textual processing. Therefore, cross-language background knowledge is another major challenge in visual WSD.
In this paper, we propose a Fine-grained Contrastive Language-Image Learning (FCLL) model for learning fine-grained image-text knowledge. The main contributions of this work lies on three aspects:
1. A new model named FCLL is proposed, in which a subtlety discriminative text encoder and an image encoder are designed from textinferred images and image-inferred texts perspective respectively.
2. A new visual WSD knowledge base named VWSD KB as an easily expandable multimodalmultilingual knowledge base is constructed. It contains three languages, 12956 ambiguous
506
target words, 20904 senses and 97267 relevant images.
3. A new sense auto-complementing strategy is proposed, where an exclusive sense encoder is activated to complement English contextual information to phrases.

heading: 2 Related Work
text: In recent years, the dominant approaches in general WSD have included KB-based and supervised methods. Most KB-based models, such as SyntagRank (Scozzafava et al., 2020) and SREFKB (Wang and Wang, 2020) , employ specific rank algorithms and retrieval rules. Most supervised approaches (e.g., SensEmBERT (Scarlini et al., 2020) and BEM (Blevins and Zettlemoyer, 2020)) are based on neural network systems that use pretrained models. Besides, several studies (e.g., (Gella et al., 2019)) have utilized visual features to solve the problem of textual WSD. However, these models do not involve fine-grained imagetext knowledge and cannot be directly applied to visual WSD.
In vision-language pre-training, contrastive learning (Hadsell et al., 2006) focuses on learning common features between similar instances.
CLIP (Radford et al., 2021) employs a classical two-stream structure using more than 400 million image-caption pairs collected from the Internet as training data, which is equipped with an outstanding performance for zero-shot. BLIP (Li et al., 2022) is an extension of CLIP, learning from noisy image-text pairs by bootstrapping the captions. Moreover, FLAVA (Singh et al., 2022) is an alignment model, which obtains image-text features separately using unimodal encoders, followed by multimodal pre-training. However, most of the existing multimodal pre-trained models use manually managed datasets that are coarse-grained, such as COCO (Lin et al., 2014) and Flickr-30 (Plummer et al., 2015). Furthermore, these models are studied in the monolingual background.

heading: 3 Method

heading: 3.1 The FCLL Model
text: We propose the FCLL model for learning multimodal-multilingual fine-grained image-text knowledge. Figure 1 illustrates the learning framework of the model, which is composed by four modules including a fine-grained text encoder, a fine-grained image encoder, a fine-grained imagetext matching predictor and a word sense auto-
complementing module. Inspired by MoCo (He et al., 2020), the FCLL model maintains an image queue and a text queue separately. Fine-grained Text Encoder (the blue dashed box). All the relevant images of ambiguous target words in a mini-batch are retrieved, and a text encoder based on CLIP promotes semantic consistency between the senses guided by these words and the images from the text-inferred image perspective. Then, these images are updated to an image queue with length 80000. Fine-grained Image Encoder (the red dashed box). Differently, all the senses of ambiguous target words in a mini-batch are retrieved, and an image encoder based on CLIP activates semantic consistency among the images guided by these words and the senses from the image-inferred text perspective. Then, these senses are updated to a text queue with length 20000. Fine-grained Image-Text Matching Predictor (the solid purple box). A transformer connected by a simple classifier is trained to fuse the multimodal features of the “words + senses + images” combinations. The matching predictor as well as facilitates the contrastive learning from the perspective of feature fusion.

heading: 3.2 The Word Sense Auto-complementing
text: Theoretically, the length of phrases affects the performance of a visual WSD model to some extent. Therefore, an exclusive sense encoder is trained for complementing a desirable sense to a phrase. As shown in Figure 2, FCLL removes the stop words of each original sense (called the full sense) and combines any one of the remaining words with their corresponding ambiguous target word to form a concept, and subsequently promotes semantic consistency between the concept and the full sense. Note that all the non-English texts in test set are translated to English by Google Translate, and the candidate synsets (synonym sets) are provided by BabelNet (Navigli and Ponzetto, 2010; Navigli et al., 2021).

heading: 3.3 Loss Function
text: Contrastive losses (Hadsell et al., 2006) is used to calculate the similarities of two vectors in a representation space. Based on ALBEF (Li et al., 2021), we define three contrastive losses including Senseatw-Image Contrastive (SIC) loss, ImageatwSense Contrastive (ISC) loss and TextconceptTextfull Contrastive (TTC) loss. In the SIC loss,
the combinations of the senses guided by an ambiguous target word (atw) and positive images are used as two vectors. In the ISC loss, the combinations of images guided by an ambiguous target word and positive senses are used as two vectors. Similarly, in the TTC loss, the combinations of concepts and full texts are used as two vectors. In addition, we follow BLIP and define a cross entropy loss named Image-Text Matching loss, which is a binary classification task essentially.

heading: 3.4 The Visual WSD Knowledge Base Construction
text: Empirically, the datasets provided by SemEval2023 Task 1 suffer from low data volume, monolingual English, and the lack of contextual information, which is further discussed in Section 4.1. To this end, we firstly select ambiguous target words as queries based on officially released datasets and then crawl all the synsets of these queries online by the API guidelines2 from BabelNet. Here we choose English, Farsi and Italian. The resource of synsets comes from WordNet (Miller et al., 1990), and the part-of-speech is set as noun. For each sense, several relevant images are displayed in BabelNet. We employ a top-5 strategy, i.e., only the top five images are collected (due to the format and legality of images, less than five images can be collected in some cases). After nearly two months of collection and validation, V-WSD KB contains 12,956 multilingual ambiguous target words, 20,904 English senses, and 97,267 relevant images. The maximum number of senses for an ambiguous target word is 23, the minimum number
2https://babelnet.org/guide
is 1, and the mean number is 1.613. The maximum number of images for a sense is 5, the minimum number is 1, and the mean number is 4.653. Note that SemEval-2023 Task 1 claims that participants are allowed to use external data sources. Accordingly, V-WSD KB is constructed based on BabelNet 5.2 under SapienzaNLP license agreement and is used as complementary training data for FCLL.

heading: 4 Experiments and Results

heading: 4.1 Datasets
text: We use the training/test sets released by SemEval2023 Task 1, and the distribution of ambiguous target words for each language in the datasets is shown in Table 1. Although the training set involves multiple languages, their proportions are severely imbalanced. Instead, only three languages are involved in the test set. Furthermore, there is only one data record in Farsi and even none in Italian in training set, and there is only one reduplication of target words in training/test sets.
Table 2 shows the differences among training set, test set and V-WSD KB. Note that the structure of “atw-text-image” (i.e., word-sense-image)
correspondence as “1-n-n” means that each ambiguous target word may correspond to multiple meanings, and each meaning also corresponds to multiple associated images.

heading: 4.2 Settings
text: Visual WSD is recognized as a ranking problem and it adopts Hit Rate at 1 (H@1) and Mean Reciprocal Rank (MRR) to measure model performance, which can be calculated by Eq. (1) and (2):
H@1 = Number of Hits @1
Q (1)
MRR = 1
Q
|Q|∑
i
1
Ranki (2)
Q indicates the total number of recommendation lists, and Number of Hits @1 represents the number of items in the first position of each recommendation list that are consistent with groundtruth, while Ranki denotes the column position of the item in the i-th recommendation list that is consistent with ground-truth.
Our model was implemented on Pytorch and a single Nvidia A100 GPU in about 35 hours. We set the mini-batch size to 2, the initial learning rate to 0.0001, and the number of training epochs to 10. Furthermore, we use AdamW as the optimizer, and the weight decay of AdanW is 0.05. After each epoch, the model performs a cosine learning rate decay. We follow CLIP and specify an image resolution of 224×224, a maximum text length of 77, and a text that starts with “A photo of”.

heading: 4.3 Results
text: There are 98 submissions in this visual WSD campaign. FCLL wins the first place on the leaderboard for the final average score in multiple languages. We collect the results of the official baseline and
the top-3 models, and then we use CLIP, BLIP and FLAVA for zero-shot on the test set that has been translated to English. The comparison results are shown in Table 3. FCLL outperforms the baseline models and shows the flexibility to language changes, whose balanced performance is benefited from its fine-grained core modules. In addition, BLIP is comparable to CLIP in terms of average score. The former has 447M parameters, while the later has 151M. In contrast, FCLL has only 38M parameters more than the original CLIP and further advance the performance on learning fine-grained image-text knowledge.

heading: 4.4 Ablation study
text: To conduct an ablation study, we remove the finegrained text encoder, the fine-grained image encoder, the fine-grained image-text matching predictor and the sense auto-complementing module. The results, as shown in Table 4, illustrate that our fine-grained text encoder and sense autocomplementing module are the most critical components. The fine-grained image encoder and the matching predictor can further contribute to the performance of understanding on fine-grained imagetext knowledge.

heading: 5 Conclusion
text: This paper proposes a FCLL model for inference on fine-grained image-text knowledge and complementing additional English contextual information to phrases composed by limited words. Besides, we demonstrate that our fine-grained contrastive language-image learning approach and sense autocomplementing module support FCLL for understanding fine-grained image-text knowledge. Moreover, we construct a new multimodal-multilingual fine-grained image-text knowledge base, which can be applied to visual WSD to improve FCLL performance. The results on the benchmark test set from SemEval-2023 Task 1 show that FCLL achieves an average H@1 of 72.56% and an average MRR of 82.22%, ranking at the first in overall evaluation. Source codes and the knowledge base have been released to facilitate future fine-grained languageimage research.

heading: Future Work
text: Source codes and the knowledge base have been released to facilitate future fine-grained languageimage research