ECNU MIV at SemEval-2023 Task 1: CTIM - Contrastive Text-Image Model for Multilingual Visual Word Sense Disambiguation
Year: 2023

Abstract:
This paper describes the performance of MIV team in SemEval-2023 Task-1-Visual Word Sense Disambiguation (Visual-WSD). This task is to give a potentially ambiguous word and some limited textual context and select among a set of candidate images the one which corresponds to the intended meaning of the target word, which plays a critical role in human language understanding. Our team focuses on the multimodal domain of images and texts, we propose a model that can learn the matching relationship between text-image pairs by contrastive learning. More specifically, We train the model from the labeled data provided by the official organizer, after pre-training, texts are used to reference learned visual concepts enabling visual word sense disambiguation tasks. In addition, the top results our teams got have been released showing the effectiveness of our solution. We release our code at https://github.com/Insaner1004/VWSD.

heading: 1 Introduction
text: Word sense disambiguation (WSD) consists of associating words in context with their most suitable entry in a predefined sense inventory, which is a historical and hot point in natural language processing and artificial intelligence. Supervised (Luo et al., 2018; Barba et al., 2021) and knowledgebased(Moro et al., 2014; Chaplot and Salakhutdinov, 2018) are two common research methods that are widely used. However, visual word sense disambiguation aims to select among a set of candidate images the one which corresponds to the intended meaning of the target ambiguous word, which is completely different from the previous work(Raganato et al., 2023). In the real world, information often exists in different modalities, therefore, the fusion of multimodal information plays an important role in the research of deep learning.
This task is also useful for vision-and-language tasks (Lu et al., 2019) and text-to-image genera-
tor(Wang et al., 2022). In this paper, we focus on the computing of similarity between text and image track. As shown in Figure 1, given the ambiguous word "andromeda" and some limited textual context "andromeda tree", select one of the best corresponding images from candidate pictures to match the semantics according to the similarity. Apart from that, it is worth noting that this task includes datasets in three languages: English, Farsi, and Italian.
In recent years, pre-training methods have revolutionized NLP and achieved state-of-the-art performance in many fields(Sarzynska-Wawer et al., 2021; Devlin et al., 2018; Raffel et al., 2020). To solve the above problems effectively, we use a selfsupervised pre-training method, the text is used as a supervisory signal to train a better visual model(Li et al., 2017; Radford et al., 2021). For the first, we use an image encoder and text encoder to extract feature representations of each modality separately. The second is joint multimodal embedding and calculating cosine similarities. The experimental results show that our approach has achieved competitive results.
101

heading: 2 Related Work

heading: 2.1 Word Sense Disambiguation
text: Word sense disambiguation (WSD) is the process of identifying the meaning of a word in a sentence or other contextual fragment. Previous researches mainly focus on English only, which can be divided into two groups: supervised systems and knowledge-based systems.
Integrate the context and glosses of the target word in order to make full use of both labeled data and lexical knowledge(Luo et al., 2018), a flexible toolkit (Papandrea et al., 2017) be designed for feature extraction. Besides, lexical knowledge(Blevins and Zettlemoyer, 2020) or graph structure(Bevilacqua and Navigli, 2020) has been applied to achieve impressive performance in many supervised systems (Loureiro and Jorge, 2019; Scarlini et al., 2020). Knowledge-based systems typically leverage WordNet(Miller, 1998) or BabelNet(Navigli and Ponzetto, 2012) as semantic networks.
In recent years, multilingual word sense disambiguation has become a hot research topic. OneSeC(Scarlini et al., 2019) proposed a method, which extract hundreds of thousands of sentences in Wikipedia to generate multilingual datasets automatically, Hauer(Hauer et al., 2021)apply machine translation to transfer existing sense annotations to other languages. With the unified sense representations, (Su et al., 2022) transferring annotations from rich sourced languages to poorer ones to address the annotation scarcity problem.

heading: 2.2 Contrastive Learning
text: In the early days of deep learning, superfluous labeling work and fixed prediction object category may restrict generality to models that transfer to other tasks. Recently, self-supervised learning has attracted much attention due to its success in computer vision(Doersch et al., 2015). Furthermore, contrastive learning is a kind of self-supervised learning method, it is also worth conducting further research.
In recent work, Zhirong Wu(Wu et al., 2018) learns a good feature representation that captures the apparent similarity among instances and saves numbers of negative samples in a discrete memory bank. According to the features of the same instance from different data augmentations should be invariant, Mang Ye(Ye et al., 2019) introduce a novel instance feature-based softmax embedding
method. CPC(Oord et al., 2018) demonstrates that their approach can learn useful representations to achieve a strong performance on different modalities. MoCo(He et al., 2020)build a dynamic dictionary with a queue and a moving-averaged encoder to facilitate contrastive unsupervised learning and have a profound impact on the subsequent research work. SimCLR(Chen et al., 2020) (A simple framework for contrastive learning of visual representations) without requiring specialized architectures or a memory bank.
With previous contrastive learning research methods, different BYOL(Grill et al., 2020)achieves a new state of the art without negative samples. DINO(Caron et al., 2021) design a simple self-supervised approach that can be interpreted as a form of knowledge distillation with no labels to address the difficulties encountered by Vision Transformers (ViT)(Dosovitskiy et al., 2020).

heading: 3 System overview

heading: 3.1 Task Definition
text: In this subsection, we redefine the visual word sense disambiguation task to an image-text similarity task. Given a potentially ambiguous word w and some limited textual context T . The goal of this task is to select one image among a set of candidate images M = {M1, ...,M|I|}, which corresponds to the intended meaning of the target word. Calculate the Cosine similarity between the text embedding and the image embedding then rank the candidate images based on the calculated result. We illustrate the framework of our model in Figure 2, which consists of three components: Multilingual Machine Translation, Text Encoder, and Image Encoder.

heading: 3.2 Multilingual Machine Translation
text: We input the Italian and Farsi limited textual context in the test data into the text encoder directly and achieved poor results. The main reason is that the text encoder pre-trained focuses heavily on English and the training data in this task is English. Therefore, it is difficult to achieve the desired result when using other languages. So, it is necessary to preprocess the given context in different languages. We used a ready-made solution (multilingual machine translation model) to overcome this difficulty. M4(Arivazhagan et al., 2019) model is a massively multilingual neural machine translation
system, released by Google Research that can translate between 103 languages. The encoder converts sentences in different languages into a semantic representation vector, and the decoder gradually translates them into the corresponding target language. Specific steps are as follows: Preprocessing stage: Segment, tokenize, and encode the input text so that the model can process it.
Encoder: Encode the input text to generate a semantic representation.
ht = fenc(xt, ht−1) (1)
Decoder: Use semantic representation to generate text in the target language.
st+1 = fdec(yt, st, c) (2)
Loss function:
L = − T∑
t=1
log p(yt|y<t, x) (3)

heading: 3.3 Text Encoder
text: We use a BERT(Devlin et al., 2018) network as our core architecture to extract text features. The input text is segmented, embedded, and position encoded, and then the context representation of each word is calculated by a multi-layer Transformer encoder. Finally, the representations of these words are pooled on average to obtain a vector representation of the entire sentence, which is the output of the text encoder. This vector is represented as
a fixed-size text embedding vector, which can be used to calculate the similarity between texts and can also be used to calculate the similarity between text and images with the image embedding vector obtained by the image encoder.
Word embedding layer: Map each word xi in the input text sequence x = [x1, x2, . . . , xn] to a d-dimensional vector ei, where d is the dimension of the word embedding. The calculation formula of the word embedding layer is:
E = [e1, e2, . . . , en] = Embedding(x) (4)
Multi-head self-attention layer: Perform multihead self-attention calculation on the word embedding vector to obtain an output vector A of n× d, where n is the length of the text sequence, d is the hidden layer dimension. The calculation formula of the multi-head self-attention layer is:
MHA(Q,K, V ) = Concat(head1, ..., headh)WO
(5) headi = Attention(QW Q i ,KW K i , V W V i ) (6)
A = MHA(E,E,E) (7)
Among them, Q = K = V = E is the input of the self-attention mechanism, and WQi ,W K i ,W V i are the weight matrices acting on Q,K, V respectively, h is the number of heads, Attention is the self-attention mechanism formula, Concat is the output of all heads spliced together, WO is the weight matrix of the output vector.
Feedforward neural network layer: perform feedforward neural network calculation on the output vector A of the multi-head self-attention layer, and obtain an output vector T of n× d. The calculation formula of the feedforward neural network layer is:
T = max(0,AW1 + b1)W2 + b2 (8)
Among them, W1,b1,W2,b2 are learnable weight matrices and bias vectors, and max is the ReLU activation function. Ultimately, the output vector of the text encoder is T for n× d.

heading: 3.4 Image Encoder
text: We leverage Vision Transformer (ViT)(Dosovitskiy et al., 2020) model as our core architecture. Vision Transformer model has received extensive attention in the field of computer vision and achieved excellent performance in multiple tasks, which may indicate that the golden age of convolutional neural networks (CNN) in the field of computer vision last for several years will be replaced by the Vision Transformer model.
Due to the wide application of the VIT model and its great success in the field of vision, we used the VIT model in our model to extract feature representations for every candidate image. Assume H × W is the input image size and C refers to the channel. The output is a dense embedding I ∈ RH×W×C .

heading: 3.5 Cosine Similarity
text: After extracting feature representations including text and image, scaling cosine similarities of each modality. Cosine similarity uses the value of the angle between two vectors in the multi-modal embedding space, which compared to other measures, cosine similarity pays more attention to the difference in direction of two vectors, rather than distance or length.
Similarity (T,M|I|) = T ·M|I|
∥T∥ × ∥M|I|∥ (9)
In all cases, our model can be trained on multilingual textual context and supports flexible mounts of candidate images through the image encoder.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets The Visual Word Sense Disambiguation (VisualWSD) task organizers share datasets in
two different phases. In the trial phase, datasets include trial data in English only, which contains 12869 train samples and 16 trial samples, including gold keys. The evaluation phase differs greatly from to trial phase, this phase is the main phase of the competition where input test sets in three different languages are available (English/Farsi/Italian). Each dataset is divided into three sections: ambiguous word, limited textual context, and candidate images. A depiction of the dataset shared by the organizer is shown in Table 1.
Concretely, in terms of data, we are allowed to use the training/trial data set provided by the organizers, pretrained vision and language models, as well as other sources of training data (we sholud be explain in detail the data sources).
Implementation Details For the text encoder, we utilize HuggingFace’s implementation of the BERT model, which has been pre-trained on English sentences for the task of language understanding. As a base size, we use a 63M-parameter 12-layer 512- wide model with 8 attention heads. The text sequence will be extracted and treated as the feature representation of the context, which is layer normalized and then linearly projected into the multimodal embedding space.
For the vision encoder, we have re-implemented the ViT model using Google’s repository as a reference. We select a ViT-B/32(Dosovitskiy et al., 2020) as our core architecture due to less computation. During our experiments, we observed poor performance when directly inputting unprocessed Italian and Farsi limited textual context in the test data into the text encoder. Therefore, we introduced a multilingual machine translation module which translated the other two languages into the language that the model was pre-trained and trained on, resulting in significantly better results.
We train our model for 32 epochs and keep the model that performs best regarding HR on the trial set. We use Adam optimizer with the learning rates of 1e-5. The dimensions of word embedding are 128. The max sequence length is 512. The dropout is 0.1. The reported test results are based on the
parameters that obtain the best performance on the development.

heading: 4.2 Evaluation Metrics
text: Visual word sense disambiguation is a ranking problem, where each instance contains a query of the target phrase in the text and multiple candidates of images, and you must predict the candidate described by the query. The organizers evaluate the quality of our model by computing the mean reciprocal rank (MRR) and hit rate (HR).
Mean Reciprocal Ranking (MRR) is a measure to evaluate systems that return a ranked list of answers to queries. For a single query, the reciprocal rank is 1rank where rank is the position of the highest-ranked answer. If no correct answer was returned in the query, then the reciprocal rank is 0.
MRR = 1
Q
Q∑
i=1
1
ranki (10)
Hit Rate at 1 (HR) is defined as each sample having only one gold key, and the gold key should be classified correctly.
HR = ∑N−1 i=0 (f (xi) == yi)
N (11)
where xi is the sample and the sample’s gold key is yi,the prediction function is f . Meanwhile, the sample in the test sets and the size-N present the total number of the test sets list.

heading: 4.3 Development Results
text: Our main development experimental results are presented in Table 2, which trains and tests on English datasets. The top 5 HR scores are between 75.00% and 93.75%. Also, most of the scores were concentrated at 60% and 70%. For MRR scores, it ranges from 79.16% to 95.83%, which spans a wide range. Our model obtains good performance on the two evaluation metrics. Particularly, our model obtains the third place in HR.

heading: 4.4 Evaluation Results
text: The best models verified on the development sets were used on the test sets which are the official competition sets. Table 3 shows the results on the test sets for our module and compares them with the baseline, which shows that our approach significantly better than the baseline model including three different languages. Moreover, Our model has a huge improvement over the baseline model in Farsi and Italian. Table 4 displays the results of the evaluation phase.

heading: 5 Conclusions
text: In this paper, we propose a unified and general model to deal with the visual word sense disambiguation task, which was constructed so that different modalities could be colluded including the image and text. Moreover, effective interaction and fusion between multimodal information play a key role in the creation and perception of information in computer vision and deep learning research.
Different from the previous word sense disambiguation studies, we transform this task into an image-text similarity task, and the model shows competitive results in most languages. As part of future work, we plan to further improve our model better handle the problems of the multilingual task.

heading: Acknowledgements
text: We express our gratitude to the organizers for providing such an inspiring competition and resolving our questions patiently. We would like to express our sincere gratitude to those who have provided detailed suggestions for our paper. Their diligent efforts are greatly appreciated.

heading: Future Work
text: As part of future work, we plan to further improve our model better handle the problems of the multilingual task