YNU-HPCC at SemEval-2023 Task 9: Pretrained Language Model for Multilingual Tweet Intimacy Analysis
Year: 2023

Abstract:
This paper describes our fine-tuned pretrained language model for task 9 (Multilingual Tweet Intimacy Analysis, MTIA) of the SemEval 2023 competition. MTIA aims to quantitatively analyze tweets in 6 languages for intimacy, giving a score from 1 to 5. The challenge of MTIA is in semantically extracting information from code-mixed texts. To alleviate this difficulty, we suggested a solution that combines attention and memory mechanisms. The preprocessed tweets are input to the XLM-T layer to get sentence embeddings and subsequently to the bidirectional GRU layer to obtain intimacy ratings. Experimental results show an improvement in the overall performance of our model in both seen and unseen languages.

heading: 1 Introduction
text: Intimacy involves lots of social information (e.g., social norms). (Pei and Jurgens, 2020) By evaluating the intimacy score of tweets crawled on social media sites like Twitter quantitatively, it is possible to mine more meaningful social information. MTIA aims to build a sentiment analysis model (or system) to predict the intimacy scores, ranging from one to five, of tweets in a Twitter dataset containing ten languages, including English, Spanish, Portuguese, Italian, French, Chinese, Hindi, Dutch, Korean and Arabic. Among those ten languages, the first six are observable (i.e., appear during training), and the rest are unobservable (used for zero-shot learning).
Tweets abound with online slang, emojis and face characters, which usually contain extra intimate thoughts. Thus, it is critical to extract semantic information from those unconventional texts. However, many accessible pre-trained models (e.g., XLM) lack consideration of non-traditional text in the training process. The non-traditional text needs elimination to de-noise when we perform transfer learning on these models. Another feasible approach is to convert these non-traditional symbols
to conventional words. However, the extraction of information from face characters remains a challenge.
Fortunately, we can perform transfer training using the XLM-T model (Barbieri et al., 2021) trained on the Twitter dataset, thus avoiding information loss in the symbolic text. Additionally, we suggest a better way to link bidirectional GRU (Cho et al., 2014) layers at the XLM-T. The approach, which combines attention and memory mechanisms, is inspired by the analogy to humans. We will demonstrate experimentally that this measure enhances the model’s overall performance, both seen and unseen text. Our model achieved 13/45 places in terms of overall performance in the leaderboard, with both Dutch and Arabic ranking fourth. The code will be available at https://github. com/cskujou/tweet-intimacy.
The roadmap for this paper is as follows. Section 2 describes the work related to sentiment analysis. Section 3 describes the architecture of our model. Section 5 reports the experimental results. Section 6 concludes the paper.

heading: 2 Related Works
text: In the past, researchers have used lexicon-based methods and machine-learning approaches for sentiment analysis. (Wankhade et al., 2022) Since then, the performance of sentiment analysis tasks has significantly increased because of the development of deep learning algorithms. CNN (LeCun et al., 1998), LSTM (Hochreiter and Schmidhuber, 1997), and GRU (Cho et al., 2014) are examples of deep learning models. Wang et al. (2022) proposed a contextual sentiment embedding model that can distinguish the meaning of the same word in different contexts and improve the performance of the sentiment task. Recently, by resolving the issue of long-distance dependency, BERT-based large-scale pre-training models achieved a significant advancement in sentiment analysis (Zheng
733
et al., 2022; Bai et al., 2022). Early multilingual NLP models included mBERT (Devlin et al., 2018), XLM (Conneau and Lample, 2019), and mBART (Liu et al., 2020). Goyal et al. merged the XLM and the RoBERTa model to create the more potent XLM-R (Conneau et al., 2020). Barbieri et al. obtained the XLM-T using an identical architecture to the XLM-R, trained on the Twitter dataset. The XLM-T has a significant improvement in its ability to extract symbolic information.

heading: 3 Model Description
text: The XLM-T model (Barbieri et al., 2021) was applied as the baseline model. Moreover, we connect two more bidirectional GRU layers on top of XLMT to get the improved model called XLMT-GRU. Figure 1 illustrates its architecture. We’ll go into more detail about this model in the following paragraphs.

heading: 3.1 Preprocess
text: Preprocessing can help to reduce the amount of bias in the model, leading to fairer and more accurate results. Notice the presence of the username in the dataset, whose pattern is a token beginning with "@". We eliminate all usernames after preprocess-
ing since we empirically think they are unre-lated to intimacy. Additionally, the presence of a username may interfere with predictions. For ex-ample, a username token “@love_jenny” may increase the intimacy of a sentence. We will maintain each individual "@" character and only remove those whose token length is more than one since the "@" character may also be used to represent the word "at".

heading: 3.2 Tokenizer
text: A tokenizer splits a text into tokens, which are then used by the language model to generate responses. SentencePiece (Kudo and Richardson, 2018), a language-independent tokenizer, is used by XLM-T. A sentence of any language can be broken up into smaller pieces efficiently by using it. It accomplishes this by combining the BPE and unigram algorithms. The unigram model assigns a probability to each word and presumes that each is independent. The probability is calculated using the maximum likelihood estimation method, which assumes that the probability of each word is equal to the number of times it appears in the corpus divided by the total number of words in the corpus. Mathematically, this can be expressed as:
P (w) = Count (w)
N ,
where P (w) is the probability of a given word w, Count(w) is the number of times the word appears in the corpus, and N is the total number of words in the corpus. The BPE algorithm examines at the frequency of each subword in the corpus and assigns a single token to the most frequent subwords. This process is repeated until no byte pair appears more than once in the vocabulary. The Sentence-Piece algorithm is a modification of the BPE algorithm that adds a smoothing factor to the probability calculation, which makes the model more robust to unseen input. Also, SentencePiece has a parameter that allows users to specify the number of tokens they want to generate.
A statement like "Using SentencePiece for tokenization." would be split up into subword units like [’_U’, ’sing’, ’_Sent’, ’ence’, ’Pie’, ’ce’, ’_for’, ’_to’, ’ken’, ’ization’, ’.’, ’_’]. Note that the character "_" is not an underscore, but a Unicode character with the code value "U+2581", indicating the beginning of a sentence or a space. XLM-T has a vocabulary size of 250K and can convert tokens into their corresponding IDs. Subword information
can be extracted efficiently using SentencePiece. As a result, the language model can comprehend the statement more clearly and make a more accurate sentiment analysis.

heading: 3.3 XLM-T Model
text: The main advantage of using pre-trained models is that they reduce the cost and effort required for deep learning since they eliminate the need to spend time and money gathering and cleaning data when training a model from scratch. Additionally, pre-trained models typically provide superior results because they already have optimized weights and can converge faster. The XLM-T is a pre-trained model that performs superiorly on multilingual natural language processing tasks. It consists of an embedding layer and twelve transformerbased encoding layers, each with 768 hidden units and 12 attentional heads. The most important among them is the encoding layer. Each encoding layer comprises two sublayers: a multi-head self-attention layer and a feed-forward layer.
Each token is converted into a word embedding, while the position of the tokenizer in the sen-tence is converted into a positional embedding. The first layer of encoding receives the sum of these two embeddings and feeds its output to the second layer, then the second layer feeds its output to the third layer, and so on. In the end, the token passes through 12 encoding layers, resulting in a vector of length 768. The main advantage of XLM-T is that it is trained on the Twitter dataset, in line with MINT. Thus, XLM-T is better suited to unconventional text, particularly the semantics of emojis.

heading: 3.4 Bidirectional GRU
text: Gate Recurrent Unit (GRU) can learn representations from both the past and future time sequences to capture long-term dependencies, and it can help to reduce the vanishing and exploding gradients issue. In comparison with the LSTM, the GRU has fewer parameters, resulting in faster training and less data. Two GRUs make up a bidirectional GRU, also known as a BiGRU. One processes the input moving ahead, while the other moves backwards. GRU comprises two gates: the update gate and the reset gate. The update gate instructs the model on which parts of the input are important or insignificant, whereas the reset gate instructs the model on how to keep track of the context and how much of the previous state to pass on to the next step. By doing this, the model can capture long-
term dependencies in both directions. Finally, the output values of the two GRUs are concatenated and used as the bidirectional output. For instance, input a token of length N and then BiGRU will output a matrix of 2H×N , where H is the number of hidden cells in BiGRU.

heading: 3.5 Linear Layer
text: The goal of the linear layer is to transform the BiGRU result into a scalar. Due to the GRU output being a sequence, we concatenate the first and last outputs to obtain a vector of length 4H . A linear layer with 4H hidden cells is then used to translate the vector into a value.
The loss function in our model is the MSE function. Assume that the actual value is y and the model’s predicted value is ŷ, where i is the index. Then the MSE can be calculated by the following formula.
1
n
n∑
i=1
(y − ŷ)2,
where n is the size of training set.

heading: 4 Experimental Setup
text: In this section, we first introduce the dataset and the evaluation metric and then go over the model implementation detail. Next, we demonstrate the improvement of our model over the baseline model through experiments. Further, we will include two models for comparison purposes.
• XLM-R: XLM-RoBERTa-base model
• XLMR-GRU: Similar structure to the XLMTGRU, except that the XLM-T layer has been replaced by the XLM-RoBERTa-base.
The experiments are divided into two parts. The first part reveals the performance of our model on the observable language, while the second part shows the performance on the unobservable language.

heading: 4.1 Datasets
text: The dataset used for the experiments is the MINT provided by Pei et al. (Pei et al., 2022) There are 13,384 tweets in the MINT, and there are tweets in ten different languages. Table 1 lists the number of tweets in each language. The model will be trained using the first six languages in the table, and the final four languages will be used as test data
to evaluate the performance of zero-shot learning. We split the training set in the ratio of 8:2 into a training set and a development set for the finetuning phase. Only the training set is used to train the model of different hyperparameters. And we choose the model that outperforms the others on the development set.

heading: 4.2 Evaluation Metrics
text: MTIA requires the model to output a number between one and five as a predictor of the intimacy score, where one means not intimate and five means most intimate. We will use Pearson’s r as an evaluation metric of model performance. The closer this indicator is to 1, the more closely the model’s prediction resembles the actual number, indicating a higher degree of accuracy. The formula of Pearson’s r is as follows.
r = ∑m i=1(ŷi − ¯̂y)(yi − ȳ)√∑m
i=1(ŷi − ¯̂y)2 ∑n i=1(yi − ȳ)2 .

heading: 4.3 Implementation Details
text: To implement the entire model, we used the transformer library from the Hugging Face community and the PyTorch deep learning framework. The pre-training parameters of the model are twitterxlm-roberta-base developed by cardiffnlp. Regarding the hyperparameters, we keep the parameters of XLM-T as default (i.e., twelve encoding layers, 768 hidden units and 12 attention heads). There are two bidirectional GRU layers containing 256 hidden units. The dropout ratio between the XLM-T, BiGRU and linear layers is 0.1. We use the Adam optimizer to train the model. Adam can often converge faster than other algorithms, and it helps to mitigate the weight decay problem. The learning rate is 5e-6, the number of epochs is 3, the size of
the batch at training is 8, and the size of the batch at evaluation is 16.

heading: 4.4 Comparative Results
text: Table 2 and Figure 2 (a) show the performance of models on seen languages. Note that XLMT-GRU greatly enhances the model’s overall performance in the seen languages by significantly increasing prediction accuracy in both English and Spanish. On the other hand, the XLMR-GRU also significantly outperforms the XLM-R, confirming the GRU layer’s capability to boost the performance of the XLM model. We observe that XLM-T outperforms XLM-R in general. It motivates us to believe that pre-trained models and tasks should correspond as closely as possible, particularly for the dataset. To our disappointment, the performance of XLM-T on Chinese is somewhat weakened. This is because the Chinese text in the Twitter dataset is relatively small, roughly one thousandth the size of the English text. Moreover, the Pearson’s r be-
tween our model’s predictions and the actual values is 0.7109 on the test set and 0.7064 on the development set. The similarity between the two numbers suggests that our model is not overfitting.
Table 3 and Figure 2 (b) show the performance of models on unseen languages. Note that the XLMT-GRU’s performance in Hindi has regressed. However, there is a significant improvement in performance in Dutch and Arabic. Thus, our model outperformed the baseline model on zero-sample learning. The same results can also be obtained by comparing the XLMR-GRU with the XLM-R. To explain why the performances of zero-shot learning results are so different, we analyze them in terms of linguistic similarity. First, Dutch and Portuguese are both Romance languages and share several similarities in terms of grammar, syntax, and vocabulary. Second, the typology and script of Arabic differ from that of English, but the representations are quite similar. However, Hindi and the six seen languages have very different grammatical rules and writing systems. Finally, the Korean language has been influenced by Chinese and therefore performs slightly better than Hindi. However, Korean is an agglutinative language and Chinese is an analytic language, and they have different grammar and morphology. Those facts support our experimental results: all models performed much better in Arabic and German than in Hindi and Korean. Figure 3 illustrates the prediction errors of the XLMT-GRU model in various languages. The horizontal coordinate represents the difference between the predicted and actual values, while the vertical coordinate represents the probability density. Notice that the expectation of the error on
the seen languages is close to 0. The maximum deviation value is around 1.8. On the unseen languages, the expectation deviates significantly from 0. The larger the offset value, the worse the prediction. Our model underestimates the intimacy of Hindi and Korean and slightly overestimates the intimacy of German and Arabic. Comparing the two experiments together, we find that integrating the attention and memory mechanisms results in an improvement in model performance. This is because GRU converges more quickly on small data sets and tends to outperform a simple linear layer.

heading: 5 Conclusion
text: In this paper, we study the multilingual intimacy analysis task. We attempted to combine XLM-T and BiGRU and found that the overall performance of our model outperformed the baseline model. Our model ranked 13th on the leaderboard and 11th on unobservable languages. Combining at-
tentional and memory mechanisms can boost the performance of our model for sentiment analysis on small datasets. In the future, we will also try to integrate other models (e.g., TreeLSTM), mainly to complement the shortcomings of our current model on Hindi.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051. The authors would like to thank the anonymous reviewers for their constructive comments.

heading: Future Work
text: In the future, we will also try to integrate other models (e. g. , TreeLSTM), mainly to complement the shortcomings of our current model on Hindi